Compare commits

..
Author SHA1 Message Date
N0zoM1z0 c2e2ed9aa2 test(firmware): retain unverified workers before further launches 2026-10-11 02:38:48 +08:00
N0zoM1z0 1ae4bec45b Merge remote-tracking branch 'origin/main' into test/maintenance-boundary-contracts-20261011 2026-10-11 02:38:48 +08:00
N0zoM1z0 058bc0af35 test: verify current maintenance boundary behavior 2026-10-11 02:34:45 +08:00
morluto dce81ca400 perf(javascript): count candidate relevance without a second adjacency
Scan eligible candidate relations directly for query ambiguity instead of allocating and sorting another traversal index. Preserve direction, relation filters and cycle handling.

Semantic graph and builder suites plus public application workflow CLI tests passed. Full query outputs matched on a generated 20000-node graph; the single repeated-query timing run is indicative only.
2026-10-11 02:32:11 +08:00
morluto bb764d26bd perf(artifacts): stop parsing parent paths after a missing branch
Walk delimiter-terminated parent segments lazily instead of splitting full paths before discovering that no deeper observed parent exists. Preserve canonical paths and observed ancestry.

Verified real ZIP inventory with multiple 30000-component siblings and the full entry-order suite. A 96-member deep ZIP benchmark retained an identical full inventory digest and reduced median scan time from 517.5 to 464.5 ms across five measured inventories.
2026-10-11 02:32:04 +08:00
morluto acd8d86898 perf(android): validate prerequisites once per retained JADX session
Resolve current paths and JVM options and rehash source identities for each request. Probe Java modules and engine capabilities only when creating a new session; remove the superseded full configuration resolver.

Focused configuration, session retirement, cancellation, cleanup and Android MCP boundaries passed. The 34-operation retained session now performs one prerequisite probe. Protocol fixtures do not establish real JADX or JDK performance.
2026-10-11 02:31:57 +08:00
morluto e6d0b3993d perf(process): batch liveness rechecks within ownership phases
Recheck unreadable token and identity observations against one fresh live process snapshot per phase. Preserve token mismatches, fail closed on inspection errors, and retain liveness failure diagnostics.

Validated with focused process ownership, native capture, CLI and MCP boundaries. An eight-member injected-unreadable workload using native ps reduced snapshots from nine to two with equivalent failures; real permission-denied provider behavior was not exercised.
2026-10-11 02:31:07 +08:00
morluto 92a901e0db perf(process): share settlement probes across captured groups 2026-10-11 02:21:50 +08:00
morluto cef6d6d3c4 perf(artifacts): reuse trie membership for case-folded siblings 2026-10-11 02:21:40 +08:00
morluto bd972bf80b perf(javascript): reuse capture completeness during load classification 2026-10-11 02:21:35 +08:00
morluto ac9709a80a perf(reconstruction): index semantic evidence contexts per derivation 2026-10-11 02:21:28 +08:00
morluto d1e6fe2eca test(artifacts): close owned Mach-O readers after each case 2026-10-11 02:11:13 +08:00
morluto a147ecd4b4 perf(artifacts): index observed parents for deep archive paths 2026-10-11 02:10:50 +08:00
morluto 22fdcb8ab7 perf(process): share batched token observation across ownership checks 2026-10-11 02:10:44 +08:00
morluto b22d817073 perf(process): batch ownership tokens during rooted cleanup
Use the host bulk token reader for each cleanup validation while preserving identity checks before and after token reads. Fall back for unsupported readers or omitted rows, retain explicit unavailable observations, and preserve batch and individual failures. Source ownership regressions and real owned-process boundary tests pass; a warm Darwin validation probe reduced median time from 90 ms to 52 ms.
2026-10-11 01:59:30 +08:00
morluto 3d876c5c1a perf(javascript): validate semantic factory inputs once
Construct typed semantic records directly from cloned, normalized input and derived commitments instead of reparsing the same fields. Keep full graph integrity validation and verify caller mutation isolation. Four alternating cold runs preserved complete result digests while median projection time fell about 8 percent and 17 percent for the two 20,000-node capacity workloads.
2026-10-11 01:54:46 +08:00
morluto cf346368c4 perf(javascript): reuse graph buckets and query indexes
Append overlapping observations to node buckets instead of recopying groups, and share one node index across semantic seed admission and traversal. Preserve merged provenance and absent-seed behavior.
2026-10-11 01:54:36 +08:00
morluto e52c0a72c5 fix(browser): retain incomplete WebMCP replay coverage
Mark malformed WebMCP event envelopes, tool lists and entries as invalid protocol values while preserving valid siblings and out-of-scope classifications. Verify toolsAdded and toolsRemoved replay uncertainty through CDP discovery and capture comparison.
2026-10-11 01:54:26 +08:00
morluto 999e3e2098 fix(android): cancel admitted snapshot copies with their operation
Forward caller and timeout cancellation through engine and APK snapshots, release stream handles before removing partial output, and retain the existing root cleanup authority. Verify cancellation after destination bytes appear with a sparse filesystem fixture.
2026-10-11 01:54:20 +08:00
dakjdakdandYsopwnable c1217e711b fix(javascript): retain runtime scope in bundled module relationships 2026-10-11 01:36:12 +08:00
morluto 0d89978333 fix(android): release snapshot streams before closing file handles
Retain both non-auto-closing copy streams and destroy them after verification, before awaiting their FileHandle closure. This releases stream-held descriptor references on success and failure instead of deadlocking Android snapshot cleanup.

Validation: all 33 JADX subprocess integration tests passed on macOS with Node 24.18.0; check:fast and formatting passed. No real JADX provider execution or Linux/Windows verification was performed locally.
2026-10-11 01:12:48 +08:00
morluto 11e3fae697 fix(ci): repair boundary regressions and evidence assertions
Detach cancelled CDP sessions within the cleanup budget and retain event-finalization failures across retries. Honor Git filesystem encodings and exclude the promise-API probe from source-read failures.

Align existing integration regressions with retained process resources, filesystem-order-independent truncation, owned skill files, explicit native access facts and streamed JSON errors.

Validation: 132 acceptance tests passed (2 skipped), 83 affected boundary tests passed, Git/JSON/CLI boundary regressions passed, check:ci, docs:check and test discovery passed. Local verification used Node 24.18.0 on macOS; cross-platform CI remains pending.
2026-10-11 01:00:30 +08:00
morluto 603d04ee3a fix(ci): repair format gate and pin maintenance skill destinations
format:check failed on ApktoolProvider; existingMaintenanceScope now
returns skillDestinations (consumed by maintenance planning), so pin
the field in the exact-shape scope assertions.
2026-10-11 00:33:36 +08:00
morluto 472f313232 refactor(native): match UI controls by explicit object identity 2026-10-11 00:28:39 +08:00
morluto 1f7e279aa2 fix(adb): retain rollback ownership and cancelled acquisition facts 2026-10-11 00:28:39 +08:00
morluto db8ccaa251 fix(android): bind JADX snapshots and hashes to admitted sources 2026-10-11 00:28:25 +08:00
morluto 5daff4d67e fix(artifacts): retain Mach-O source identity through inventory 2026-10-11 00:28:25 +08:00
morluto a9fd1aecae fix(application): verify configuration and reference input reads 2026-10-11 00:28:24 +08:00
morluto abb8e81d3a fix(filesystem): bound and verify admitted file and JSON reads 2026-10-11 00:28:24 +08:00
YsopwnableandGitHub 6d8f95e38d fix(clients): preserve portable MCP schemas in native workflows (#1580)
* test(clients): verify native Command Code workflows

* docs: synchronize Italian provider guide links

* fix(contracts): make ADB identifier patterns portable

* fix(contracts): preserve portable ADB patterns after formatting
2026-10-11 00:09:43 +08:00
morluto c1b5ccf0a4 refactor(server): admit complete evidence workflows and share handlers
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:24 +08:00
morluto 5f30ea6ff2 fix(engines): serialize JEB acquisition and respect IDA cancellation
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:23 +08:00
morluto d438675677 fix(process): retain capture resources and bound filesystem snapshots
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:23 +08:00
morluto a84dea5500 fix(apktool): join active work and retain cleanup authority
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:22 +08:00
morluto 2a66ec9094 fix(browser): retain startup owners and coordinate retryable close
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:22 +08:00
morluto d1235fadc9 fix(providers): retain worker cleanup owners through shutdown
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:21 +08:00
morluto a01148553e fix(setup): preserve recovery backups and retain installer cleanup
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:21 +08:00
morluto a6f76f5da9 fix(graphs): separate source identities from display representations
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:20 +08:00
morluto 2ca3ad2386 fix(reference): preserve exact paths and unify symlink lookup
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:20 +08:00
morluto 5d0a883605 fix(native): bound metadata projection and compact Mach-O bind runs
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:19 +08:00
morluto 0f0d6e4d43 fix(managed): budget decoding before representation expansion
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:19 +08:00
morluto 6d4cefc4f7 fix(artifacts): admit bounded reads and preserve source identity
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:18 +08:00
morluto 2a0b58e440 refactor(cleanup): share failure composition and retain native UI owners
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:18 +08:00
246 changed files with 9668 additions and 3185 deletions
+6
View File
@@ -136,6 +136,12 @@ existing analysis families directly, for example
- Pull and capture outputs are additive: the destination must not already
exist. Re-running a completed pull refuses the existing directory instead
of overwriting evidence.
- Cancellation during a package pull preserves caller-selected output. The
error includes partial observations identifying completed artifacts and the
interrupted local path; remaining APKs were not attempted.
- Failed package acquisition rollback retains its directory identity and retries
cleanup before later work and during shutdown. Cleanup refuses a replacement
directory. Terminal shutdown stops admission and drains active operations.
- The provider holds no long-lived processes; every invocation is one
bounded adb run. The adb server belongs to the caller.
+6
View File
@@ -25,6 +25,12 @@ extraction have independent prerequisites; one does not require the other.
Both require util-linux `prlimit`, normally `/usr/bin/prlimit`; override its
absolute path with `REA_FIRMWARE_PRLIMIT_COMMAND` if needed.
REA retains at most 8 MiB of combined stdout and stderr per command. Exceeding
that diagnostic budget stops analysis with an explicit truncated-output error;
child process limits do not bound parent-side pipe retention. Completed commands
drain their output streams before recording diagnostics. Failed worker cleanup
retains the worker and workspace for retry before another request or at shutdown.
Unblob needs the external extractor for the selected format on its inherited
`PATH`: for example, `debugfs` for ext2/ext3/ext4. Its default Landlock sandbox
and skip rules stay enabled. Sandbox startup failures remain actionable errors;
+14
View File
@@ -38,6 +38,20 @@ Inspect separately obtained components explicitly and preserve their identities.
A valid implementation marker establishes a candidate native boundary, not
recovered native semantics or runtime behavior.
### Resource admission
The static provider admits regular input files up to 128 MiB before retaining
their bytes. Reads remain bounded if a file grows after admission. Metadata,
CIL decoding, and projected inspection records have a separate conservative
64 MiB representation budget; shared heap strings are cached, but repeated
output occurrences still consume that budget. These are inspection resource
policies, not PE format limits or a hard bound on total V8 heap use.
Input overflow returns a resource constraint. Decode or projection overflow
reports `resource-limit` with unavailable or partial coverage; omitted facts do
not become empty observations. Ambiguous PE section mappings are rejected
rather than interpreted according to section order.
## Analysis objective
REA's managed-code track is intended to answer five different questions without
+4
View File
@@ -187,6 +187,10 @@ Every capture requires `truncation_details`, with separate accounting for:
input, so raw omissions also limit rendered coverage.
- `filesystem_before` and `filesystem_after`: file-count/depth limits,
enumeration failures, whole-file hash budget and bytes successfully hashed.
Directory names are read incrementally within the remaining entry capacity.
When a directory exceeds that capacity, the retained subset follows filesystem
enumeration order and is then sorted; it is explicitly incomplete and cannot
establish that an omitted path is absent.
Each retained regular file without a digest has an aliased path, size,
remaining budget and reason: `file_bytes_budget`,
`file_changed_or_short_read`, or `file_unavailable`. Its `system_code` is
+24
View File
@@ -187,6 +187,30 @@ The loopback OpenAI completions/SSE custom model declares a one-million-token
context and synthetic usage. Live xAI APIs, native Windows and default OS-home
skill discovery remain unverified.
`verify:commandcode-client` requires installed Command Code (verified with npm
1.79.2); select it with `REA_VERIFY_COMMANDCODE_COMMAND`. Auth, MCP tokens and
sessions use the actual OS home, so provision a disposable POSIX account rather
than overriding `HOME`. Set `REA_VERIFY_COMMANDCODE_ACCOUNT_HOME` to that account's
actual home and create `.rea-client-verification` there with exactly
`Disposable REA client verification account` followed by a newline. The lane
owns that account's `.commandcode` configuration and shared REA skill. Build the
runtime as the checkout owner, then run `node scripts/verify-commandcode-client.mjs`
as the disposable account; the npm shortcut also needs a writable checkout.
The verifier does not create accounts or install clients. It guards setup
targets, preserves an unrelated registration and backups, checks idempotence,
and verifies native default shared-skill discovery and complete activation.
Default deferred-schema delivery stays enabled: the native prompt advertises
every REA tool, exact-name `search_tools` returns every input schema, and a native
shell query recovers the saved catalog's count and digest when the client spills
it. Actual analysis of a Unicode-path fixture delivers full named-schema Evidence
to the next model request. Use `-- chat` for ordinary chat or
`REA_VERIFY_RUNTIME_ROOT` for a production-only installed package. The loopback
OpenAI completions/SSE model is a keyless BYOK endpoint declaring a million-token
context with synthetic usage. A synthetic account-key value satisfies the
client's print-mode gate; native local-only mode refuses hosted API calls.
Updates, telemetry and cron are disabled. Hosted authentication, live models
and Windows remain unverified.
`verify:omp-client` requires installed OMP (verified with the official Linux x64
18.8.7 binary); select it with `REA_VERIFY_OMP_COMMAND`. This optional POSIX lane
isolates the default agent, global config and XDG roots, guards setup targets,
+1
View File
@@ -188,6 +188,7 @@
"verify:gemini-client": "npm run build:cached && node scripts/verify-gemini-client.mjs",
"verify:opencode-client": "npm run build:cached && node scripts/verify-opencode-client.mjs",
"verify:grok-client": "npm run build:cached && node scripts/verify-grok-client.mjs",
"verify:commandcode-client": "npm run build:cached && node scripts/verify-commandcode-client.mjs",
"verify:omp-client": "npm run build:cached && node scripts/verify-omp-client.mjs",
"verify:copilot-client": "npm run build:cached && node scripts/verify-copilot-client.mjs",
"verify:claude-client": "npm run build:cached && node scripts/verify-claude-client.mjs",
+452
View File
@@ -0,0 +1,452 @@
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
import { homedir } from "node:os";
import { isAbsolute, join, relative } from "node:path";
import { fileURLToPath } from "node:url";
import Ajv2020 from "ajv/dist/2020.js";
import {
TOOL_CONTRACTS,
toolContract,
} from "../dist/contracts/toolContracts.js";
import { exec } from "./lib/verify-package-core.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
import { createOpenAiModelFixture } from "./verify/clients/model-fixture.mjs";
assert(
process.platform !== "win32",
"This verifier requires a disposable POSIX account; Windows is unverified",
);
const account = process.env.REA_VERIFY_COMMANDCODE_ACCOUNT_HOME;
assert(
account && isAbsolute(account) && account === homedir(),
"Run as a disposable account whose actual OS home matches REA_VERIFY_COMMANDCODE_ACCOUNT_HOME; do not override HOME",
);
assert.equal(
await readFile(join(account, ".rea-client-verification"), "utf8"),
"Disposable REA client verification account\n",
);
const repo = fileURLToPath(new URL("..", import.meta.url));
const runtimeRoot = process.env.REA_VERIFY_RUNTIME_ROOT ?? repo;
const command = process.env.REA_VERIFY_COMMANDCODE_COMMAND ?? "cmd";
const mode = process.argv[2] ?? "call";
assert(
["chat", "call"].includes(mode),
"Usage: verify-commandcode-client.mjs [chat|call]",
);
const verifier = createVerifierRun();
const lab = await mkdtemp(join(account, "rea-commandcode-client-"));
const workspace = join(lab, "workspace");
const target = join(workspace, "fixture α测试");
const profile = join(account, ".commandcode");
const configPath = join(profile, "mcp.json");
const skillDirectory = join(
account,
".agents",
"skills",
"reverse-engineer-anything",
);
await Promise.all([
mkdir(target, { recursive: true }),
mkdir(profile, { recursive: true }),
mkdir(join(lab, "tmp")),
]);
await exec("git", ["init", "--quiet", workspace], { timeout: 30_000 });
await writeFile(
join(target, "app.js"),
'export function result() { return { clientCompatibility: "REA_CLIENT_FIXTURE", count: 7 }; }\n',
);
const environment = {
PATH: process.env.PATH,
NODE_OPTIONS: process.env.NODE_OPTIONS,
USERPROFILE: account,
TMPDIR: join(lab, "tmp"),
CMD_LOCAL_ONLY: "1",
// Native print mode checks for an account key even with a keyless BYOK model.
// This synthetic value is never a real credential; local-only refuses hosted API calls.
COMMAND_CODE_API_KEY: "local-fixture-only",
COMMANDCODE_SKIP_UPDATES: "1",
COMMANDCODE_DISABLE_CRON: "1",
OTEL_SDK_DISABLED: "true",
REA_PROCESS_RUN_ID: verifier.run_id,
};
const expectedNames = TOOL_CONTRACTS.map(
(tool) => "mcp__rea__" + tool.name,
).sort();
const schemaValidator = new Ajv2020({ strict: false, validateFormats: false });
const textValues = (value) =>
typeof value === "string"
? [value]
: value !== null && typeof value === "object"
? Object.values(value).flatMap(textValues)
: [];
let clientVersion;
let expectedSkill;
let inventoryChecked = false;
let skillLoaded = false;
let schemasChecked = 0;
let catalogArtifact;
let catalogDigest;
let catalogProjection;
let artifactProjectionSeen = false;
let evidenceSeen = false;
let evidenceBytes;
let finalSeen = false;
let stage = 0;
let failure;
const call = (name, args) => ({
role: "assistant",
content: null,
tool_calls: [
{
id: `call_commandcode_${++stage}`,
type: "function",
function: { name, arguments: JSON.stringify(args) },
},
],
});
const validateCatalog = async (text) => {
const entries = [
...text.matchAll(
/^### (mcp__rea__[A-Za-z0-9_]+)\n[\s\S]*?\nParameters:\n```json\n([\s\S]*?)\n```/gm,
),
];
assert.deepEqual(
entries.map((entry) => entry[1]).sort(),
expectedNames,
"Native exact-name search must return every REA input schema",
);
for (const entry of entries) {
const schema = JSON.parse(entry[2]);
assert(
schemaValidator.validateSchema(schema),
`${entry[1]}: invalid native schema: ${JSON.stringify(schemaValidator.errors)}`,
);
}
schemasChecked = entries.length;
catalogDigest = createHash("sha256").update(text).digest("hex");
await writeFile(join(lab, "native-catalog.md"), text);
};
const fixture = await createOpenAiModelFixture({
directory: lab,
prefix: "commandcode",
onRequest: async ({ body, tools }) => {
assert(tools.some((tool) => tool.name === "search_tools"));
const strings = textValues(body.messages);
if (!inventoryChecked) {
const names = [
...new Set(
strings.flatMap(
(text) => text.match(/mcp__rea__[A-Za-z0-9_]+/g) ?? [],
),
),
].sort();
assert.deepEqual(
names,
expectedNames,
"Default native deferred-tool prompt must advertise the complete REA inventory",
);
assert.equal(
tools.filter((tool) => tool.name.startsWith("mcp__rea__")).length,
0,
"Keep native lazy schema delivery enabled",
);
assert(
tools
.find((tool) => tool.name === "activate_skill")
?.parameters.properties.name.enum.includes(
"reverse-engineer-anything",
),
);
inventoryChecked = true;
}
if (mode === "chat") {
assert.equal(
body.messages.filter((message) => message.role === "tool").length,
0,
);
finalSeen = true;
return { role: "assistant", content: "REA_CLIENT_COMPATIBILITY_OK" };
}
if (stage === 0)
return call("activate_skill", { name: "reverse-engineer-anything" });
assert(
strings.some((text) => text.includes(expectedSkill)),
"The model must receive the complete activated skill body",
);
skillLoaded = true;
const last = body.messages
.filter((message) => message.role === "tool")
.at(-1)?.content;
assert.equal(typeof last, "string");
if (stage === 1)
return call("search_tools", {
query: "select:" + expectedNames.join(","),
});
if (stage === 2) {
const spill = last.match(
/\[full output saved to: (.+?) — read it with read_file/,
);
if (spill) {
catalogArtifact = spill[1];
const child = relative(environment.TMPDIR, catalogArtifact);
assert(
isAbsolute(catalogArtifact) &&
child &&
!child.startsWith("..") &&
!isAbsolute(child),
"Native spill must remain inside this run's temporary directory",
);
const text = await readFile(catalogArtifact, "utf8");
await validateCatalog(text);
catalogProjection = { schemas: schemasChecked, sha256: catalogDigest };
const query =
"const fs=require('node:fs'),crypto=require('node:crypto');const text=fs.readFileSync(process.argv[1],'utf8');const schemas=[...text.matchAll(/^### (mcp__rea__[A-Za-z0-9_]+)\\n/gm)].length;console.log(JSON.stringify({rea_catalog:{schemas,sha256:crypto.createHash('sha256').update(text).digest('hex')}}));";
return call("shell_command", {
command: process.execPath,
args: ["-e", query, catalogArtifact],
});
}
await validateCatalog(last);
return call("mcp__rea__analyze_javascript_application", {
input_path: target,
format: "directory",
});
}
if (catalogArtifact && !artifactProjectionSeen) {
const line = last
.split("\n")
.find((item) => item.startsWith('{"rea_catalog":'));
assert(line, "Native shell must return the selected saved-catalog facts");
assert.deepEqual(JSON.parse(line).rea_catalog, catalogProjection);
artifactProjectionSeen = true;
return call("mcp__rea__analyze_javascript_application", {
input_path: target,
format: "directory",
});
}
const evidence = toolContract(
"analyze_javascript_application",
).outputSchema.parse(JSON.parse(last));
assert.equal(evidence.subject.local_path, target);
const shapes = evidence.normalized_result.graph.nodes
.flatMap((node) => node.observations)
.map((observation) => observation.properties)
.filter(
(properties) =>
properties.semantic_role === "export-return-shapes" &&
properties.exported_name === "result",
);
assert(
shapes.length > 0 &&
shapes.every(
(shape) => shape.return_shape_coverage?.projection_complete,
),
);
const fields = shapes.flatMap((shape) =>
shape.static_return_shapes.flatMap((item) => item.fields),
);
assert(
fields.some(
(field) =>
field.path === "/clientCompatibility" &&
field.state === "literal" &&
field.value === "REA_CLIENT_FIXTURE",
),
);
assert(
fields.some(
(field) =>
field.path === "/count" &&
field.state === "literal" &&
field.value === 7,
),
);
evidenceBytes = Buffer.byteLength(last);
await writeFile(join(lab, "native-evidence.json"), last);
evidenceSeen = true;
finalSeen = true;
return { role: "assistant", content: "REA_CLIENT_COMPATIBILITY_OK" };
},
});
await writeFile(
join(profile, "providers.json"),
JSON.stringify({
provider: {
"rea-local": {
baseURL: fixture.baseUrl,
apiKey: false,
models: {
"rea-client-fixture": { contextWindow: 1_000_000, maxOutput: 4096 },
},
},
},
}),
);
const original =
JSON.stringify(
{
mcpServers: { other: { command: "unrelated-server", enabled: false } },
preserved: "keep unrelated settings",
},
null,
2,
) + "\n";
await writeFile(configPath, original);
const rea = (args) =>
exec(
process.execPath,
[join(runtimeRoot, "scripts/rea.mjs"), ...args, "--json"],
{ env: environment, cwd: workspace, timeout: 60_000 },
);
try {
clientVersion = (
await exec(command, ["--version"], {
env: environment,
cwd: workspace,
timeout: 30_000,
})
).stdout.trim();
for (const args of [
["setup", "--client", "commandcode", "--dry-run"],
["setup", "--client", "commandcode", "--yes"],
["doctor", "--client", "commandcode", "--skill"],
]) {
const response = await rea(args);
await writeFile(
join(lab, `${args[0]}${args.includes("--dry-run") ? "-plan" : ""}.json`),
response.stdout,
);
const result = JSON.parse(response.stdout);
if (args[0] === "doctor") assert.equal(result.healthy, true);
else
assert.equal(
result.status,
args.includes("--dry-run") ? "planned" : "ready",
);
if (args.includes("--dry-run")) {
assert(
result.plannedActions.some((action) => action.target === configPath),
);
assert(
result.plannedActions.every(
(action) =>
action.target === configPath ||
action.target === skillDirectory ||
action.target.startsWith(skillDirectory + "/"),
),
"Refuse setup writes outside the disposable account's configuration and skill",
);
}
}
assert.equal(await readFile(`${configPath}.rea.backup`, "utf8"), original);
const configured = await readFile(configPath, "utf8");
assert.equal(JSON.parse(configured).preserved, "keep unrelated settings");
assert.equal(
JSON.parse(configured).mcpServers.other.command,
"unrelated-server",
);
const repeated = JSON.parse(
(await rea(["setup", "--client", "commandcode", "--yes"])).stdout,
);
assert.deepEqual(repeated.appliedActions, []);
assert.equal(await readFile(configPath, "utf8"), configured);
assert.equal(await readFile(`${configPath}.rea.backup`, "utf8"), original);
expectedSkill = (await readFile(join(skillDirectory, "SKILL.md"), "utf8"))
.replace(/^---\r?\n[\s\S]*?\r?\n---\r?\n/, "")
.trim();
const execution = exec(
command,
[
"-p",
mode === "chat"
? "Say hello briefly."
: `Use the REA skill to analyze ${target}.`,
"-m",
"rea-local/rea-client-fixture",
"--local-only",
"--skip-onboarding",
"--trust",
"--output-format",
"json",
"--max-turns",
"8",
"--yolo",
],
{
env: environment,
cwd: workspace,
timeout: 120_000,
maxBuffer: 10 * 1024 * 1024,
},
);
execution.child.stdin.end();
const response = await execution;
await writeFile(join(lab, `${mode}.stdout`), response.stdout);
await writeFile(join(lab, `${mode}.stderr`), response.stderr);
assert.equal(fixture.failure, undefined);
const events = response.stdout
.trim()
.split("\n")
.map((line) => JSON.parse(line));
assert(
events.some(
(event) =>
event.type === "result" &&
event.subtype === "success" &&
event.finalText === "REA_CLIENT_COMPATIBILITY_OK",
),
);
assert(inventoryChecked && finalSeen);
if (mode === "call")
assert(
skillLoaded &&
schemasChecked === expectedNames.length &&
evidenceSeen &&
(!catalogArtifact || artifactProjectionSeen),
);
} catch (cause) {
failure = cause;
if (typeof cause?.stdout === "string")
await writeFile(join(lab, `${mode}.stdout`), cause.stdout);
if (typeof cause?.stderr === "string")
await writeFile(join(lab, `${mode}.stderr`), cause.stderr);
} finally {
await fixture.close();
const receipt = {
client: "Command Code",
clientVersion,
mode,
passed: failure === undefined,
artifacts: lab,
runtimeRoot,
host: {
platform: process.platform,
architecture: process.arch,
node: process.versions.node,
},
modelFixture:
"loopback OpenAI completions/SSE, keyless BYOK, synthetic account-key gate and usage; no hosted authentication or live-model claim",
discovery:
"actual disposable OS account home; native user MCP configuration and default shared skill discovery; Windows unverified",
interface:
"default deferred schemas, complete exact-name search, native saved-catalog query when needed, actual JavaScript MCP call",
inventoryChecked,
catalogSize: inventoryChecked ? expectedNames.length : 0,
schemasChecked,
skillLoaded,
catalogArtifact,
catalogDigest,
artifactProjectionSeen,
evidenceSeen,
evidenceBytes,
finalSeen,
requests: fixture.requests,
probes: fixture.probes,
fixtureFailure: fixture.failure,
verifier: await completeVerifierRun(verifier),
};
await writeFile(join(lab, "receipt.json"), JSON.stringify(receipt, null, 2));
console.log(JSON.stringify(receipt));
}
if (failure !== undefined) throw failure;
+348 -46
View File
@@ -1,6 +1,6 @@
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import { createReadStream, constants as fsConstants } from "node:fs";
import { constants as fsConstants } from "node:fs";
import { access, lstat, mkdir, rm, stat, writeFile } from "node:fs/promises";
import { basename, join, resolve } from "node:path";
@@ -24,8 +24,11 @@ import {
AnalysisTimeoutError,
AnalysisUnsupportedTargetError,
} from "../domain/analysisErrorCore.js";
import type { AnalysisError } from "../domain/analysisErrorBase.js";
import { AnalysisError } from "../domain/analysisErrorBase.js";
import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js";
import { err, ok, type Result } from "../domain/result.js";
import { hashStableFile } from "../filesystem/StableFileHash.js";
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
import {
execFileOutput,
execFileOutputFailure,
@@ -97,11 +100,18 @@ const STDERR_EXCERPT_BYTES = 512;
const UNPARSED_LINES_REPORTED = 10;
const STOP_GRACE_MS = 5_000;
/** Stream a local file digest without retaining large files in memory. */
const sha256File = async (path: string): Promise<string> => {
const hash = createHash("sha256");
for await (const chunk of createReadStream(path)) hash.update(chunk);
return hash.digest("hex");
const hashAdbFile = async (
path: string,
operation: string,
signal: AbortSignal | undefined,
) => {
try {
return await hashStableFile(path, signal);
} catch (cause: unknown) {
if (signal?.aborted === true)
throw new AnalysisCancelledError(operation, { cause });
throw cause;
}
};
/** adb joins shell operands into a remote POSIX command; quote at that boundary. */
@@ -335,6 +345,48 @@ export interface AdbProviderOptions {
readonly environment?: Readonly<Record<string, string | undefined>>;
}
interface PendingPackageDirectoryCleanup {
readonly path: string;
readonly identity: PackageDirectoryIdentity | undefined;
readonly primary: AnalysisError;
readonly remove: () => Promise<void>;
}
interface PackageDirectoryIdentity {
readonly dev: number;
readonly ino: number;
readonly mode: number;
}
const removeOwnedPackageDirectory = async (
path: string,
identity: PackageDirectoryIdentity | undefined,
): Promise<void> => {
if (identity === undefined)
throw new Error(
`Cannot verify the created package directory identity; refusing to remove ${path}`,
);
let current: Awaited<ReturnType<typeof lstat>>;
try {
current = await lstat(path);
} catch (cause: unknown) {
if (cause instanceof Error && Reflect.get(cause, "code") === "ENOENT")
return;
throw cause;
}
if (
!current.isDirectory() ||
current.isSymbolicLink() ||
current.dev !== identity.dev ||
current.ino !== identity.ino ||
current.mode !== identity.mode
)
throw new Error(
`The created package directory was replaced or changed; refusing to remove ${path}`,
);
await rm(path, { recursive: true, force: true });
};
/**
* ADB device inspection and acquisition over a caller-selected adb binary.
*
@@ -346,6 +398,12 @@ export interface AdbProviderOptions {
*/
export class AdbProvider {
private readonly environment: Readonly<Record<string, string | undefined>>;
readonly #pendingPackageDirectoryCleanup =
new Set<PendingPackageDirectoryCleanup>();
readonly #activeOperations = new Set<Promise<void>>();
#closing = false;
#closePromise: Promise<void> | undefined;
#packageCleanupPromise: Promise<void> | undefined;
constructor(options: AdbProviderOptions = {}) {
this.environment = options.environment ?? process.env;
@@ -378,12 +436,121 @@ export class AdbProvider {
}
}
/** The provider holds no long-lived resources; the adb server is caller-owned. */
async close(): Promise<void> {}
/** Retry removal of package directories retained after failed rollback. */
close(): Promise<void> {
this.#closing = true;
if (this.#closePromise !== undefined) return this.#closePromise;
const closing = (async () => {
await Promise.all(this.#activeOperations);
await this.#retryPackageDirectoryCleanup();
})().finally(() => {
if (this.#closePromise === closing) this.#closePromise = undefined;
});
this.#closePromise = closing;
return this.#closePromise;
}
#retryPackageDirectoryCleanup(): Promise<void> {
if (this.#packageCleanupPromise !== undefined)
return this.#packageCleanupPromise;
const cleanup = this.#performPackageDirectoryCleanup().finally(() => {
if (this.#packageCleanupPromise === cleanup)
this.#packageCleanupPromise = undefined;
});
this.#packageCleanupPromise = cleanup;
return cleanup;
}
async #performPackageDirectoryCleanup(): Promise<void> {
const failures: AnalysisError[] = [];
for (const owner of this.#pendingPackageDirectoryCleanup) {
try {
await owner.remove();
this.#pendingPackageDirectoryCleanup.delete(owner);
} catch (cause: unknown) {
const cleanup = new ProviderCleanupError(
ADB_PROVIDER_IDENTITY.id,
[owner.path],
{
reason: cause instanceof Error ? cause.message : String(cause),
},
{ operation: "pull_adb_package", cause },
);
failures.push(
analysisErrorWithCleanupFailure(
owner.primary,
cleanup,
"pull_adb_package",
),
);
}
}
if (failures.length === 1) throw failures[0];
if (failures.length > 1)
throw new ProviderCleanupError(
ADB_PROVIDER_IDENTITY.id,
[...this.#pendingPackageDirectoryCleanup].map(({ path }) => path),
{
reason:
"Failed package output directories remain after cleanup retry",
failures: failures.map(({ message, cleanupResources }) => ({
message,
resources: [...cleanupResources],
})),
},
{
operation: "pull_adb_package",
cause: new AggregateError(
failures,
"Multiple ADB package rollback cleanups failed",
),
},
);
}
async execute(
request: AdbRequest,
options?: ExecutionOptions,
): Promise<Result<AnalysisExecution, AnalysisError>> {
if (this.#closing)
return err(new AnalysisCancelledError(request.operation));
let release!: () => void;
const active = new Promise<void>((resolve) => {
release = resolve;
});
this.#activeOperations.add(active);
try {
if (this.#pendingPackageDirectoryCleanup.size > 0) {
try {
await this.#retryPackageDirectoryCleanup();
} catch (cause: unknown) {
return err(
cause instanceof AnalysisError
? cause
: new ProviderCleanupError(
ADB_PROVIDER_IDENTITY.id,
[...this.#pendingPackageDirectoryCleanup].map(
({ path }) => path,
),
{
reason:
cause instanceof Error ? cause.message : String(cause),
},
{ operation: request.operation, cause },
),
);
}
}
return await this.#executeOperation(request, options);
} finally {
this.#activeOperations.delete(active);
release();
}
}
async #executeOperation(
request: AdbRequest,
options?: ExecutionOptions,
): Promise<Result<AnalysisExecution, AnalysisError>> {
const signal = options?.signal;
if (signal?.aborted === true)
@@ -1010,9 +1177,11 @@ export class AdbProvider {
);
if (!directory.ok) return directory;
const packageDirectory = join(directory.value, packageName);
let packageDirectoryCreated = false;
let packageDirectoryIdentity: PackageDirectoryIdentity | undefined;
try {
if (
await stat(packageDirectory).then(
await lstat(packageDirectory).then(
() => true,
() => false,
)
@@ -1021,14 +1190,44 @@ export class AdbProvider {
`The package output directory already exists: ${packageDirectory}`,
);
await mkdir(packageDirectory);
packageDirectoryCreated = true;
const created = await lstat(packageDirectory);
if (!created.isDirectory() || created.isSymbolicLink())
throw new Error(
`The created package output is not a regular directory: ${packageDirectory}`,
);
packageDirectoryIdentity = {
dev: created.dev,
ino: created.ino,
mode: created.mode,
};
} catch (cause) {
return err(
new AnalysisOutputError(
operation,
cause instanceof Error ? cause.message : String(cause),
{ cause },
),
const primary = new AnalysisOutputError(
operation,
cause instanceof Error ? cause.message : String(cause),
{ cause },
);
if (!packageDirectoryCreated) return err(primary);
const owner: PendingPackageDirectoryCleanup = {
path: packageDirectory,
identity: packageDirectoryIdentity,
primary,
remove: () =>
removeOwnedPackageDirectory(
packageDirectory,
packageDirectoryIdentity,
),
};
this.#pendingPackageDirectoryCleanup.add(owner);
const cleanup = new ProviderCleanupError(
ADB_PROVIDER_IDENTITY.id,
[packageDirectory],
{
reason: "Could not establish an owned package directory identity",
},
{ operation, cause },
);
return err(analysisErrorWithCleanupFailure(primary, cleanup, operation));
}
const artifacts: {
device_path: string;
@@ -1044,6 +1243,37 @@ export class AdbProvider {
stage: "resolve_paths" | "pull" | "digest";
message: string;
}[] = [];
const cancelledPackagePull = (
error: AnalysisCancelledError,
localPath: string,
devicePath: string,
stage: "pull" | "digest",
) => {
error.retainPartialObservation({
provider_id: "adb",
operation,
result: {
client,
serial,
package_name: packageName,
output_directory: directory.value,
artifacts,
failures: [
...failures,
{
device_path: devicePath,
stage,
message:
stage === "pull"
? `Pull cancelled; any incomplete file is retained at ${localPath}. Remaining APKs were not attempted.`
: `Digesting the pulled file was cancelled; the file is retained at ${localPath}. Remaining APKs were not attempted.`,
},
],
coverage: "partial",
},
});
return err(error);
};
const seenFileNames = new Set<string>();
for (const entry of parsed.paths) {
if (seenFileNames.has(entry.file_name)) {
@@ -1065,7 +1295,13 @@ export class AdbProvider {
signal,
);
if (!pulled.ok) {
if (pulled.error instanceof AnalysisCancelledError) return pulled;
if (pulled.error instanceof AnalysisCancelledError)
return cancelledPackagePull(
pulled.error,
localPath,
entry.device_path,
"pull",
);
failures.push({
device_path: entry.device_path,
stage: "pull",
@@ -1074,18 +1310,24 @@ export class AdbProvider {
continue;
}
try {
const info = await stat(localPath);
const sha256 = await sha256File(localPath);
const digest = await hashAdbFile(localPath, operation, signal);
artifacts.push({
device_path: entry.device_path,
file_name: entry.file_name,
local_path: localPath,
bytes: info.size,
sha256,
bytes: digest.bytes,
sha256: digest.sha256,
role: entry.role,
role_basis: "file_name",
});
} catch {
} catch (cause: unknown) {
if (cause instanceof AnalysisCancelledError)
return cancelledPackagePull(
cause,
localPath,
entry.device_path,
"digest",
);
failures.push({
device_path: entry.device_path,
stage: "digest",
@@ -1094,14 +1336,36 @@ export class AdbProvider {
}
}
if (artifacts.length === 0 && failures.length > 0) {
await rm(packageDirectory, { recursive: true, force: true }).catch(
() => undefined,
);
return err(
new AnalysisProtocolError(
`Every APK pull failed for ${packageName} on ${serial}: ${failures[0]!.message}`,
),
const primary = new AnalysisProtocolError(
`Every APK pull failed for ${packageName} on ${serial}: ${failures[0]!.message}`,
);
const owner: PendingPackageDirectoryCleanup = {
path: packageDirectory,
identity: packageDirectoryIdentity,
primary,
remove: () =>
removeOwnedPackageDirectory(
packageDirectory,
packageDirectoryIdentity,
),
};
try {
await owner.remove();
} catch (cause: unknown) {
this.#pendingPackageDirectoryCleanup.add(owner);
const cleanup = new ProviderCleanupError(
ADB_PROVIDER_IDENTITY.id,
[packageDirectory],
{
reason: cause instanceof Error ? cause.message : String(cause),
},
{ operation: operation, cause },
);
return err(
analysisErrorWithCleanupFailure(primary, cleanup, operation),
);
}
return err(primary);
}
return ok(
createAnalysisExecution(
@@ -1156,8 +1420,7 @@ export class AdbProvider {
);
if (!pulled.ok) return pulled;
try {
const info = await stat(localPath);
const sha256 = await sha256File(localPath);
const digest = await hashAdbFile(localPath, operation, signal);
return ok(
createAnalysisExecution(
{
@@ -1165,8 +1428,8 @@ export class AdbProvider {
serial: input.serial,
device_path: input.device_path,
local_path: localPath,
bytes: info.size,
sha256,
bytes: digest.bytes,
sha256: digest.sha256,
},
provider,
{
@@ -1177,6 +1440,20 @@ export class AdbProvider {
),
);
} catch (cause) {
if (cause instanceof AnalysisCancelledError) {
cause.retainPartialObservation({
provider_id: "adb",
operation,
result: {
client,
serial: input.serial,
local_path: localPath,
device_path: input.device_path,
digest_status: "unknown",
},
});
return err(cause);
}
return err(
new AnalysisOutputError(
operation,
@@ -1211,7 +1488,13 @@ export class AdbProvider {
cause: new Error(`The local file does not exist: ${localPath}`),
}),
);
const sha256 = await sha256File(localPath);
let digest: Awaited<ReturnType<typeof hashAdbFile>>;
try {
digest = await hashAdbFile(localPath, operation, signal);
} catch (cause: unknown) {
if (cause instanceof AnalysisCancelledError) return err(cause);
throw cause;
}
const existsProbe = await runAdb(
operation,
input.device_path,
@@ -1257,7 +1540,7 @@ export class AdbProvider {
);
if (deviceDigest.ok) {
const digestMatch = /^([a-f0-9]{64})\s/u.exec(deviceDigest.value.stdout);
if (digestMatch !== null && digestMatch[1] !== sha256)
if (digestMatch !== null && digestMatch[1] !== digest.sha256)
return err(
new AnalysisArtifactChangedError(
operation,
@@ -1272,8 +1555,8 @@ export class AdbProvider {
serial: input.serial,
local_path: localPath,
device_path: input.device_path,
bytes: info.size,
sha256,
bytes: digest.bytes,
sha256: digest.sha256,
device_sha256: digestMatch?.[1] ?? null,
overwritten: deviceExists,
device_digest_source: "device_sha256sum",
@@ -1294,8 +1577,8 @@ export class AdbProvider {
serial: input.serial,
local_path: localPath,
device_path: input.device_path,
bytes: info.size,
sha256,
bytes: digest.bytes,
sha256: digest.sha256,
device_sha256: null,
overwritten: deviceExists,
device_digest_source: "unavailable",
@@ -1334,7 +1617,13 @@ export class AdbProvider {
cause: new Error(`The local APK does not exist: ${apkPath}`),
}),
);
const sha256 = await sha256File(apkPath);
let digest: Awaited<ReturnType<typeof hashAdbFile>>;
try {
digest = await hashAdbFile(apkPath, operation, signal);
} catch (cause: unknown) {
if (cause instanceof AnalysisCancelledError) return err(cause);
throw cause;
}
const installed = await runAdb(
operation,
input.serial,
@@ -1363,8 +1652,8 @@ export class AdbProvider {
client,
serial: input.serial,
apk_path: apkPath,
bytes: info.size,
sha256,
bytes: digest.bytes,
sha256: digest.sha256,
replaced: input.replace,
},
provider,
@@ -1734,16 +2023,15 @@ export class AdbProvider {
),
);
try {
const info = await stat(reportedPath);
const sha256 = await sha256File(reportedPath);
const digest = await hashAdbFile(reportedPath, operation, signal);
return ok(
createAnalysisExecution(
{
client,
serial: input.serial,
local_path: reportedPath,
bytes: info.size,
sha256,
bytes: digest.bytes,
sha256: digest.sha256,
adb_reported_path: reportedPath,
},
provider,
@@ -1755,6 +2043,20 @@ export class AdbProvider {
),
);
} catch (cause) {
if (cause instanceof AnalysisCancelledError) {
cause.retainPartialObservation({
provider_id: "adb",
operation,
result: {
client,
serial: input.serial,
local_path: reportedPath,
adb_reported_path: reportedPath,
digest_status: "unknown",
},
});
return err(cause);
}
return err(
new AnalysisOutputError(
operation,
+129 -21
View File
@@ -1,40 +1,43 @@
import { constants, createReadStream } from "node:fs";
import { chmod, copyFile } from "node:fs/promises";
import { createHash } from "node:crypto";
import type { ReadStream, Stats, WriteStream } from "node:fs";
import { chmod, open, rm, stat, type FileHandle } from "node:fs/promises";
import { join } from "node:path";
import { Transform, type TransformCallback } from "node:stream";
import { pipeline } from "node:stream/promises";
import {
AnalysisInputError,
AnalysisCapabilityUnavailableError,
} from "../domain/analysisErrorCore.js";
import {
openRegularFile,
sameRegularFileState,
} from "../filesystem/RegularFile.js";
import type { AndroidOperation } from "../domain/android/androidAnalysis.js";
/** Stream a local file digest without retaining the APK or JAR in memory. */
export const hashAndroidFile = async (path: string): Promise<string> => {
const hash = createHash("sha256");
for await (const chunk of createReadStream(path)) hash.update(chunk);
return hash.digest("hex");
};
/** Bind decompiler input to the admitted artifact, even if the original changes. */
export const snapshotAndroidTarget = async (
path: string,
sha256: string,
root: string,
operation: AndroidOperation,
signal?: AbortSignal,
): Promise<string> => {
const snapshot = join(root, "target.apk");
await copyFile(path, snapshot, constants.COPYFILE_EXCL);
await chmod(snapshot, 0o400);
if ((await hashAndroidFile(snapshot)) !== sha256)
throw new AnalysisInputError(operation, undefined, [
try {
await copyAndroidSnapshot(path, snapshot, sha256, signal);
signal?.throwIfAborted();
await chmod(snapshot, 0o400);
return snapshot;
} catch (cause: unknown) {
if (!(cause instanceof AndroidSnapshotChangedError)) throw cause;
throw new AnalysisInputError(operation, { cause }, [
{
path: ["path"],
reason: "invalid_value",
message:
"APK bytes changed after admission; retry against a stable file.",
message: `APK bytes changed after admission at ${path}; retry against a stable file.`,
},
]);
return snapshot;
}
};
/** Fingerprint the actual immutable engine bytes executed for this observation. */
@@ -43,15 +46,120 @@ export const snapshotAndroidEngine = async (
sha256: string,
root: string,
operation: AndroidOperation,
signal?: AbortSignal,
): Promise<{ path: string; sha256: string }> => {
const path = join(root, "engine.jar");
await copyFile(source, path, constants.COPYFILE_EXCL);
await chmod(path, 0o400);
if ((await hashAndroidFile(path)) !== sha256)
try {
const observed = await copyAndroidSnapshot(source, path, sha256, signal);
signal?.throwIfAborted();
await chmod(path, 0o400);
return { path, sha256: observed };
} catch (cause: unknown) {
if (!(cause instanceof AndroidSnapshotChangedError)) throw cause;
throw new AnalysisCapabilityUnavailableError(
"jadx",
operation,
"REA_JADX_MCP_JAR bytes changed during admission; retry with a stable engine file.",
`REA_JADX_MCP_JAR bytes changed during admission at ${source}; retry with a stable engine file.`,
{ cause },
);
return { path, sha256 };
}
};
const copyAndroidSnapshot = async (
sourcePath: string,
snapshotPath: string,
expectedSha256: string,
signal?: AbortSignal,
): Promise<string> => {
signal?.throwIfAborted();
const source = await openRegularFile(sourcePath, {
symlinks: "follow",
signal,
});
let destination: FileHandle | undefined;
let inputStream: ReadStream | undefined;
let outputStream: WriteStream | undefined;
let removeSnapshot = false;
let copiedBytes = 0;
const hash = createHash("sha256");
try {
const initial = await source.stat();
signal?.throwIfAborted();
const digest = new Transform({
transform(
chunk: Buffer,
_encoding: BufferEncoding,
callback: TransformCallback,
) {
copiedBytes += chunk.length;
if (copiedBytes > initial.size) {
callback(
new AndroidSnapshotChangedError(
`Android input grew while being copied: ${sourcePath}`,
),
);
return;
}
hash.update(chunk);
callback(null, chunk);
},
});
try {
destination = await open(snapshotPath, "wx", 0o600);
signal?.throwIfAborted();
inputStream = source.createReadStream({
start: 0,
end: initial.size,
autoClose: false,
});
outputStream = destination.createWriteStream({ autoClose: false });
await pipeline(inputStream, digest, outputStream, { signal });
signal?.throwIfAborted();
await verifySourceState(source, sourcePath, initial, copiedBytes);
const observedSha256 = hash.digest("hex");
if (observedSha256 !== expectedSha256)
throw new AndroidSnapshotChangedError(
`Android snapshot digest mismatch for ${sourcePath}`,
);
return observedSha256;
} catch (cause: unknown) {
removeSnapshot = destination !== undefined;
throw cause;
}
} finally {
// Non-auto-closing streams retain a reference to their FileHandle even
// after pipeline completion. Release those references before awaiting
// descriptor closure, after the admitted source state has been verified.
inputStream?.destroy();
outputStream?.destroy();
try {
await Promise.all([source.close(), destination?.close()]);
} finally {
if (removeSnapshot)
await rm(snapshotPath, { force: true }).catch(() => undefined);
}
}
};
const verifySourceState = async (
handle: Awaited<ReturnType<typeof openRegularFile>>,
path: string,
initial: Stats,
observedBytes: number,
): Promise<void> => {
const [opened, currentPath] = await Promise.all([handle.stat(), stat(path)]);
if (
observedBytes !== initial.size ||
!sameRegularFileState(initial, opened) ||
!sameRegularFileState(initial, currentPath)
)
throw new AndroidSnapshotChangedError(
`Android input changed while being read: ${path}`,
);
};
class AndroidSnapshotChangedError extends Error {
constructor(message: string, options?: ErrorOptions) {
super(message, options);
}
}
+64 -29
View File
@@ -52,7 +52,11 @@ export const inspectJadxAvailability = async (
signal?: AbortSignal,
): Promise<ProviderAvailability> => {
try {
const configuration = await readJadxConfiguration(environment, signal);
const configuration = await readJadxConfigurationInputs(
environment,
signal,
);
await validateJadxConfigurationContents(configuration, environment, signal);
return {
status: "available",
code: null,
@@ -77,52 +81,83 @@ export const inspectJadxAvailability = async (
}
};
/** Resolve caller-selected tools and preserve the same prerequisite diagnostics. */
export const resolveJadxConfiguration = async (
/** Resolve caller-selected paths and JVM options without probing their contents. */
export const resolveJadxConfigurationInputs = async (
environment: Readonly<Record<string, string | undefined>>,
operation: AndroidOperation,
signal?: AbortSignal,
): Promise<JadxConfiguration> => {
try {
return await readJadxConfiguration(environment, signal);
return await readJadxConfigurationInputs(environment, signal);
} catch (cause) {
if (signal?.aborted === true) throw cause;
if (!(cause instanceof JadxConfigurationFailure)) throw cause;
const output = cause.diagnostics;
throw new AnalysisCapabilityUnavailableError(
"jadx",
operation,
cause.message,
{
cause,
userMessage: cause.message,
...(typeof output.stdout === "string" &&
typeof output.stderr === "string"
? {
capturedOutput: {
stdout: output.stdout,
stderr: output.stderr,
truncated: output.output_truncated === true,
},
}
: {}),
},
);
throw projectConfigurationFailure(operation, cause);
}
};
const readJadxConfiguration = async (
/** Probe Java and archive prerequisites before starting a new engine session. */
export const validateJadxConfiguration = async (
configuration: JadxConfiguration,
environment: Readonly<Record<string, string | undefined>>,
operation: AndroidOperation,
signal?: AbortSignal,
): Promise<void> => {
try {
await validateJadxConfigurationContents(configuration, environment, signal);
} catch (cause) {
if (signal?.aborted === true) throw cause;
if (!(cause instanceof JadxConfigurationFailure)) throw cause;
throw projectConfigurationFailure(operation, cause);
}
};
const readJadxConfigurationInputs = async (
environment: Readonly<Record<string, string | undefined>>,
signal?: AbortSignal,
): Promise<JadxConfiguration> => {
signal?.throwIfAborted();
requireSupportedHost();
const jar = await resolveJar(environment);
const java = await resolveJava(environment, signal);
await requireJavaHomeExecutable(environment.JAVA_HOME, java);
const jvmArguments = resolveJvmArguments(environment);
await inspectJavaRuntime(java, environment, signal);
await requireEngineArchive(jar, signal);
return { jar, java, jvmArguments };
signal?.throwIfAborted();
return { jar, java, jvmArguments: resolveJvmArguments(environment) };
};
const validateJadxConfigurationContents = async (
configuration: JadxConfiguration,
environment: Readonly<Record<string, string | undefined>>,
signal?: AbortSignal,
): Promise<void> => {
signal?.throwIfAborted();
await inspectJavaRuntime(configuration.java, environment, signal);
await requireEngineArchive(configuration.jar, signal);
};
const projectConfigurationFailure = (
operation: AndroidOperation,
cause: JadxConfigurationFailure,
): AnalysisCapabilityUnavailableError => {
const output = cause.diagnostics;
return new AnalysisCapabilityUnavailableError(
"jadx",
operation,
cause.message,
{
cause,
userMessage: cause.message,
...(typeof output.stdout === "string" && typeof output.stderr === "string"
? {
capturedOutput: {
stdout: output.stdout,
stderr: output.stderr,
truncated: output.output_truncated === true,
},
}
: {}),
},
);
};
const requireEngineArchive = async (
+98 -28
View File
@@ -21,12 +21,14 @@ import { err, ok, type Result } from "../domain/result.js";
import { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js";
import {
snapshotAndroidEngine,
hashAndroidFile,
snapshotAndroidTarget,
} from "./AndroidTargetSnapshot.js";
import { hashStableFile } from "../filesystem/StableFileHash.js";
import { RegularFileChangedError } from "../filesystem/RegularFile.js";
import {
inspectJadxAvailability,
resolveJadxConfiguration,
resolveJadxConfigurationInputs,
validateJadxConfiguration,
} from "./JadxConfiguration.js";
import type { JadxLauncher } from "./JadxMcpTransport.js";
@@ -104,6 +106,50 @@ const executionError = (context: {
});
};
const androidTargetChanged = (
operation: string,
path: string,
cause?: unknown,
): AnalysisInputError =>
new AnalysisInputError(
operation,
cause === undefined ? undefined : { cause },
[
{
path: ["path"],
reason: "invalid_value",
message: `APK bytes changed after admission at ${path}; retry against a stable file.`,
},
],
);
const androidSourceChanged = (
operation: string,
source: string,
cause?: unknown,
): AnalysisCapabilityUnavailableError =>
new AnalysisCapabilityUnavailableError(
"jadx",
operation,
`Selected JADX input bytes changed during admission at ${source}; retry with a stable installation.`,
cause === undefined ? undefined : { cause },
);
const hashAndroidSource = async (
path: string,
operation: string,
signal: AbortSignal,
): Promise<string> => {
try {
return (await hashStableFile(path, signal)).sha256;
} catch (cause: unknown) {
if (signal.aborted) throw cause;
if (cause instanceof RegularFileChangedError)
throw androidSourceChanged(operation, path, cause);
throw cause;
}
};
interface PendingCleanup {
session: JadxSession | undefined;
root: PrivateRuntimeRoot | undefined;
@@ -147,17 +193,18 @@ const cleanup = async (
await root?.close();
resources.root = undefined;
} catch (cause) {
return err(
new ProviderCleanupError(
"jadx",
[root?.path ?? "unknown"],
{
...diagnostics,
reason: cause instanceof Error ? cause.message : String(cause),
},
{ cause: previousError ?? cause },
),
const error = new ProviderCleanupError(
"jadx",
[root?.path ?? "unknown"],
{
...diagnostics,
reason: cause instanceof Error ? cause.message : String(cause),
},
{ cause: previousError ?? cause },
);
if (previousError?.partialObservation !== undefined)
error.retainPartialObservation(previousError.partialObservation);
return err(error);
}
return ok(undefined);
};
@@ -287,23 +334,33 @@ export class JadxProvider implements AndroidAnalysisPort {
request.operation,
`Target ${target.path} is ${target.format}; provide one standalone APK.`,
);
const configuration = await resolveJadxConfiguration(
const configuration = await resolveJadxConfigurationInputs(
this.environment,
request.operation,
signal,
);
signal.throwIfAborted();
const jarHash = await hashAndroidFile(configuration.jar);
const bridgeHash = await hashAndroidFile(BRIDGE_SOURCE);
if ((await hashAndroidFile(target.path)) !== target.sha256)
throw new AnalysisInputError(request.operation, undefined, [
{
path: ["path"],
reason: "invalid_value",
message:
"APK bytes changed after admission; retry against a stable file.",
},
]);
const jarHash = await hashAndroidSource(
configuration.jar,
request.operation,
signal,
);
const bridgeHash = await hashAndroidSource(
BRIDGE_SOURCE,
request.operation,
signal,
);
let targetHash: string;
try {
targetHash = (await hashStableFile(target.path, signal)).sha256;
} catch (cause: unknown) {
if (signal.aborted) throw cause;
if (cause instanceof RegularFileChangedError)
throw androidTargetChanged(request.operation, target.path, cause);
throw cause;
}
if (targetHash !== target.sha256)
throw androidTargetChanged(request.operation, target.path);
const key = JSON.stringify({
path: target.path,
sha256: target.sha256,
@@ -326,27 +383,40 @@ export class JadxProvider implements AndroidAnalysisPort {
}
let retained = this.#retained;
if (retained === undefined) {
await validateJadxConfiguration(
configuration,
this.environment,
request.operation,
signal,
);
root = await PrivateRuntimeRoot.create({ prefix: "rea-android-" });
const engine = await snapshotAndroidEngine(
configuration.jar,
jarHash,
root.path,
request.operation,
signal,
);
const snapshot = await snapshotAndroidTarget(
target.path,
target.sha256,
root.path,
request.operation,
signal,
);
const bridge = join(root.path, "ReaJadxBridge.java");
await copyFile(BRIDGE_SOURCE, bridge);
await chmod(bridge, 0o400);
if ((await hashAndroidFile(bridge)) !== bridgeHash)
throw new AnalysisCapabilityUnavailableError(
"jadx",
const copiedBridgeHash = await hashAndroidSource(
bridge,
request.operation,
signal,
);
if (copiedBridgeHash !== bridgeHash)
throw androidSourceChanged(
request.operation,
"REA Android bridge bytes changed during admission; retry with a stable installation.",
BRIDGE_SOURCE,
undefined,
);
signal.throwIfAborted();
session = new JadxSession(
+2
View File
@@ -78,12 +78,14 @@ export const parseApktoolVersionOutput = (
export const inspectApktoolClient = async (
selection: ApktoolCommandSelection,
signal?: AbortSignal,
environment: Readonly<Record<string, string | undefined>> = process.env,
): Promise<ApktoolClientIdentity> => {
try {
const { stdout } = await execFileOutput(selection.command, ["--version"], {
timeout: VERSION_TIMEOUT_MS,
stopSignal: "SIGTERM",
maxBuffer: VERSION_MAX_OUTPUT_BYTES,
env: { ...process.env, ...environment },
...(signal === undefined ? {} : { signal }),
});
return parseApktoolVersionOutput(stdout);
+327 -133
View File
@@ -1,18 +1,20 @@
import { createHash } from "node:crypto";
import { createReadStream, constants as fsConstants } from "node:fs";
import { createHash, randomUUID } from "node:crypto";
import { createWriteStream, constants as fsConstants } from "node:fs";
import {
access,
chmod,
copyFile,
mkdir,
mkdtemp,
opendir,
open,
readdir,
rm,
stat,
} from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { Transform, type TransformCallback } from "node:stream";
import { pipeline } from "node:stream/promises";
import type { FileHandle } from "node:fs/promises";
import type {
ExecutionOptions,
@@ -33,11 +35,19 @@ import {
AnalysisUnsupportedTargetError,
} from "../domain/analysisErrorCore.js";
import type { AnalysisError } from "../domain/analysisErrorBase.js";
import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js";
import { err, ok, type Result } from "../domain/result.js";
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
import {
execFileOutput,
execFileOutputFailure,
} from "../process/ExecFileOutput.js";
openRegularFile,
sameRegularFileState,
} from "../filesystem/RegularFile.js";
import {
OwnedCommandFailure,
runOwnedCommand,
} from "../process/OwnedCommand.js";
import type { ProviderProcessSupervisor } from "../process/ProviderProcess.js";
import { snapshotEnvironment } from "../process/snapshotEnvironment.js";
import {
ApktoolConfigurationFailure,
apktoolConfigurationAvailability,
@@ -64,20 +74,22 @@ const DECODE_MAX_OUTPUT_BYTES = 4 * 1024 * 1024;
const MANIFEST_MAX_BYTES = 512 * 1024;
const STRINGS_XML_MAX_BYTES = 8 * 1024 * 1024;
const MAX_STRINGS = 2_000;
const MAX_DECODED_FILES_COUNTED = 100_000;
const MAX_DECODED_ENTRIES_VISITED = 100_000;
const STDERR_EXCERPT_BYTES = 512;
/** Stream a local file digest without retaining large APKs in memory. */
const sha256File = async (path: string): Promise<string> => {
const hash = createHash("sha256");
for await (const chunk of createReadStream(path)) hash.update(chunk);
return hash.digest("hex");
const throwIfCancelled = (
operation: string,
signal: AbortSignal | undefined,
): void => {
if (signal?.aborted === true) throw new AnalysisCancelledError(operation);
};
/** Read at most the byte budget plus one byte, before decoding UTF-8. */
const readBoundedText = async (
path: string,
budget: number,
operation: string,
signal: AbortSignal | undefined,
): Promise<{ text: string | null; truncated: boolean }> => {
const file = await open(path, "r").catch((cause: unknown) => {
if ((cause as NodeJS.ErrnoException).code === "ENOENT") return null;
@@ -85,9 +97,11 @@ const readBoundedText = async (
});
if (file === null) return { text: null, truncated: false };
try {
throwIfCancelled(operation, signal);
const bytes = Buffer.alloc(budget + 1);
let length = 0;
while (length < bytes.length) {
throwIfCancelled(operation, signal);
const result = await file.read(
bytes,
length,
@@ -97,6 +111,7 @@ const readBoundedText = async (
if (result.bytesRead === 0) break;
length += result.bytesRead;
}
throwIfCancelled(operation, signal);
const truncated = length > budget;
// A prefix may end within a UTF-8 character; omit those trailing bytes.
const decoder = new TextDecoder("utf-8");
@@ -125,97 +140,141 @@ const runApktool = async (
arguments_: readonly string[],
limits: DecodeLimits,
signal: AbortSignal | undefined,
environment: Readonly<Record<string, string | undefined>>,
onCleanupOwner: (owner: ProviderProcessSupervisor) => void,
): Promise<Result<{ stdout: string; stderr: string }, AnalysisError>> => {
if (signal?.aborted === true)
return err(new AnalysisCancelledError(operation));
try {
return ok(
await execFileOutput(command, arguments_, {
timeout: limits.timeoutMs,
maxBuffer: limits.maxOutputBytes,
stopSignal: "SIGTERM",
...(signal === undefined ? {} : { signal }),
}),
const output = await runOwnedCommand(
{
command,
arguments: arguments_,
runId: randomUUID(),
hostEnvironment: { ...process.env, ...environment },
},
{
timeoutMs: limits.timeoutMs,
diagnosticBytes: limits.maxOutputBytes,
},
signal === undefined ? {} : { signal },
);
return ok({ stdout: output.stdout.text, stderr: output.stderr.text });
} catch (cause) {
if (signal?.aborted || execFileOutputFailure(cause)?.code === "ABORT_ERR")
return err(new AnalysisCancelledError(operation, { cause }));
const failure = execFileOutputFailure(cause);
if (failure === undefined)
if (!(cause instanceof OwnedCommandFailure))
return err(
new AnalysisProtocolError(`apktool failed for ${operation}`, { cause }),
);
if (failure.outputTruncated)
if (cause.cleanupOwner !== undefined) onCleanupOwner(cause.cleanupOwner);
const output = cause.snapshot;
const capturedOutput =
output === null
? undefined
: {
stdout: output.stdout.text,
stderr: output.stderr.text,
truncated: output.diagnosticTruncated === true,
stdout_bytes: output.stdout.bytes,
stderr_bytes: output.stderr.bytes,
exit_code: output.exitCode ?? null,
signal: output.signal ?? null,
};
const cleanup =
cause.cleanupFailure === null
? undefined
: {
reason: cause.cleanupFailure,
resources: cause.resources,
};
if (cause.reason === "cancelled")
return err(
new AnalysisCancelledError(operation, {
cause,
...(cleanup === undefined ? {} : { cleanup }),
...(capturedOutput === undefined ? {} : { capturedOutput }),
}),
);
if (cause.reason === "timeout")
return err(
new AnalysisTimeoutError(operation, limits.timeoutMs, {
cause,
...(cleanup === undefined ? {} : { cleanup }),
...(capturedOutput === undefined ? {} : { capturedOutput }),
}),
);
if (cause.reason === "output-limit")
return err(
new AnalysisResourceConstraintError(
operation,
"memory",
`apktool output exceeded the ${String(limits.maxOutputBytes)}-byte capture budget`,
{ max_output_bytes: limits.maxOutputBytes },
{ cause },
{
cause,
...(cleanup === undefined ? {} : { cleanup }),
...(capturedOutput === undefined ? {} : { capturedOutput }),
},
),
);
const excerpt = failure.stderr.trim().slice(0, STDERR_EXCERPT_BYTES);
if (failure.killed)
return err(
new AnalysisTimeoutError(operation, limits.timeoutMs, { cause }),
);
return err(
new AnalysisUnsupportedTargetError(
operation,
command,
excerpt === ""
? `apktool exited with code ${String(failure.code)} for ${operation}`
: excerpt,
{ cause },
output?.stderr.text.trim().slice(0, STDERR_EXCERPT_BYTES) ||
`apktool exited with code ${String(output?.exitCode ?? "unknown")} for ${operation}`,
{
cause,
...(cleanup === undefined ? {} : { cleanup }),
...(capturedOutput === undefined ? {} : { capturedOutput }),
},
),
);
}
};
/** Count files in the decoded workspace with a hard ceiling. */
/** Count files while bounding streamed directory-entry traversal. */
const countDecodedFiles = async (
root: string,
): Promise<{ count: number; capped: boolean }> => {
signal: AbortSignal | undefined,
operation: string,
): Promise<{
count: number;
capped: boolean;
locales: readonly string[];
}> => {
let count = 0;
let capped = false;
const walk = async (directory: string): Promise<void> => {
const entries = await readdir(directory, { withFileTypes: true });
for (const entry of entries) {
if (count >= MAX_DECODED_FILES_COUNTED) {
capped = true;
return;
}
const path = join(directory, entry.name);
if (entry.isDirectory()) await walk(path);
else if (entry.isFile()) count += 1;
}
};
await walk(root);
return { count, capped };
};
/** Locale codes present as res/values-<locale>/strings.xml. */
const listStringLocales = async (
workspace: string,
): Promise<readonly string[]> => {
const res = join(workspace, "res");
const entries = await readdir(res, { withFileTypes: true }).catch(() => []);
let visited = 0;
const directories = [root];
const locales: string[] = [];
for (const entry of entries) {
if (!entry.isDirectory()) continue;
const match = /^values-(.+)$/u.exec(entry.name);
if (match === null) continue;
const strings = join(res, entry.name, "strings.xml");
if (
await access(strings).then(
() => true,
() => false,
)
)
locales.push(match[1]!);
while (directories.length > 0 && !capped) {
throwIfCancelled(operation, signal);
const directory = directories.pop()!;
for await (const entry of await opendir(directory)) {
throwIfCancelled(operation, signal);
if (visited >= MAX_DECODED_ENTRIES_VISITED) {
capped = true;
break;
}
visited += 1;
const path = join(directory, entry.name);
if (entry.isDirectory()) {
directories.push(path);
if (directory === join(root, "res")) {
const match = /^values-(.+)$/u.exec(entry.name);
if (
match !== null &&
(await access(join(path, "strings.xml")).then(
() => true,
() => false,
))
)
locales.push(match[1]!);
}
} else if (entry.isFile()) count += 1;
}
}
return locales.sort();
return { count, capped, locales: locales.sort() };
};
export interface ApktoolProviderOptions {
@@ -235,9 +294,14 @@ export class ApktoolProvider {
private readonly workspaces = new Set<string>();
private readonly removeWorkspace: (path: string) => Promise<void>;
readonly #pendingCommandOwners = new Map<ProviderProcessSupervisor, string>();
readonly #shutdown = new AbortController();
readonly #active = new Set<Promise<void>>();
#closing = false;
#closePromise: Promise<void> | undefined;
constructor(options: ApktoolProviderOptions = {}) {
this.environment = options.environment ?? process.env;
this.environment = snapshotEnvironment(options.environment ?? process.env);
this.removeWorkspace =
options.removeWorkspace ??
((path) => rm(path, { recursive: true, force: true }));
@@ -248,12 +312,25 @@ export class ApktoolProvider {
}
/** Probe the selected apktool launcher; decodes nothing. */
async inspectAvailability(
signal?: AbortSignal,
inspectAvailability(signal?: AbortSignal): Promise<ProviderAvailability> {
if (this.#closing)
return Promise.reject(
new AnalysisCancelledError("inspect_apktool_client"),
);
const operationSignal =
signal === undefined
? this.#shutdown.signal
: AbortSignal.any([signal, this.#shutdown.signal]);
return this.#track(this.#inspectAvailability(operationSignal));
}
async #inspectAvailability(
signal: AbortSignal,
): Promise<ProviderAvailability> {
const selection = this.selection();
try {
await inspectApktoolClient(selection, signal);
await inspectApktoolClient(selection, signal, this.environment);
throwIfCancelled("inspect_apktool_client", signal);
return {
status: "available",
code: null,
@@ -272,30 +349,44 @@ export class ApktoolProvider {
/** Retain failed cleanup ownership and retry it on close. */
async close(): Promise<void> {
const failures: unknown[] = [];
for (const path of this.workspaces) {
try {
await this.removeWorkspace(path);
this.workspaces.delete(path);
} catch (cause) {
failures.push(cause);
}
}
if (failures.length > 0)
throw new AggregateError(failures, "Apktool workspace cleanup failed");
this.#closing = true;
this.#shutdown.abort();
this.#closePromise ??= this.#drainAndClean().catch((cause: unknown) => {
this.#closePromise = undefined;
throw cause;
});
return this.#closePromise;
}
async execute(
execute(
request: ApktoolRequest,
options?: ExecutionOptions,
): Promise<Result<AnalysisExecution, AnalysisError>> {
const signal = options?.signal;
if (this.#closing)
return Promise.resolve(
err(new AnalysisCancelledError(request.operation)),
);
const signal =
options?.signal === undefined
? this.#shutdown.signal
: AbortSignal.any([options.signal, this.#shutdown.signal]);
return this.#track(this.#execute(request, signal));
}
async #execute(
request: ApktoolRequest,
signal: AbortSignal,
): Promise<Result<AnalysisExecution, AnalysisError>> {
if (signal?.aborted === true)
return err(new AnalysisCancelledError(request.operation));
const selection = this.selection();
let identity: ApktoolClientIdentity;
try {
identity = await inspectApktoolClient(selection, signal);
identity = await inspectApktoolClient(
selection,
signal,
this.environment,
);
} catch (cause) {
if (cause instanceof AnalysisCancelledError) return err(cause);
if (cause instanceof ApktoolConfigurationFailure)
@@ -316,6 +407,8 @@ export class ApktoolProvider {
),
);
}
if (signal.aborted)
return err(new AnalysisCancelledError(request.operation));
const client = {
command: selection.command,
command_source: selection.commandSource,
@@ -337,6 +430,47 @@ export class ApktoolProvider {
);
}
#track<T>(operation: Promise<T>): Promise<T> {
const settled = operation.then(
() => undefined,
() => undefined,
);
this.#active.add(settled);
void settled.then(() => this.#active.delete(settled));
return operation;
}
async #drainAndClean(): Promise<void> {
await Promise.all(this.#active);
const failures: unknown[] = [];
for (const [owner, root] of this.#pendingCommandOwners) {
try {
const stopped = await owner.stop();
if (stopped.status === "incomplete") throw new Error(stopped.reason);
this.#pendingCommandOwners.delete(owner);
} catch (cause) {
failures.push(
new Error(
`apktool process cleanup remains incomplete for ${root}: ${cause instanceof Error ? cause.message : String(cause)}`,
{ cause },
),
);
}
}
if (failures.length > 0)
throw new AggregateError(failures, "Apktool process cleanup failed");
for (const path of this.workspaces) {
try {
await this.removeWorkspace(path);
this.workspaces.delete(path);
} catch (cause) {
failures.push(cause);
}
}
if (failures.length > 0)
throw new AggregateError(failures, "Apktool workspace cleanup failed");
}
private async decodeResources(
operation: "decode_android_resources",
selection: ApktoolCommandSelection,
@@ -353,28 +487,6 @@ export class ApktoolProvider {
},
signal: AbortSignal | undefined,
): Promise<Result<AnalysisExecution, AnalysisError>> {
const info = await stat(input.path).then(
(value) => value,
() => null,
);
if (info === null || !info.isFile())
return err(
new AnalysisInputError(operation, {
cause: new Error(
`The selected APK is not a readable file: ${input.path}`,
),
}),
);
const readable = await access(input.path, fsConstants.R_OK).then(
() => true,
() => false,
);
if (!readable)
return err(
new AnalysisInputError(operation, {
cause: new Error(`The selected APK is not readable: ${input.path}`),
}),
);
const root = await mkdtemp(join(tmpdir(), "rea-apktool-"));
this.workspaces.add(root);
const workspace = join(root, "decoded");
@@ -384,6 +496,10 @@ export class ApktoolProvider {
// The workspace is removed before the result is returned, so an
// unremovable workspace is reported instead of silently abandoned.
const removeWorkspace = async (): Promise<void> => {
if ([...this.#pendingCommandOwners.values()].includes(root)) {
cleanupFailure = `The decode workspace is retained until its apktool process stops: ${root}`;
return;
}
try {
await this.removeWorkspace(root);
this.workspaces.delete(root);
@@ -394,20 +510,78 @@ export class ApktoolProvider {
const failureWithCleanup = (failure: AnalysisError): AnalysisError =>
cleanupFailure === null
? failure
: failure instanceof AnalysisCancelledError
? new AnalysisCancelledError(operation, {
cause: failure,
cleanup: { reason: cleanupFailure, resources: [root] },
})
: new AnalysisProtocolError(failure.message, {
cause: failure,
cleanup: { reason: cleanupFailure, resources: [root] },
});
: analysisErrorWithCleanupFailure(
failure,
new ProviderCleanupError(
APKTOOL_PROVIDER_IDENTITY.id,
[root],
{ reason: cleanupFailure },
{ operation },
),
operation,
);
let source: FileHandle | undefined;
let sourceAdmissionFailed = false;
try {
await copyFile(input.path, snapshot, fsConstants.COPYFILE_EXCL);
throwIfCancelled(operation, signal);
sourceAdmissionFailed = true;
source = await openRegularFile(input.path, {
symlinks: "follow",
...(signal === undefined ? {} : { signal }),
});
const sourceInfo = await source.stat();
sourceAdmissionFailed = false;
const sourceSize = sourceInfo.size;
let copiedBytes = 0;
const sourceHash = createHash("sha256");
const boundedSource = new Transform({
transform(
chunk: Buffer,
_encoding: BufferEncoding,
callback: TransformCallback,
) {
copiedBytes += chunk.length;
if (copiedBytes > sourceSize) {
callback(
new AnalysisInputError(operation, {
cause: new Error(
`The selected APK changed while being copied: ${input.path}`,
),
}),
);
return;
}
sourceHash.update(chunk);
callback(null, chunk);
},
});
await pipeline(
source.createReadStream({
start: 0,
end: sourceSize,
autoClose: false,
...(signal === undefined ? {} : { signal }),
}),
boundedSource,
createWriteStream(snapshot, { flags: "wx", mode: 0o600 }),
signal === undefined ? {} : { signal },
);
const finalSourceInfo = await source.stat();
if (
copiedBytes !== sourceSize ||
!sameRegularFileState(sourceInfo, finalSourceInfo)
)
throw new AnalysisInputError(operation, {
cause: new Error(
`The selected APK changed while being copied: ${input.path}`,
),
});
const sha256 = sourceHash.digest("hex");
await source.close();
source = undefined;
await chmod(snapshot, 0o400);
throwIfCancelled(operation, signal);
const snapshotInfo = await stat(snapshot);
const sha256 = await sha256File(snapshot);
await mkdir(workspace);
await mkdir(framework);
const decoded = await runApktool(
@@ -428,14 +602,19 @@ export class ApktoolProvider {
maxOutputBytes: DECODE_MAX_OUTPUT_BYTES,
},
signal,
this.environment,
(owner) => this.#pendingCommandOwners.set(owner, root),
);
if (!decoded.ok) {
await removeWorkspace();
return err(failureWithCleanup(decoded.error));
}
throwIfCancelled(operation, signal);
const yml = await readBoundedText(
join(workspace, "apktool.yml"),
MANIFEST_MAX_BYTES,
operation,
signal,
);
if (yml.text === null || yml.truncated) {
throw new AnalysisProtocolError(
@@ -446,11 +625,12 @@ export class ApktoolProvider {
const manifestRead = await readBoundedText(
join(workspace, "AndroidManifest.xml"),
MANIFEST_MAX_BYTES,
operation,
signal,
);
const manifest = manifestRead.text;
const packageName =
manifest === null ? null : parseManifestPackage(manifest);
const locales = await listStringLocales(workspace);
const stringsDirectory =
input.locale === undefined
? "values"
@@ -459,6 +639,8 @@ export class ApktoolProvider {
? await readBoundedText(
join(workspace, "res", stringsDirectory, "strings.xml"),
STRINGS_XML_MAX_BYTES,
operation,
signal,
)
: { text: null, truncated: false };
// Do not parse an incomplete XML document as though its projection were complete.
@@ -468,7 +650,8 @@ export class ApktoolProvider {
? { entries: [], unparsedCount: 0 }
: parseDecodedStrings(stringsXml);
const bounded = parsedStrings.entries.slice(0, MAX_STRINGS);
const files = await countDecodedFiles(workspace);
const files = await countDecodedFiles(workspace, signal, operation);
throwIfCancelled(operation, signal);
const coverage =
(stringsXml !== null && parsedStrings.entries.length > MAX_STRINGS) ||
manifestRead.truncated ||
@@ -493,7 +676,7 @@ export class ApktoolProvider {
manifest,
strings: bounded,
locale: input.locale ?? null,
locales,
locales: files.locales,
decoded_file_count: files.count,
coverage,
},
@@ -535,7 +718,7 @@ export class ApktoolProvider {
: []),
...(files.capped
? [
`decoded_file_count is capped at ${String(MAX_DECODED_FILES_COUNTED)}`,
`Decoded workspace enumeration stopped after ${String(MAX_DECODED_ENTRIES_VISITED)} entries; decoded_file_count and locales may be partial`,
]
: []),
...(cleanupFailure === null ? [] : [cleanupFailure]),
@@ -547,14 +730,25 @@ export class ApktoolProvider {
await removeWorkspace();
return err(
failureWithCleanup(
cause instanceof AnalysisCancelledError ||
cause instanceof AnalysisProtocolError
? cause
: new AnalysisProtocolError("Apktool resource projection failed", {
cause,
}),
signal?.aborted === true
? new AnalysisCancelledError(operation, { cause })
: sourceAdmissionFailed
? new AnalysisInputError(operation, { cause })
: cause instanceof AnalysisInputError
? cause
: cause instanceof AnalysisCancelledError ||
cause instanceof AnalysisProtocolError
? cause
: new AnalysisProtocolError(
"Apktool resource projection failed",
{
cause,
},
),
),
);
} finally {
await source?.close().catch(() => undefined);
}
}
}
+9 -8
View File
@@ -1,5 +1,5 @@
import { execFile } from "node:child_process";
import { lstat, readdir, readFile, realpath, stat } from "node:fs/promises";
import { lstat, opendir, readFile, realpath, stat } from "node:fs/promises";
import { join } from "node:path";
import { promisify } from "node:util";
@@ -156,13 +156,14 @@ const wrappedBundle = async (bundle: string): Promise<string | undefined> => {
if (isAbsence(cause)) return undefined;
throw cause;
}
const apps = (await readdir(wrapper, { withFileTypes: true })).filter(
(entry) => entry.isDirectory() && entry.name.toLowerCase().endsWith(".app"),
);
const [app] = apps;
return apps.length === 1 && app !== undefined
? join(wrapper, app.name)
: undefined;
let app: string | undefined;
for await (const entry of await opendir(wrapper)) {
if (!entry.isDirectory() || !entry.name.toLowerCase().endsWith(".app"))
continue;
if (app !== undefined) return undefined;
app = join(wrapper, entry.name);
}
return app;
};
const resolveLayoutExecutable = async (
+10 -3
View File
@@ -1,4 +1,4 @@
import { constants } from "node:fs";
import { constants, type Stats } from "node:fs";
import { createHash } from "node:crypto";
import {
access,
@@ -248,7 +248,7 @@ const isArchiveFormat = (
const sha256Handle = async (
handle: FileHandle,
path: string,
initial: Awaited<ReturnType<FileHandle["stat"]>>,
initial: Stats,
signal?: AbortSignal,
): Promise<string> => {
const hash = createHash("sha256");
@@ -256,11 +256,18 @@ const sha256Handle = async (
let position = 0;
while (true) {
throwIfTargetResolutionCancelled(signal);
const observed = await handle.read(chunk, 0, chunk.length, position);
const observed = await handle.read(
chunk,
0,
Math.min(chunk.length, initial.size - position + 1),
position,
);
throwIfTargetResolutionCancelled(signal);
if (observed.bytesRead === 0) break;
hash.update(chunk.subarray(0, observed.bytesRead));
position += observed.bytesRead;
if (position > initial.size)
throw new BinaryTargetError(path, "target grew while being read");
}
throwIfTargetResolutionCancelled(signal);
const [opened, currentPath] = await Promise.all([handle.stat(), lstat(path)]);
@@ -6,6 +6,7 @@ import type { Result } from "../domain/result.js";
/** Provider-neutral application boundary for controlled browser scenarios. */
export interface BrowserScenarioCapturePort {
close?(): Promise<void>;
identity(): ProviderIdentity;
captureScenario(
scenario: BrowserScenario,
+2 -2
View File
@@ -1,5 +1,5 @@
import { err, ok, type Result } from "../domain/result.js";
import { readFile } from "node:fs/promises";
import { readRegularFileText } from "./RegularFileRead.js";
import {
effectiveClientServer,
parseClientConfiguration,
@@ -19,7 +19,7 @@ export const readConfigurationText = async (
path: string,
): Promise<Result<string | undefined, ClientConfigurationFileError>> => {
try {
return ok(await readFile(path, "utf8"));
return ok(await readRegularFileText(path));
} catch (cause: unknown) {
if (cause instanceof Error && "code" in cause && cause.code === "ENOENT")
return ok(undefined);
+1 -1
View File
@@ -6,7 +6,7 @@ import { ProviderAdapterError } from "../domain/providerAdapterError.js";
import { err, type Result } from "../domain/result.js";
import type { AnalysisExecution } from "./AnalysisProvider.js";
import type { BinarySession } from "./binary/BinarySession.js";
import { analysisErrorWithCleanupFailure } from "./binary/AnalysisClientCleanup.js";
import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js";
/** Await one-shot cleanup and retain either the outcome or both failures. */
export const withSessionCleanup = async <Value>(
+43
View File
@@ -0,0 +1,43 @@
/** Maximum admitted vendor response bytes, shared by Hopper setup adapters. */
export const HOPPER_DOWNLOAD_BYTES = 100_000_000;
/** Bound observed response bytes before retaining them, including chunked bodies. */
export const downloadHopperPackage = async (
url: string,
options: { readonly signal?: AbortSignal } = {},
): Promise<{ readonly ok: boolean; readonly bytes: Uint8Array }> => {
const deadline = AbortSignal.timeout(120_000);
const response = await fetch(url, {
headers: { accept: "application/json", "user-agent": "rea-installer" },
signal:
options.signal === undefined
? deadline
: AbortSignal.any([options.signal, deadline]),
});
const refused = { ok: false, bytes: new Uint8Array() };
if (
!response.ok ||
Number(response.headers.get("content-length")) > HOPPER_DOWNLOAD_BYTES
) {
await response.body?.cancel();
return refused;
}
if (response.body === null) return { ok: true, bytes: new Uint8Array() };
const reader = response.body.getReader();
const chunks: Uint8Array[] = [];
let bytes = 0;
try {
while (true) {
const chunk = await reader.read();
if (chunk.done) return { ok: true, bytes: Buffer.concat(chunks, bytes) };
if (chunk.value.byteLength > HOPPER_DOWNLOAD_BYTES - bytes) {
await reader.cancel();
return refused;
}
chunks.push(chunk.value);
bytes += chunk.value.byteLength;
}
} finally {
reader.releaseLock();
}
};
+12 -5
View File
@@ -16,8 +16,7 @@ import {
AnalysisResourceConstraintError,
} from "../domain/analysisErrorCore.js";
import { err, ok, type Result } from "../domain/result.js";
import { parseUtf8Json } from "./Utf8JsonInput.js";
import { readRegularFile } from "./RegularFileRead.js";
import { readJsonInputFile } from "./JsonInputFile.js";
import { NonRegularFileReadError } from "../filesystem/RegularFile.js";
/** Request control and its owning operation for an interruptible atomic write. */
@@ -37,12 +36,20 @@ export const readJsonFile = async (
> => {
const requestedPath = resolve(path);
try {
const encoded = await readRegularFile(requestedPath);
const decoded = parseUtf8Json(encoded, "read_evidence_file", requestedPath);
const decoded = await readJsonInputFile(
requestedPath,
"read_evidence_file",
);
if (!decoded.ok) {
if (decoded.error instanceof AnalysisResourceConstraintError)
return err(decoded.error);
return err(
new EvidenceFileError("read", "invalid-json", {
cause: decoded.cause,
cause:
decoded.error.cause ??
new SyntaxError(
decoded.error.issues[0]?.message ?? decoded.error.message,
),
path: requestedPath,
}),
);
@@ -6,61 +6,76 @@ import { getHeapStatistics } from "node:v8";
import parser from "stream-json/parser.js";
import streamValues from "stream-json/streamers/stream-values.js";
import { withRegularFile } from "./application/RegularFileRead.js";
import { RegularFileChangedError } from "../filesystem/RegularFile.js";
import { withRegularFile } from "./RegularFileRead.js";
import {
JSON_BYTE_ORDER_MARK_MESSAGE,
JSON_INPUT_RESOURCE_REMEDIATION,
parseUtf8Json,
} from "./application/Utf8JsonInput.js";
} from "./Utf8JsonInput.js";
import {
AnalysisInputError,
AnalysisResourceConstraintError,
} from "./domain/analysisErrorCore.js";
import { err, ok, type Result } from "./domain/result.js";
} from "../domain/analysisErrorCore.js";
import { err, ok, type Result } from "../domain/result.js";
type JsonFileFailure = AnalysisInputError | AnalysisResourceConstraintError;
type JsonInputFileFailure =
| AnalysisInputError
| AnalysisResourceConstraintError;
const READ_CHUNK_BYTES = 64 * 1024;
const PACKED_STRING_PART_CODE_UNITS = 64 * 1024;
const PACKED_TOKEN_PART_CODE_UNITS = 64 * 1024;
// Retain native-parser speed for modest inputs while bounding its extra copies.
// This selects an implementation; larger files remain accepted through streaming.
const NATIVE_PARSE_READ_BUDGET = 8 * 1024 * 1024;
/** Keep native parsing only when input copies and value expansion fit live headroom. */
const nativeParseFits = (inputBytes: number): boolean =>
inputBytes <= NATIVE_PARSE_READ_BUDGET &&
// The assembler estimate reserves up to 128 bytes per two-byte container.
// Allow that 64x expansion plus byte/text copies before native JSON.parse,
// whose individual container allocations cannot be intercepted.
inputBytes * 72 + READ_CHUNK_BYTES <=
getHeapStatistics().total_available_size;
/** Parse strict UTF-8 JSON without requiring a whole-document string. */
export const readCliJsonFile = (
export const readJsonInputFile = (
path: string,
operation: string,
signal?: AbortSignal,
): Promise<Result<unknown, JsonFileFailure>> =>
): Promise<Result<unknown, JsonInputFileFailure>> =>
withRegularFile(
path,
async (handle, stats) => {
let found = false;
let value: unknown;
try {
const prefix =
stats.size <= NATIVE_PARSE_READ_BUDGET
? await readPrefix(handle, stats.size + 1, signal)
: undefined;
if (prefix?.complete === true) {
const prefix = nativeParseFits(stats.size)
? await readPrefix(handle, stats.size + 1, signal)
: undefined;
if (prefix !== undefined) {
if (!prefix.complete || prefix.bytes.length !== stats.size)
throw new RegularFileChangedError(path);
const parsed = parseUtf8Json(
prefix.bytes,
operation,
path,
"cli-json-input",
"json-file-input",
);
return parsed.ok
? ok(parsed.value)
: err(invalidJson(operation, parsed.error, parsed.cause));
}
await pipeline(
decodedChunks(handle, signal, prefix?.bytes),
decodedChunks(handle, stats.size, path, signal),
parser.asStream({
jsonStreaming: false,
streamValues: false,
packKeys: false,
packStrings: false,
packNumbers: false,
}),
packJsonStrings(operation, stats.size),
packJsonScalars(operation, stats.size, path),
admitJsonAssembly(operation, stats.size, path),
streamValues.asStream(),
async (rows: AsyncIterable<unknown>) => {
for await (const row of rows) {
@@ -107,8 +122,9 @@ export const readCliJsonFile = (
new AnalysisResourceConstraintError(
operation,
"memory",
"An individual JSON string exceeds the runtime string limit",
"An individual JSON string or number token exceeds the runtime string limit",
{
input_path: path,
input_file_bytes: stats.size,
max_string_code_units: constants.MAX_STRING_LENGTH,
},
@@ -121,12 +137,16 @@ export const readCliJsonFile = (
throw cause;
}
},
signal,
{ signal },
);
const packJsonStrings = (operation: string, inputFileBytes: number) =>
const packJsonScalars = (
operation: string,
inputFileBytes: number,
path: string,
) =>
async function* (tokens: AsyncIterable<unknown>): AsyncGenerator<unknown> {
let kind: "keyValue" | "stringValue" | undefined;
let kind: "keyValue" | "stringValue" | "numberValue" | undefined;
let length = 0;
let value = "";
let parts: string[] = [];
@@ -139,23 +159,32 @@ const packJsonStrings = (operation: string, inputFileBytes: number) =>
case "startString":
kind = token.name === "startKey" ? "keyValue" : "stringValue";
break;
case "startNumber":
kind = "numberValue";
break;
case "numberChunk":
case "stringChunk":
if (
kind === undefined ||
!("value" in token) ||
typeof token.value !== "string"
)
throw new Error("Unexpected JSON string fragment");
throw new Error("Unexpected JSON scalar fragment");
length += token.value.length;
if (length > constants.MAX_STRING_LENGTH)
throw new RangeError("Invalid string length");
parts.push(token.value);
partLength += token.value.length;
if (partLength >= PACKED_STRING_PART_CODE_UNITS) {
if (partLength >= PACKED_TOKEN_PART_CODE_UNITS) {
// The tokenizer emits short fragments. Coalesce them before retaining
// a string so millions of substring/rope nodes cannot exhaust the heap
// a scalar so millions of substring/rope nodes cannot exhaust the heap
// before the native string-length constraint can be reported.
requireStringAssemblyHeadroom(operation, inputFileBytes, length);
requireStringAssemblyHeadroom(
operation,
inputFileBytes,
path,
length,
);
value += parts.join("");
parts = [];
partLength = 0;
@@ -163,8 +192,14 @@ const packJsonStrings = (operation: string, inputFileBytes: number) =>
break;
case "endKey":
case "endString":
if (kind === undefined) throw new Error("Unexpected JSON string end");
requireStringAssemblyHeadroom(operation, inputFileBytes, length);
case "endNumber":
if (kind === undefined) throw new Error("Unexpected JSON scalar end");
requireStringAssemblyHeadroom(
operation,
inputFileBytes,
path,
length,
);
yield { name: kind, value: value + parts.join("") };
kind = undefined;
length = 0;
@@ -178,22 +213,78 @@ const packJsonStrings = (operation: string, inputFileBytes: number) =>
}
};
/** Admit the retained value graph before the upstream assembler grows it. */
const admitJsonAssembly = (
operation: string,
inputFileBytes: number,
path: string,
) =>
async function* (tokens: AsyncIterable<unknown>): AsyncGenerator<unknown> {
let projectedBytes = 0;
let nextCheck = 0;
for await (const token of tokens) {
if (typeof token !== "object" || token === null || !("name" in token))
throw new Error("Unexpected JSON assembly token");
if (token.name === "startObject" || token.name === "startArray") {
projectedBytes += 128;
} else if (token.name === "keyValue" || token.name === "stringValue") {
if (!("value" in token) || typeof token.value !== "string")
throw new Error("Unexpected packed JSON string");
projectedBytes += 64 + token.value.length * 2;
} else if (
token.name === "numberValue" ||
token.name === "nullValue" ||
token.name === "trueValue" ||
token.name === "falseValue"
) {
projectedBytes += 64;
}
if (projectedBytes >= nextCheck) {
const heap = getHeapStatistics();
// Conservative capacity for containers, property/array slots and UTF-16
// values. Keep another whole projection available for container growth,
// plus the next chunk of estimates between samples. This is admission
// against live runtime headroom, not a fixed document-size policy.
const requiredBytes = projectedBytes + READ_CHUNK_BYTES;
if (requiredBytes > heap.total_available_size)
throw new AnalysisResourceConstraintError(
operation,
"memory",
"Insufficient heap headroom to materialize the complete JSON value",
{
input_path: path,
input_file_bytes: inputFileBytes,
projected_value_bytes: projectedBytes,
assembly_headroom_bytes: requiredBytes,
available_heap_bytes: heap.total_available_size,
heap_size_limit_bytes: heap.heap_size_limit,
},
{ remediationAction: JSON_INPUT_RESOURCE_REMEDIATION },
);
nextCheck = projectedBytes + READ_CHUNK_BYTES;
}
yield token;
}
};
const requireStringAssemblyHeadroom = (
operation: string,
inputFileBytes: number,
path: string,
stringCodeUnits: number,
): void => {
if (stringCodeUnits < PACKED_STRING_PART_CODE_UNITS) return;
if (stringCodeUnits < PACKED_TOKEN_PART_CODE_UNITS) return;
const heap = getHeapStatistics();
// Concatenation or key interning may flatten the retained rope. Reserve its
// UTF-16 representation and the next coalesced part before that allocation.
const requiredBytes = 2 * (stringCodeUnits + PACKED_STRING_PART_CODE_UNITS);
const requiredBytes = 2 * (stringCodeUnits + PACKED_TOKEN_PART_CODE_UNITS);
if (requiredBytes > heap.total_available_size)
throw new AnalysisResourceConstraintError(
operation,
"memory",
"Insufficient heap headroom to assemble a complete JSON string",
{
input_path: path,
input_file_bytes: inputFileBytes,
string_code_units: stringCodeUnits,
string_assembly_headroom_bytes: requiredBytes,
@@ -208,8 +299,9 @@ const requireStringAssemblyHeadroom = (
async function* decodedChunks(
handle: FileHandle,
admittedBytes: number,
path: string,
signal: AbortSignal | undefined,
prefix?: Uint8Array,
): AsyncGenerator<string> {
const decoder = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
let leading = true;
@@ -220,25 +312,25 @@ async function* decodedChunks(
}
return text;
};
if (prefix !== undefined)
for (let offset = 0; offset < prefix.length; offset += READ_CHUNK_BYTES) {
signal?.throwIfAborted();
yield checked(
decoder.decode(prefix.subarray(offset, offset + READ_CHUNK_BYTES), {
stream: true,
}),
);
}
const bytes = Buffer.allocUnsafe(READ_CHUNK_BYTES);
let observedBytes = 0;
while (true) {
signal?.throwIfAborted();
const read = await handle.read(bytes, 0, bytes.length, null);
const read = await handle.read(
bytes,
0,
Math.min(bytes.length, admittedBytes - observedBytes + 1),
null,
);
signal?.throwIfAborted();
if (read.bytesRead === 0) break;
observedBytes += read.bytesRead;
if (observedBytes > admittedBytes) throw new RegularFileChangedError(path);
yield checked(
decoder.decode(bytes.subarray(0, read.bytesRead), { stream: true }),
);
}
if (observedBytes !== admittedBytes) throw new RegularFileChangedError(path);
yield checked(decoder.decode());
}
@@ -264,8 +356,7 @@ const readPrefix = async (
return { bytes: bytes.subarray(0, offset), complete: true };
offset += read.bytesRead;
}
// The file grew after admission. Continue from this same handle, preserving
// the prefix already read rather than reopening or dropping its bytes.
// One byte beyond the admitted extent proves the selected file changed.
return { bytes, complete: false };
};
+6 -12
View File
@@ -7,10 +7,13 @@ import { join } from "node:path";
import { promisify } from "node:util";
import { execFileOutput } from "../process/ExecFileOutput.js";
import {
downloadHopperPackage,
HOPPER_DOWNLOAD_BYTES,
} from "./HopperDownload.js";
const execFileAsync = promisify(execFile);
const DOWNLOAD_PREFIX = "https://www.hopperapp.com:443/downloader/public/";
const MAX_PACKAGE_BYTES = 100_000_000;
const SUPPORTED_HOPPER_SHA256 = new Set([
"0294ced141cc373468ee22d8343e7dac41980cb05a937994ca81c9f09afe7ded",
"1339f9e58377442b0c6fcb0dfc3cec20d593cc557408521fad9a00dbc6b8da13",
@@ -231,16 +234,7 @@ export const systemLinuxHopperInstallHost = (
environment: Readonly<NodeJS.ProcessEnv>,
): LinuxHopperInstallHost => ({
distribution: readLinuxDistribution,
async download(url, options) {
const response = await fetch(url, {
headers: { accept: "application/json", "user-agent": "rea-installer" },
...(options.signal === undefined ? {} : { signal: options.signal }),
});
return {
ok: response.ok,
bytes: new Uint8Array(await response.arrayBuffer()),
};
},
download: downloadHopperPackage,
createTemporaryDirectory: () => mkdtemp(join(tmpdir(), "rea-hopper-")),
writeArchive: (path, bytes) => writeFile(path, bytes, { mode: 0o600 }),
installPackage: (family, archive) =>
@@ -350,7 +344,7 @@ const packageIntegrityMatches = (
release: LinuxHopperRelease,
): boolean =>
bytes.byteLength === Number(release.file_length) &&
bytes.byteLength <= MAX_PACKAGE_BYTES &&
bytes.byteLength <= HOPPER_DOWNLOAD_BYTES &&
createHash("sha1").update(bytes).digest("hex") === release.file_hash;
const packageFamilyFor = (
+213 -167
View File
@@ -15,12 +15,15 @@ import { promisify } from "node:util";
import { z } from "zod";
import { homeDirectoryFromEnvironment } from "../config/homeDirectory.js";
import {
downloadHopperPackage,
HOPPER_DOWNLOAD_BYTES,
} from "./HopperDownload.js";
const execFileAsync = promisify(execFile);
const RELEASES_URL =
"https://www.hopperapp.com/include/files-api.php?request=releases&public=true";
const DOWNLOAD_PREFIX = "https://www.hopperapp.com:443/downloader/public/";
const MAX_PACKAGE_BYTES = 100_000_000;
const BUNDLE_IDENTIFIER = "com.cryptic-apps.hopper-web-4";
/** Stable network and command effects disclosed before macOS Hopper setup. */
@@ -36,7 +39,11 @@ export const macHopperInstallDisclosure = {
/** Result of installing the official Hopper macOS application. */
export type MacHopperInstallResult =
| { readonly status: "installed"; readonly launcherPath: string }
| {
readonly status: "installed";
readonly launcherPath: string;
readonly cleanupFailure?: string;
}
| {
readonly status: "failed";
readonly reason:
@@ -48,6 +55,7 @@ export type MacHopperInstallResult =
| "destination_exists"
| "copy"
| "launcher_missing";
readonly cleanupFailure?: string;
};
/** External effects used by the bounded macOS Hopper installer. */
@@ -72,6 +80,8 @@ export interface MacHopperInstallHost {
launcherReady(path: string): Promise<boolean>;
openApplication(path: string): Promise<void>;
cleanup(path: string): Promise<void>;
/** Retain the cleanup capability until the owning setup host retries it. */
retainCleanup?(cleanup: () => Promise<void>): void;
destination(): string;
}
@@ -85,187 +95,221 @@ export const installMacHopper = async (
): Promise<MacHopperInstallResult> => {
let temporary: string | undefined;
let mounted = false;
let result: MacHopperInstallResult;
try {
const metadata = await host.download(RELEASES_URL);
if (!metadata.ok) return { status: "failed", reason: "release_metadata" };
const parsed = releasesSchema.safeParse(parseJson(metadata.bytes));
if (!parsed.success)
return { status: "failed", reason: "release_metadata" };
const release = parsed.data["OS X"];
if (!release.filename.startsWith(DOWNLOAD_PREFIX))
return { status: "failed", reason: "release_metadata" };
const archive = await host.download(release.filename);
if (!archive.ok) return { status: "failed", reason: "download" };
if (!integrityMatches(archive.bytes, release))
return { status: "failed", reason: "integrity" };
result = await (async (): Promise<MacHopperInstallResult> => {
const metadata = await host.download(RELEASES_URL);
if (!metadata.ok) return { status: "failed", reason: "release_metadata" };
const parsed = releasesSchema.safeParse(parseJson(metadata.bytes));
if (!parsed.success)
return { status: "failed", reason: "release_metadata" };
const release = parsed.data["OS X"];
if (!release.filename.startsWith(DOWNLOAD_PREFIX))
return { status: "failed", reason: "release_metadata" };
const archive = await host.download(release.filename);
if (!archive.ok) return { status: "failed", reason: "download" };
if (!integrityMatches(archive.bytes, release))
return { status: "failed", reason: "integrity" };
const destination = host.destination();
const destinationExists = await host.destinationExists(destination);
if (destinationExists && options.replaceExisting !== true)
return { status: "failed", reason: "destination_exists" };
temporary = await host.createTemporaryDirectory();
const packagePath = join(temporary, "hopper.dmg");
const mountPath = join(temporary, "mounted");
await host.createMountDirectory(mountPath);
await host.writePackage(packagePath, archive.bytes);
if (!(await host.mount(packagePath, mountPath)))
return { status: "failed", reason: "mount" };
mounted = true;
const source = await host.appBundle(mountPath);
if (source === undefined || !(await host.validBundle(source)))
return { status: "failed", reason: "bundle" };
const stage = `${destination}.rea-stage`;
if (
!(await host.installBundle(source, destination, stage, destinationExists))
)
return { status: "failed", reason: "copy" };
const launcherPath = join(destination, "Contents/MacOS/hopper");
if (!(await host.launcherReady(launcherPath)))
return { status: "failed", reason: "launcher_missing" };
await host.openApplication(destination);
return { status: "installed", launcherPath };
const destination = host.destination();
const destinationExists = await host.destinationExists(destination);
if (destinationExists && options.replaceExisting !== true)
return { status: "failed", reason: "destination_exists" };
temporary = await host.createTemporaryDirectory();
const packagePath = join(temporary, "hopper.dmg");
const mountPath = join(temporary, "mounted");
await host.createMountDirectory(mountPath);
await host.writePackage(packagePath, archive.bytes);
if (!(await host.mount(packagePath, mountPath)))
return { status: "failed", reason: "mount" };
mounted = true;
const source = await host.appBundle(mountPath);
if (source === undefined || !(await host.validBundle(source)))
return { status: "failed", reason: "bundle" };
const stage = `${destination}.rea-stage`;
if (
!(await host.installBundle(
source,
destination,
stage,
destinationExists,
))
)
return { status: "failed", reason: "copy" };
const launcherPath = join(destination, "Contents/MacOS/hopper");
if (!(await host.launcherReady(launcherPath)))
return { status: "failed", reason: "launcher_missing" };
await host.openApplication(destination);
return { status: "installed", launcherPath };
})();
} catch (cause: unknown) {
// Install-stage failures share the download reason; the stage is cleaned below.
void cause;
return { status: "failed", reason: "download" };
} finally {
if (temporary !== undefined) {
const mountPath = join(temporary, "mounted");
if (mounted) await host.unmount(mountPath);
await host.cleanup(temporary);
result = { status: "failed", reason: "download" };
}
let cleanupFailure: string | undefined;
if (temporary !== undefined) {
const temporaryPath = temporary;
const mountPath = join(temporary, "mounted");
const cleanup = async (): Promise<void> => {
if (mounted) {
await host.unmount(mountPath);
mounted = false;
}
await host.cleanup(temporaryPath);
};
try {
await cleanup();
} catch (cause: unknown) {
cleanupFailure = `Hopper DMG cleanup failed: ${failureMessage(cause)}`;
host.retainCleanup?.(cleanup);
}
}
return cleanupFailure === undefined ? result : { ...result, cleanupFailure };
};
/** Compose installer paths and subprocesses from one selected caller context. */
export const systemMacHopperInstallHost = (
homeDirectory: string,
environment: Readonly<NodeJS.ProcessEnv>,
): MacHopperInstallHost => ({
download: downloadPackage,
createTemporaryDirectory: () => mkdtemp(join(tmpdir(), "rea-hopper-mac-")),
createMountDirectory: (path) => mkdir(path),
writePackage: (path, bytes) => writeFile(path, bytes, { mode: 0o600 }),
async mount(packagePath, mountPath) {
return commandSucceeds(
"hdiutil",
[
"attach",
"-readonly",
"-nobrowse",
"-mountpoint",
mountPath,
packagePath,
],
environment,
);
},
async unmount(mountPath) {
await commandSucceeds(
"hdiutil",
["detach", mountPath, "-force"],
environment,
);
},
async appBundle(mountPath) {
const entries = await readdir(mountPath, { withFileTypes: true });
const app = entries.find(
(entry) => entry.isDirectory() && /^Hopper.*\.app$/iu.test(entry.name),
);
return app === undefined ? undefined : join(mountPath, app.name);
},
async validBundle(appPath) {
if (
!(await commandSucceeds(
"codesign",
["--verify", "--deep", "--strict", appPath],
): MacHopperInstallHost & { close(): Promise<string | undefined> } => {
const pendingCleanup = new Set<() => Promise<void>>();
return {
download: downloadHopperPackage,
createTemporaryDirectory: () => mkdtemp(join(tmpdir(), "rea-hopper-mac-")),
createMountDirectory: (path) => mkdir(path),
writePackage: (path, bytes) => writeFile(path, bytes, { mode: 0o600 }),
async mount(packagePath, mountPath) {
return commandSucceeds(
"hdiutil",
[
"attach",
"-readonly",
"-nobrowse",
"-mountpoint",
mountPath,
packagePath,
],
environment,
))
)
return false;
try {
const identifier = (
await execFileAsync(
"/usr/libexec/PlistBuddy",
[
"-c",
"Print :CFBundleIdentifier",
join(appPath, "Contents/Info.plist"),
],
{ env: environment },
)
).stdout.trim();
return identifier === BUNDLE_IDENTIFIER;
} catch (cause: unknown) {
// best-effort cleanup: optional bundle probing; unreadable means invalid.
void cause;
return false;
}
},
async destinationExists(path) {
try {
await access(path);
return true;
} catch (cause: unknown) {
// best-effort cleanup: optional existence probing; absence means missing.
void cause;
return false;
}
},
async installBundle(source, destination, stage, replaceExisting) {
const previous = `${stage}-previous`;
try {
await mkdir(join(homeDirectory, "Applications"), { recursive: true });
await rm(stage, { recursive: true, force: true });
await rm(previous, { recursive: true, force: true });
await execFileAsync("ditto", [source, stage], { env: environment });
await access(join(stage, "Contents/MacOS/hopper"));
if (replaceExisting) await rename(destination, previous);
await rename(stage, destination);
if (replaceExisting) await rm(previous, { recursive: true, force: true });
return true;
} catch (cause: unknown) {
// Install failure rolls back below; the false return reports it.
void cause;
await rm(stage, { recursive: true, force: true });
if (replaceExisting) {
);
},
async unmount(mountPath) {
if (
!(await commandSucceeds(
"hdiutil",
["detach", mountPath, "-force"],
environment,
))
)
throw new Error(
`hdiutil detach failed for ${JSON.stringify(mountPath)}`,
);
},
async appBundle(mountPath) {
const entries = await readdir(mountPath, { withFileTypes: true });
const app = entries.find(
(entry) => entry.isDirectory() && /^Hopper.*\.app$/iu.test(entry.name),
);
return app === undefined ? undefined : join(mountPath, app.name);
},
async validBundle(appPath) {
if (
!(await commandSucceeds(
"codesign",
["--verify", "--deep", "--strict", appPath],
environment,
))
)
return false;
try {
const identifier = (
await execFileAsync(
"/usr/libexec/PlistBuddy",
[
"-c",
"Print :CFBundleIdentifier",
join(appPath, "Contents/Info.plist"),
],
{ env: environment },
)
).stdout.trim();
return identifier === BUNDLE_IDENTIFIER;
} catch (cause: unknown) {
// best-effort cleanup: optional bundle probing; unreadable means invalid.
void cause;
return false;
}
},
async destinationExists(path) {
try {
await access(path);
return true;
} catch (cause: unknown) {
// best-effort cleanup: optional existence probing; absence means missing.
void cause;
return false;
}
},
async installBundle(source, destination, stage, replaceExisting) {
const previous = `${stage}-previous`;
try {
await mkdir(join(homeDirectory, "Applications"), { recursive: true });
await rm(stage, { recursive: true, force: true });
await rm(previous, { recursive: true, force: true });
await execFileAsync("ditto", [source, stage], { env: environment });
await access(join(stage, "Contents/MacOS/hopper"));
if (replaceExisting) await rename(destination, previous);
await rename(stage, destination);
if (replaceExisting)
await rm(previous, { recursive: true, force: true });
return true;
} catch (cause: unknown) {
// Install failure rolls back below; the false return reports it.
void cause;
await rm(stage, { recursive: true, force: true });
if (replaceExisting) {
try {
await rename(previous, destination);
} catch (rollbackCause: unknown) {
// The original destination was not moved, or rollback is impossible.
void rollbackCause;
}
}
return false;
}
},
async launcherReady(path) {
try {
await access(path);
return true;
} catch (cause: unknown) {
// best-effort cleanup: optional launcher probing; absence means not ready.
void cause;
return false;
}
},
async openApplication(path) {
await execFileAsync("open", [path], { env: environment });
},
cleanup: (path) => rm(path, { recursive: true, force: true }),
retainCleanup: (cleanup) => pendingCleanup.add(cleanup),
async close() {
let failure: string | undefined;
for (const cleanup of pendingCleanup) {
try {
await rename(previous, destination);
} catch (rollbackCause: unknown) {
// The original destination was not moved, or rollback is impossible.
void rollbackCause;
await cleanup();
pendingCleanup.delete(cleanup);
} catch (cause: unknown) {
failure ??= `Hopper DMG cleanup remains incomplete: ${failureMessage(cause)}`;
}
}
return false;
}
},
async launcherReady(path) {
try {
await access(path);
return true;
} catch (cause: unknown) {
// best-effort cleanup: optional launcher probing; absence means not ready.
void cause;
return false;
}
},
async openApplication(path) {
await execFileAsync("open", [path], { env: environment });
},
cleanup: (path) => rm(path, { recursive: true, force: true }),
destination: () =>
join(homeDirectory, "Applications/Hopper Disassembler.app"),
});
const downloadPackage = async (url: string) => {
const response = await fetch(url, {
headers: { accept: "application/json", "user-agent": "rea-installer" },
signal: AbortSignal.timeout(120_000),
});
const length = Number(response.headers.get("content-length") ?? "0");
if (length > MAX_PACKAGE_BYTES) return { ok: false, bytes: new Uint8Array() };
const bytes = new Uint8Array(await response.arrayBuffer());
return { ok: response.ok && bytes.byteLength <= MAX_PACKAGE_BYTES, bytes };
return failure;
},
destination: () =>
join(homeDirectory, "Applications/Hopper Disassembler.app"),
};
};
const parseJson = (bytes: Uint8Array): unknown =>
@@ -275,7 +319,7 @@ const integrityMatches = (
release: z.infer<typeof releaseSchema>,
): boolean =>
bytes.byteLength === Number(release.file_length) &&
bytes.byteLength <= MAX_PACKAGE_BYTES &&
bytes.byteLength <= HOPPER_DOWNLOAD_BYTES &&
createHash("sha1").update(bytes).digest("hex") === release.file_hash;
const commandSucceeds = async (
command: string,
@@ -291,6 +335,8 @@ const commandSucceeds = async (
return false;
}
};
const failureMessage = (cause: unknown): string =>
cause instanceof Error ? cause.message : String(cause);
const releaseSchema = z.object({
filename: z.string().url(),
file_length: z.string().regex(/^\d+$/u),
@@ -1,3 +1,11 @@
import { javascriptApplicationAnalysisResultSchema } from "../domain/javascript/javascriptApplicationAnalysis.js";
import { JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE } from "../contracts/javascript/javascriptRuntimeReconciliationExample.js";
import {
createJavaScriptSemanticGraph,
createJavaScriptSemanticGraphNode,
JavaScriptSemanticEvidenceContextRegistry,
} from "../domain/javascript/javascriptSemanticGraph.js";
import { proofEvidence } from "./ReconstructionObligationLedger.fixture.js";
import { describe, expect, it } from "vitest";
import { createEvidence } from "../domain/evidence.js";
@@ -171,3 +179,115 @@ describe("reconstruction obligation candidates", () => {
);
});
});
it("keeps each semantic candidate bound to its own file and authority context", () => {
const contexts = new JavaScriptSemanticEvidenceContextRegistry();
const base = javascriptApplicationAnalysisResultSchema.parse(
JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE.normalized_result,
);
const sourceRecords = [
proofEvidence("first-file"),
proofEvidence("second-file"),
];
const module = createJavaScriptSemanticGraphNode(
{
kind: "module",
identity: {
artifact_sha256: base.root_artifact_sha256,
module_path: "app.js",
source_range: null,
role_key: "module",
},
function_node_id: null,
application_node_ids: [],
label: "app.js",
properties: {},
evidence: artifactEvidence(base.root_artifact_sha256, "app.js"),
},
contexts,
);
const requests = sourceRecords.flatMap((source, fileIndex) =>
[1, 2].map((line) => {
const sha256 = String(fileIndex + 4).repeat(64);
const path = `file-${String(fileIndex)}.js`;
return createJavaScriptSemanticGraphNode(
{
kind: "request",
identity: {
artifact_sha256: sha256,
module_path: path,
source_range: {
start: { line, column: 0 },
end: { line, column: 8 },
},
role_key: `request-${String(line)}`,
},
function_node_id: null,
application_node_ids: [],
label: `${path}:${String(line)}`,
properties: {},
evidence: {
...artifactEvidence(sha256, path, "ast-static-analysis"),
evidence_ids: [source.evidence_id],
},
},
contexts,
);
}),
);
const { graph_id: _graphId, ...semantic } = base.semantic_graph;
const semanticGraph = createJavaScriptSemanticGraph({
...semantic,
root_node_ids: [module.node_id],
nodes: [module, ...requests],
evidence_contexts: contexts.contexts,
});
const example = JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE;
const application = createEvidence(
{
path: base.input_path,
sha256: base.root_artifact_sha256,
format: base.format,
},
example.provider,
{
predicateType: example.predicate_type,
operation: example.operation,
parameters: example.parameters,
result: jsonValueSchema.parse({ ...base, semantic_graph: semanticGraph }),
confidence: "derived",
authority: "shipped-artifact",
evidenceLinks: sourceRecords.map(({ evidence_id }) => evidence_id),
},
);
const generated = deriveReconstructionObligationCandidates(
createEvidenceBundle([...sourceRecords, application]),
[],
);
expect(generated.limitations).toEqual([]);
expect(generated.candidates).toHaveLength(requests.length);
for (const [index, node] of requests.entries()) {
const source = sourceRecords[Math.floor(index / 2)];
expect(
generated.candidates.find(
({ target }) => target.semantic_node_id === node.node_id,
),
).toMatchObject({
title: node.label,
family: "request",
source_state: "candidate",
target: {
artifact_sha256: node.identity.artifact_sha256,
semantic_node_id: node.node_id,
},
authority_references: [
{
evidence_id: source?.evidence_id,
authority: "controlled-replay",
state: "candidate",
location: `${semanticGraph.graph_id}/node/${node.node_id}`,
},
],
});
}
});
@@ -12,7 +12,6 @@ import type {
ReconstructionObligation,
ReviewedReconstructionObligation,
} from "../domain/reconstructionObligationLedgerSchemas.js";
import { resolveJavaScriptSemanticEvidence } from "../domain/javascript/javascriptSemanticGraph.js";
import { parseApplicationGraphEvidence } from "./javascript/JavaScriptApplicationEvidenceGraph.js";
import {
applicationObligationPolicy,
@@ -115,14 +114,21 @@ const deriveApplicationCandidates = (
}),
);
}
if (source.semanticGraph !== null)
if (source.semanticGraph !== null) {
const evidenceContexts = new Map(
source.semanticGraph.evidence_contexts.map((context) => [
context.context_id,
context,
]),
);
for (const node of source.semanticGraph.nodes) {
const candidatePolicy = semanticObligationPolicy(node.kind);
if (candidatePolicy === undefined) continue;
const evidenceContext = resolveJavaScriptSemanticEvidence(
source.semanticGraph,
node.evidence,
);
const evidenceContext = evidenceContexts.get(node.evidence.context_id);
if (evidenceContext === undefined)
throw new TypeError(
"Semantic evidence reference names an absent context",
);
addCandidate(
candidates,
generatedCandidate({
@@ -144,6 +150,7 @@ const deriveApplicationCandidates = (
}),
);
}
}
if (source.graph.coverage.status !== "complete")
limitations.add(
`Application graph ${source.graph.graph_id} coverage is ${source.graph.coverage.status}; omitted candidates remain unknown.`,
+22 -12
View File
@@ -1,10 +1,12 @@
import { createHash } from "node:crypto";
import { posix } from "node:path";
import { logicalPathEscapesRoot } from "../domain/artifactIdentity.js";
import {
logicalPathEscapesRoot,
normalizeJoinedLogicalPath,
} from "../domain/artifactIdentity.js";
createReferenceSourcePathLookup,
isPortableAbsoluteReferenceTarget,
type ReferenceSourcePathLookup,
} from "../domain/referenceSourcePathIdentity.js";
import {
classifyReferenceSourcePath,
detectReferenceSourceLanguage,
@@ -83,11 +85,11 @@ const failedEntry = (
const resolveInternalSpecifier = (
fromPath: string,
specifier: string,
filePaths: ReadonlySet<string>,
lookupPath: ReferenceSourcePathLookup,
): { to: string; resolution: "internal" | "unresolved" } => {
if (posix.isAbsolute(specifier))
return { to: specifier, resolution: "unresolved" };
const normalized = normalizeJoinedLogicalPath(
const normalized = posix.normalize(
posix.join(posix.dirname(fromPath), specifier),
);
if (logicalPathEscapesRoot(normalized))
@@ -113,7 +115,7 @@ const resolveInternalSpecifier = (
const stem = normalized.slice(0, -posix.extname(normalized).length);
candidates.push(...sourceSuffixes.map((suffix) => `${stem}${suffix}`));
}
const match = candidates.find((candidate) => filePaths.has(candidate));
const match = lookupPath.resolveCandidates(candidates);
return match === undefined
? { to: normalized, resolution: "unresolved" }
: { to: match, resolution: "internal" };
@@ -121,7 +123,7 @@ const resolveInternalSpecifier = (
const appendFile = (
entry: Extract<ReferenceSourceEntry, { status: "read"; kind: "file" }>,
filePaths: ReadonlySet<string>,
lookupPath: ReferenceSourcePathLookup,
output: ParsedReferenceSourceEntries,
): void => {
const language = detectReferenceSourceLanguage(entry.path);
@@ -146,7 +148,7 @@ const appendFile = (
const resolved = resolveInternalSpecifier(
relationship.from_path,
relationship.to,
filePaths,
lookupPath,
);
output.relationships.push({
...relationship,
@@ -159,11 +161,11 @@ const appendFile = (
const appendReadEntry = (
entry: Extract<ReferenceSourceEntry, { status: "read" }>,
filePaths: ReadonlySet<string>,
lookupPath: ReferenceSourcePathLookup,
output: ParsedReferenceSourceEntries,
): void => {
if (entry.kind === "file") {
appendFile(entry, filePaths, output);
appendFile(entry, lookupPath, output);
return;
}
const classifications = classifyReferenceSourcePath(entry.path);
@@ -180,7 +182,14 @@ const appendReadEntry = (
output.entries.push({
path: entry.path,
kind: "symlink",
target: entry.target,
// The reader resolves inventory identities from its root; the persisted
// graph represents relative symlink targets from the link's directory.
target:
entry.targetState === "internal" ||
(entry.targetState === "missing" &&
!isPortableAbsoluteReferenceTarget(entry.target))
? posix.relative(posix.dirname(entry.path), entry.target) || "."
: entry.target,
target_state: entry.targetState,
classifications,
limitations: [],
@@ -199,10 +208,11 @@ export const parseReferenceSourceEntries = (
parseFailures: [],
limitations: [...read.limitations],
};
const lookupPath = createReferenceSourcePathLookup(filePaths);
for (const entry of read.entries) {
if (signal?.aborted === true) break;
if (entry.status === "failed") output.entries.push(failedEntry(entry));
else appendReadEntry(entry, filePaths, output);
else appendReadEntry(entry, lookupPath, output);
}
return output;
};
+36 -9
View File
@@ -1,7 +1,8 @@
import { readFile, realpath, stat } from "node:fs/promises";
import { lstat, realpath, stat } from "node:fs/promises";
import { join, resolve } from "node:path";
import ignore from "ignore";
import { readRegularFileText } from "./RegularFileRead.js";
import { err, ok, type Result } from "../domain/result.js";
import {
@@ -47,14 +48,27 @@ const resolveRoot = async (
const buildIgnored = async (
root: string,
excludePaths: readonly string[],
signal?: AbortSignal,
): Promise<ReturnType<typeof ignore>> => {
const ignored = ignore();
const policyPath = join(root, ".gitignore");
let present = true;
try {
ignored.add(await readFile(join(root, ".gitignore"), "utf8"));
await lstat(policyPath);
} catch (cause: unknown) {
// Missing or unreadable ignore file does not authorize broader access.
void cause;
if (
!(
typeof cause === "object" &&
cause !== null &&
"code" in cause &&
cause.code === "ENOENT"
)
)
throw cause;
present = false;
}
signal?.throwIfAborted();
if (present) ignored.add(await readRegularFileText(policyPath, { signal }));
ignored.add([...DEFAULT_REFERENCE_SOURCE_IGNORE_PATTERNS]);
for (const path of excludePaths) {
ignored.add(path);
@@ -71,9 +85,22 @@ export const prepareReferenceSourceImport = async (
> => {
const root = await resolveRoot(options.root);
if (!root.ok) return root;
return ok({
root: root.value,
ignored: await buildIgnored(root.value, options.excludePaths ?? []),
secrets: ignore().add([...options.policy.secretPatterns]),
});
try {
return ok({
root: root.value,
ignored: await buildIgnored(
root.value,
options.excludePaths ?? [],
options.signal,
),
secrets: ignore().add([...options.policy.secretPatterns]),
});
} catch (cause: unknown) {
return err(
failure(
options.signal?.aborted === true ? "cancelled" : "io",
`Reference source ignore policy could not be read at ${join(root.value, ".gitignore")}: ${cause instanceof Error ? cause.message : String(cause)}`,
),
);
}
};
+62 -17
View File
@@ -1,6 +1,7 @@
import fs from "node:fs";
import { lstat, readFile } from "node:fs/promises";
import { lstat } from "node:fs/promises";
import { join, resolve } from "node:path";
import { readRegularFile, readRegularFileText } from "./RegularFileRead.js";
export type ReferenceSourceVcsInfo =
| {
@@ -29,7 +30,7 @@ export const readReferenceSourceVcs = async (
: { kind: "unknown", head: null, dirty: null };
}
try {
const head = await resolveSourceHead(root);
const head = await resolveSourceHead(root, signal);
if (isAborted(signal)) return { kind: "unknown", head: null, dirty: null };
return { kind: "git", head, dirty: null };
} catch (cause: unknown) {
@@ -39,13 +40,59 @@ export const readReferenceSourceVcs = async (
}
};
const resolveSourceHead = async (root: string): Promise<string> => {
const resolveSourceHead = async (
root: string,
signal?: AbortSignal,
): Promise<string> => {
// isomorphic-git is loaded on first use so CLI and MCP startup skip it.
const { resolveRef } = await import("isomorphic-git");
let failedRead: { readonly cause: unknown } | undefined;
const referenceFs = {
...fs,
promises: {
...fs.promises,
// isomorphic-git treats every read failure as absence. Preserve real
// failures so a readable packed ref cannot hide an unreadable loose ref.
readFile: async (
path: string | undefined,
options:
| BufferEncoding
| { readonly encoding?: BufferEncoding | null }
| null = null,
) => {
// The library probes readFile() without a path to detect promise APIs.
// Reject that invocation without attributing it to a source read.
if (path === undefined)
throw new TypeError("Git filesystem read requires a path");
try {
const bytes = await readRegularFile(path, { signal });
const encoding =
typeof options === "string" ? options : options?.encoding;
return encoding === undefined || encoding === null
? bytes
: bytes.toString(encoding);
} catch (cause: unknown) {
if (errorCode(cause) !== "ENOENT") failedRead ??= { cause };
throw cause;
}
},
},
};
const readRef = async (options: {
readonly gitdir?: string;
readonly ref: string;
}) => {
const head = await resolveRef({ fs: referenceFs, dir: root, ...options });
signal?.throwIfAborted();
if (failedRead !== undefined) throw failedRead.cause;
return head;
};
const dotgit = join(root, ".git");
if (!(await lstat(dotgit)).isFile())
return resolveRef({ fs, dir: root, ref: "HEAD" });
const pointer = (await readFile(dotgit, "utf8")).replace(/\r?\n$/u, "");
if (!(await lstat(dotgit)).isFile()) return readRef({ ref: "HEAD" });
const pointer = (await readRegularFileText(dotgit, { signal })).replace(
/\r?\n$/u,
"",
);
if (!pointer.startsWith("gitdir: ") || pointer.length === 8)
throw new Error("Invalid Git directory pointer");
const gitdir = resolve(root, pointer.slice(8));
@@ -53,25 +100,23 @@ const resolveSourceHead = async (root: string): Promise<string> => {
try {
commonDirectory = resolve(
gitdir,
(await readFile(join(gitdir, "commondir"), "utf8")).replace(
/\r?\n$/u,
"",
),
(
await readRegularFileText(join(gitdir, "commondir"), { signal })
).replace(/\r?\n$/u, ""),
);
} catch (cause: unknown) {
if (errorCode(cause) !== "ENOENT") throw cause;
return resolveRef({ fs, dir: root, gitdir, ref: "HEAD" });
return readRef({ gitdir, ref: "HEAD" });
}
const head = (await readFile(join(gitdir, "HEAD"), "utf8")).trim();
if (!head.startsWith("ref: "))
return resolveRef({ fs, dir: root, gitdir, ref: "HEAD" });
const head = (
await readRegularFileText(join(gitdir, "HEAD"), { signal })
).trim();
if (!head.startsWith("ref: ")) return readRef({ gitdir, ref: "HEAD" });
const ref = head.slice(5);
const privateRef = ["refs/bisect/", "refs/rewritten/", "refs/worktree/"].some(
(prefix) => ref.startsWith(prefix),
);
return resolveRef({
fs,
dir: root,
return readRef({
gitdir: privateRef ? gitdir : commonDirectory,
ref,
});
+49 -6
View File
@@ -1,20 +1,44 @@
import type { Stats } from "node:fs";
import type { FileHandle } from "node:fs/promises";
import { lstat, stat, type FileHandle } from "node:fs/promises";
import { openRegularFile } from "../filesystem/RegularFile.js";
import {
openRegularFile,
sameRegularFileState,
RegularFileChangedError,
} from "../filesystem/RegularFile.js";
import { readBoundedFileBytes } from "../process/BoundedFileBytes.js";
/** Filesystem admission policy for one selected regular input. */
export interface RegularFileReadOptions {
readonly signal?: AbortSignal | undefined;
readonly symlinks?: "follow" | "reject";
}
/** Admit one regular-file handle, retain it through the consumer, and always close it. */
export const withRegularFile = async <Value>(
path: string,
read: (handle: FileHandle, stats: Stats) => Promise<Value>,
signal?: AbortSignal,
options: RegularFileReadOptions = {},
): Promise<Value> => {
const handle = await openRegularFile(path, { symlinks: "follow", signal });
const { signal } = options;
const handle = await openRegularFile(path, {
symlinks: options.symlinks ?? "follow",
signal,
});
try {
const stats = await handle.stat();
signal?.throwIfAborted();
const value = await read(handle, stats);
signal?.throwIfAborted();
const [opened, currentPath] = await Promise.all([
handle.stat(),
options.symlinks === "reject" ? lstat(path) : stat(path),
]);
if (
!sameRegularFileState(stats, opened) ||
!sameRegularFileState(stats, currentPath)
)
throw new RegularFileChangedError(path);
return value;
} finally {
await handle.close();
@@ -24,6 +48,25 @@ export const withRegularFile = async <Value>(
/** Admit a regular file without waiting for a pipe, then read its verified handle. */
export const readRegularFile = (
path: string,
signal?: AbortSignal,
options: RegularFileReadOptions = {},
): Promise<Buffer> =>
withRegularFile(path, (handle) => handle.readFile({ signal }), signal);
withRegularFile(
path,
async (handle, stats) => {
const bytes = await readBoundedFileBytes(
handle,
stats.size,
options.signal,
);
if (bytes === undefined || bytes.length !== stats.size)
throw new RegularFileChangedError(path);
return bytes;
},
options,
);
/** Read one admitted regular file as UTF-8 without waiting on a pipe. */
export const readRegularFileText = async (
path: string,
options: RegularFileReadOptions = {},
): Promise<string> => (await readRegularFile(path, options)).toString("utf8");
+59 -6
View File
@@ -29,6 +29,49 @@ export const runSetup = async (
options: SetupOptions,
host: SetupHost = systemSetupHost(),
confirm?: SetupConfirmation,
): Promise<SetupResult> => {
let result: SetupResult | undefined;
let primaryFailure: unknown;
let threw = false;
try {
result = await runSetupWorkflow(options, host, confirm);
} catch (cause: unknown) {
threw = true;
primaryFailure = cause;
}
let cleanupFailure: string | undefined;
try {
cleanupFailure = await host.close?.();
} catch (cause: unknown) {
cleanupFailure = `Setup resource cleanup failed: ${failureMessage(cause)}`;
}
if (threw) {
if (cleanupFailure !== undefined)
throw new AggregateError(
[primaryFailure, new Error(cleanupFailure)],
"Setup and resource cleanup failed.",
{ cause: primaryFailure },
);
throw primaryFailure;
}
if (result === undefined) throw new Error("Setup returned no result.");
if (cleanupFailure === undefined) return result;
return {
...result,
status: "needs_human",
code: result.code ?? "cleanup_failed",
remediation: result.remediation
? `${result.remediation} ${cleanupFailure}`
: cleanupFailure,
};
};
const runSetupWorkflow = async (
options: SetupOptions,
host: SetupHost,
confirm?: SetupConfirmation,
): Promise<SetupResult> => {
const appliedActions: string[] = [];
let plannedActions: readonly SetupAction[] = [];
@@ -59,6 +102,7 @@ export const runSetup = async (
options.readinessScope?.clients === undefined &&
options.allDetectedClients !== true &&
options.installHopper !== true &&
options.skillClientIds === undefined &&
options.installSkill === undefined;
const interactiveSelection =
confirm !== undefined && !options.structured && clientSelectionAllowed;
@@ -67,10 +111,12 @@ export const runSetup = async (
: (options.clientIds ??
options.readinessScope?.clients ??
discovery.defaultClientIds);
let skillNeedsInstall = await host.skillNeedsInstall(selectedClientIds);
let selectedSkillClientIds = options.skillClientIds ?? selectedClientIds;
let skillNeedsInstall = await host.skillNeedsInstall(selectedSkillClientIds);
let skillSelected =
options.installSkill === true ||
(options.installSkill !== false && selectedClientIds.length > 0);
(options.installSkill !== false &&
(selectedClientIds.length > 0 || selectedSkillClientIds.length > 0));
let installSkill = skillSelected && skillNeedsInstall;
let planDiscovery = discovery;
if (interactiveSelection) {
@@ -138,6 +184,7 @@ export const runSetup = async (
selectedClientIds.length === 0) ||
(options.installSkill !== false && selectedClientIds.length > 0);
installSkill = skillSelected && skillNeedsInstall;
selectedSkillClientIds = selectedClientIds;
}
const planned = await planSetupActions({
discovery: planDiscovery,
@@ -145,7 +192,7 @@ export const runSetup = async (
providerEnvironment,
command: host.registrationCommand,
clientIds: selectedClientIds,
skillClientIds: selectedClientIds,
skillClientIds: selectedSkillClientIds,
installSkill,
});
plannedActions = planned.plannedActions;
@@ -256,7 +303,7 @@ export const runSetup = async (
host,
options,
appliedActions,
selectedClientIds,
selectedSkillClientIds,
))
)
return fail(
@@ -325,8 +372,11 @@ const installHopperAction = async (input: {
emitProgress(input.options, {
actionId: "install_hopper",
label,
state: "completed",
detail: installed.launcherPath,
state: installed.cleanupFailure === undefined ? "completed" : "warning",
detail:
installed.cleanupFailure === undefined
? installed.launcherPath
: `${installed.launcherPath}; ${installed.cleanupFailure}`,
});
const providerEnvironment = {
...input.providerEnvironment,
@@ -492,3 +542,6 @@ const summarizeDoctor = (report: DoctorReport): SetupDoctorSummary => ({
},
}),
});
const failureMessage = (cause: unknown): string =>
cause instanceof Error ? cause.message : String(cause);
+6 -5
View File
@@ -14,7 +14,7 @@ import {
type ClientRegistrationDialect,
} from "./ClientConfigurationDocument.js";
import { constants as fsConstants } from "node:fs";
import { copyFile, mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { copyFile, mkdir, rm, writeFile } from "node:fs/promises";
import { dirname } from "node:path";
import writeFileAtomic from "write-file-atomic";
@@ -22,6 +22,7 @@ import { PRODUCT_IDENTITY } from "../identity.js";
import { npxRegistrationCommand } from "./ClientRegistrationIdentity.js";
import { MCP_STARTUP_POLICY } from "../mcpStartupPolicy.js";
import { resolveClientConfigTransactionPath } from "./ClientConfigPath.js";
import { readRegularFileText } from "./RegularFileRead.js";
import type {
ClientConfigurationInspection,
ClientConfigurationResult,
@@ -64,7 +65,7 @@ const configureClientDocument = async (
return { status: "failed", reason: "path" };
let original: string | undefined;
try {
original = await readFile(transactionPath, "utf8");
original = await readRegularFileText(transactionPath);
} catch (cause: unknown) {
if (!isMissing(cause)) return { status: "failed", reason: "readback" };
}
@@ -149,7 +150,7 @@ const configureClientDocument = async (
}
try {
const readback = parseClientConfiguration(
await readFile(transactionPath, "utf8"),
await readRegularFileText(transactionPath),
format,
);
if (!registrationCurrent(readback, desired)) {
@@ -178,7 +179,7 @@ export const clientConfigurationAligned = async (
try {
const files = await readClientConfigurationFiles(client);
if (!files.ok) return false;
const original = await readFile(client.configPath, "utf8");
const original = await readRegularFileText(client.configPath);
const parsed = parseClientConfiguration(original, client.format);
const policy = await readClientPolicyBlock(
client,
@@ -246,7 +247,7 @@ export const inspectClientConfiguration = async (
};
let original: string;
try {
original = await readFile(transactionPath, "utf8");
original = await readRegularFileText(transactionPath);
} catch (cause: unknown) {
if (isMissing(cause)) return { status: "create" };
return {
+32 -9
View File
@@ -94,6 +94,10 @@ export const systemSetupHost = (
const doctorHost = selectedDoctorHost ?? systemDoctorHost({ environment });
const platform = doctorHost.platform;
const { homeDirectory } = doctorHost;
const macHopperHost =
platform === "darwin"
? systemMacHopperInstallHost(homeDirectory, environment)
: undefined;
return {
platform,
homeDirectory,
@@ -106,6 +110,7 @@ export const systemSetupHost = (
nodeVersion: process.versions.node,
macosVersion: () => doctorHost.macosVersion(),
linuxDistribution: readLinuxDistribution,
close: () => macHopperHost?.close() ?? Promise.resolve(undefined),
initialSetupState: async (
scope?: DoctorScope,
): Promise<SetupInitialState> => {
@@ -126,15 +131,33 @@ export const systemSetupHost = (
};
},
installHopper: async (replaceExisting) => {
const result =
platform === "linux"
? await installLinuxHopper(systemLinuxHopperInstallHost(environment))
: await installMacHopper(
{ replaceExisting },
systemMacHopperInstallHost(homeDirectory, environment),
);
if (result.status === "installed") return result;
return setupInstallFailure(result.reason);
if (platform === "linux") {
const result = await installLinuxHopper(
systemLinuxHopperInstallHost(environment),
);
return result.status === "installed"
? result
: setupInstallFailure(result.reason);
}
if (macHopperHost === undefined)
return setupInstallFailure("unsupported_host");
const result = await installMacHopper({ replaceExisting }, macHopperHost);
if (result.status === "installed")
return {
status: "installed",
launcherPath: result.launcherPath,
...(result.cleanupFailure === undefined
? {}
: { cleanupFailure: result.cleanupFailure }),
};
const failure = setupInstallFailure(result.reason);
if (result.cleanupFailure === undefined) return failure;
return failure.status === "failed"
? {
...failure,
remediation: `${failure.remediation} ${result.cleanupFailure}`,
}
: failure;
},
detectedClients: () => detectClients(homeDirectory, platform, environment),
supportedClients: () =>
+7 -2
View File
@@ -10,10 +10,15 @@ export type SetupFailureCode =
| "destination_exists"
| "mount_failed"
| "bundle_invalid"
| "copy_failed";
| "copy_failed"
| "cleanup_failed";
export type SetupHopperInstallResult =
| { readonly status: "installed"; readonly launcherPath: string }
| {
readonly status: "installed";
readonly launcherPath: string;
readonly cleanupFailure?: string;
}
| {
readonly status: "failed";
readonly code: SetupFailureCode;
+59 -7
View File
@@ -1,11 +1,14 @@
import { mkdir, readFile, rm } from "node:fs/promises";
import { copyFile, lstat, mkdir, readFile, rm } from "node:fs/promises";
import { constants as fsConstants } from "node:fs";
import { dirname, join } from "node:path";
import writeFileAtomic from "write-file-atomic";
import { PRODUCT_IDENTITY } from "../identity.js";
import { NonRegularFileReadError } from "../filesystem/RegularFile.js";
import { readRegularFileText } from "./RegularFileRead.js";
import { clientSkillDirectories } from "./SupportedClients.js";
const SKILL_FILES = [
export const MANAGED_SKILL_FILES = [
"SKILL.md",
"references/native-and-artifacts.md",
"references/javascript-applications.md",
@@ -14,6 +17,13 @@ const SKILL_FILES = [
"references/evidence-workflows.md",
] as const;
export const isManagedSkillManifest = (content: string): boolean =>
content.startsWith(`---\nname: ${PRODUCT_IDENTITY.skillName}\n`) ||
content.startsWith(`---\r\nname: ${PRODUCT_IDENTITY.skillName}\r\n`);
export const readSkillFile = (path: string): Promise<string> =>
readRegularFileText(path, { symlinks: "reject" });
interface CanonicalSkillFile {
readonly content: string;
readonly destination: string;
@@ -47,9 +57,38 @@ export const skillDestinations = (
}),
);
/** Resolve existing REA-owned skill roots using setup's canonical destinations. */
export const existingSkillDestinations = async (
home: string,
environment: Readonly<NodeJS.ProcessEnv> = {},
platform: NodeJS.Platform = process.platform,
): Promise<readonly SkillDestination[]> => {
const existing: SkillDestination[] = [];
for (const destination of skillDestinations(
home,
undefined,
environment,
platform,
)) {
try {
const root = await lstat(destination.path);
if (root.isSymbolicLink() || !root.isDirectory()) continue;
const manifest = join(destination.path, "SKILL.md");
const content = await readSkillFile(manifest);
if (isManagedSkillManifest(content)) existing.push(destination);
} catch (cause: unknown) {
if (cause instanceof Error && "code" in cause && cause.code === "ENOENT")
continue;
if (cause instanceof NonRegularFileReadError) continue;
throw cause;
}
}
return existing;
};
const readOptionalText = async (path: string): Promise<string | undefined> => {
try {
return await readFile(path, "utf8");
return await readSkillFile(path);
} catch (cause: unknown) {
if (cause instanceof Error && "code" in cause && cause.code === "ENOENT")
return undefined;
@@ -66,7 +105,7 @@ const canonicalSkillFiles = async (
Promise.all(
skillDestinations(home, clientIds, environment, platform).flatMap(
({ path: root }) =>
SKILL_FILES.map(async (relativePath) => {
MANAGED_SKILL_FILES.map(async (relativePath) => {
const destination = join(root, relativePath);
return {
destination,
@@ -104,6 +143,19 @@ export const canonicalSkillNeedsInstall = async (
const writeText = (path: string, content: string): Promise<void> =>
writeFileAtomic(path, content, { encoding: "utf8", mode: 0o600 });
const preserveSkillBackup = async (
source: string,
destination: string,
): Promise<void> => {
try {
await copyFile(source, destination, fsConstants.COPYFILE_EXCL);
} catch (cause: unknown) {
if (cause instanceof Error && "code" in cause && cause.code === "EEXIST")
return;
throw cause;
}
};
const restoreSkillFiles = async (
changed: readonly CanonicalSkillFile[],
): Promise<void> => {
@@ -134,12 +186,12 @@ export const installCanonicalSkill = async (
for (const { destination, original } of changed) {
await mkdir(dirname(destination), { recursive: true });
if (original !== undefined)
await writeText(`${destination}.rea.backup`, original);
await preserveSkillBackup(destination, `${destination}.rea.backup`);
}
for (const { destination, content } of changed)
await writeText(destination, content);
for (const { destination, content } of changed)
if ((await readFile(destination, "utf8")) !== content)
if ((await readSkillFile(destination)) !== content)
throw new Error(`skill readback mismatch: ${destination}`);
return "installed";
} catch (cause: unknown) {
@@ -171,7 +223,7 @@ export const readInstalledSkillIdentity = async (
let content: string | undefined;
for (const destination of destinations) {
try {
content = await readFile(join(destination.path, "SKILL.md"), "utf8");
content = await readSkillFile(join(destination.path, "SKILL.md"));
break;
} catch {
// Another selected copy may provide metadata; canonical validation below
+4
View File
@@ -50,6 +50,8 @@ export interface SetupHost {
linuxDistribution(): Promise<LinuxDistribution | undefined>;
initialSetupState(scope?: DoctorScope): Promise<SetupInitialState>;
installHopper(replaceExisting: boolean): Promise<SetupHopperInstallResult>;
/** Retry and report any cleanup resources still owned by this setup host. */
close?(): Promise<string | undefined>;
detectedClients(): Promise<readonly SetupClient[]>;
supportedClients(): Promise<readonly SetupClient[]>;
configureClient(
@@ -155,6 +157,8 @@ export interface SetupOptions {
readonly structured: boolean;
readonly proposeHopper?: boolean;
readonly clientIds?: readonly string[];
/** Skill roots to update independently of MCP registrations (maintenance). */
readonly skillClientIds?: readonly string[];
readonly installSkill?: boolean;
readonly readinessScope?: DoctorScope;
readonly dryRun?: boolean;
+155 -6
View File
@@ -6,7 +6,7 @@ import {
withClientServers,
type ClientConfigurationDocument,
} from "./ClientConfigurationDocument.js";
import { copyFile, lstat, readFile, realpath, rm } from "node:fs/promises";
import { copyFile, lstat, realpath, rm } from "node:fs/promises";
import { constants as fsConstants } from "node:fs";
import { homeDirectoryFromEnvironment } from "../config/homeDirectory.js";
import { join } from "node:path";
@@ -15,9 +15,15 @@ import writeFileAtomic from "write-file-atomic";
import { z } from "zod";
import { PRODUCT_IDENTITY } from "../identity.js";
import { skillDestinations } from "./SetupSkill.js";
import {
isManagedSkillManifest,
MANAGED_SKILL_FILES,
readSkillFile,
skillDestinations,
} from "./SetupSkill.js";
import { isOwnedClientRegistrationCommand } from "./ClientRegistrationIdentity.js";
import { resolveClientConfigTransactionPath } from "./ClientConfigPath.js";
import { readRegularFileText } from "./RegularFileRead.js";
import {
manualRegistrationRemediation,
supportedClients,
@@ -27,29 +33,32 @@ import {
interface ManagedPathStats {
readonly uid?: number;
isFile?(): boolean;
isDirectory?(): boolean;
isSymbolicLink(): boolean;
}
/** Injectable filesystem operations used to prove uninstall failure recovery. */
export interface UninstallFileSystem {
readText(path: string): Promise<string>;
readRegularText?(path: string): Promise<string>;
/** Preserve the first backup; reject an existing destination with EEXIST. */
copy(source: string, destination: string): Promise<void>;
writeText(path: string, contents: string): Promise<void>;
stat(path: string): Promise<ManagedPathStats>;
realpath?(path: string): Promise<string>;
remove(path: string): Promise<void>;
remove(path: string, recursive?: boolean): Promise<void>;
}
const systemFileSystem: UninstallFileSystem = {
readText: (path) => readFile(path, "utf8"),
readText: (path) => readRegularFileText(path),
readRegularText: readSkillFile,
copy: (source, destination) =>
copyFile(source, destination, fsConstants.COPYFILE_EXCL),
writeText: (path, contents) =>
writeFileAtomic(path, contents, { encoding: "utf8" }),
stat: (path) => lstat(path),
realpath: (path) => realpath(path),
remove: (path) => rm(path, { recursive: true }),
remove: (path, recursive = true) => rm(path, { recursive }),
};
/** One explicitly classified uninstall action. */
@@ -422,7 +431,7 @@ const removeManagedSkills = async (
const results = await Promise.all(
skillDestinations(home, undefined, environment, platform).map(
({ client, path }) =>
removeManagedPath(
removeManagedSkill(
path,
client === "claude_code" ? "Claude Code skill" : "skill",
fileSystem,
@@ -457,6 +466,146 @@ const removeManagedSkills = async (
);
};
const removeManagedSkill = async (
path: string,
name: string,
fileSystem: UninstallFileSystem,
): Promise<UninstallItem> => {
let root: ManagedPathStats;
try {
root = await fileSystem.stat(path);
} catch (cause: unknown) {
return isMissing(cause)
? item(name, "skipped", "Skill directory does not exist.")
: item(name, "failed", "Skill directory could not be inspected.");
}
if (root.isSymbolicLink())
return item(name, "retained", "Skill directory is a symbolic link.");
if (root.isDirectory?.() !== true)
return item(name, "retained", "Skill path is not a directory.");
const manifestPath = join(path, "SKILL.md");
try {
const manifest = await fileSystem.stat(manifestPath);
if (manifest.isSymbolicLink())
return item(name, "retained", "Skill manifest is a symbolic link.");
if (manifest.isFile?.() !== true)
return item(name, "retained", "Skill manifest is not a regular file.");
} catch (cause: unknown) {
return isMissing(cause)
? item(name, "retained", "Skill ownership could not be established.")
: item(name, "failed", "Skill manifest could not be inspected.");
}
let content: string;
try {
content = await (fileSystem.readRegularText?.(manifestPath) ??
fileSystem.readText(manifestPath));
} catch (cause: unknown) {
return isMissing(cause)
? item(name, "retained", "Skill ownership could not be established.")
: item(name, "failed", "Skill manifest could not be read.");
}
if (!isManagedSkillManifest(content))
return item(name, "retained", "Skill manifest is not REA-owned.");
const manifestName = "SKILL.md";
const otherFiles = MANAGED_SKILL_FILES.filter(
(relativePath) => relativePath !== manifestName,
);
const results: UninstallItem[] = [];
for (const relativePath of otherFiles)
results.push(
await removeManagedSkillFile(path, relativePath, name, fileSystem),
);
if (
results.some(({ status }) => status === "failed" || status === "retained")
)
results.push(
item(
name,
"retained",
"REA ownership manifest was retained because a managed file could not be safely removed.",
),
);
else
results.push(
await removeManagedSkillFile(path, manifestName, name, fileSystem),
);
const failed = results.find(({ status }) => status === "failed");
if (failed !== undefined)
return item(name, "failed", results.map(({ detail }) => detail).join(" "));
const removed = results.some(({ status }) => status === "removed");
return item(
name,
removed ? "removed" : "skipped",
"REA-managed skill files were removed; other files were preserved.",
);
};
const removeManagedSkillFile = async (
root: string,
relativePath: string,
name: string,
fileSystem: UninstallFileSystem,
): Promise<UninstallItem> => {
const segments = relativePath.split(/[\\/]/u);
let parent = root;
for (const segment of segments.slice(0, -1)) {
parent = join(parent, segment);
try {
const stats = await fileSystem.stat(parent);
if (stats.isSymbolicLink() || stats.isDirectory?.() !== true)
return item(
name,
"retained",
`Managed path parent was retained: ${parent}.`,
);
} catch (cause: unknown) {
return isMissing(cause)
? item(
name,
"skipped",
`${join(parent, segments.at(-1) ?? "")} does not exist.`,
)
: item(
name,
"failed",
`Managed path parent could not be inspected: ${parent}.`,
);
}
}
const target = join(root, relativePath);
let stats: ManagedPathStats;
try {
stats = await fileSystem.stat(target);
} catch (cause: unknown) {
return isMissing(cause)
? item(name, "skipped", `${target} does not exist.`)
: item(name, "failed", `Managed file could not be inspected: ${target}.`);
}
if (stats.isSymbolicLink())
return item(
name,
"retained",
`Managed file is a symbolic link: ${target}.`,
);
if (stats.isFile?.() !== true)
return item(
name,
"retained",
`Managed path is not a regular file: ${target}.`,
);
try {
await fileSystem.remove(target, false);
return item(name, "removed", `Removed ${target}.`);
} catch (cause: unknown) {
return isMissing(cause)
? item(name, "skipped", `${target} does not exist.`)
: item(name, "failed", `Managed file could not be removed: ${target}.`);
}
};
const item = (
name: string,
status: UninstallItem["status"],
+27 -23
View File
@@ -1,5 +1,3 @@
import { readFile } from "node:fs/promises";
import { join } from "node:path";
import { z } from "zod";
import { PRODUCT_IDENTITY } from "../identity.js";
@@ -12,13 +10,19 @@ import {
parseClientConfiguration,
} from "./ClientConfigurationDocument.js";
import { supportedClients } from "./SupportedClients.js";
import { existingSkillDestinations } from "./SetupSkill.js";
import type { SetupAction } from "./SetupTypes.js";
import type { Result } from "../domain/result.js";
import { readRegularFileText } from "./RegularFileRead.js";
/** Existing REA integrations selected for maintenance, without discovering new targets. */
export interface MaintenanceScope {
readonly clients: readonly string[];
readonly skill: boolean;
readonly skillDestinations?: readonly {
readonly client: string;
readonly path: string;
}[];
}
/** Unapplied maintenance evidence produced by the updated setup executable. */
@@ -49,7 +53,7 @@ const setupPlanSchema = z.object({
),
});
/** Discover owned registrations and an existing REA skill without writes. */
/** Discover owned registrations and REA skills without writes. */
export const existingMaintenanceScope = async (
home: string,
entryPoint: string,
@@ -69,7 +73,7 @@ export const existingMaintenanceScope = async (
const client = supported.find(({ name }) => name === registration.client);
if (client === undefined) continue;
const parsed = parseClientConfiguration(
await readFile(client.configPath, "utf8"),
await readRegularFileText(client.configPath),
client.format,
);
const enabled = z
@@ -87,20 +91,12 @@ export const existingMaintenanceScope = async (
)
clients.push(client.name);
}
let skill = false;
try {
const content = await readFile(
join(home, ".agents", "skills", PRODUCT_IDENTITY.skillName, "SKILL.md"),
"utf8",
);
skill =
content.startsWith(`---\nname: ${PRODUCT_IDENTITY.skillName}\n`) ||
content.startsWith(`---\r\nname: ${PRODUCT_IDENTITY.skillName}\r\n`);
} catch (cause: unknown) {
if (!(cause instanceof Error && "code" in cause && cause.code === "ENOENT"))
throw cause;
}
return { clients, skill };
const skillDestinations = await existingSkillDestinations(home, environment);
return {
clients,
skill: skillDestinations.length > 0,
skillDestinations,
};
};
/** Ask the verified new executable to plan only existing REA integrations. */
@@ -112,6 +108,7 @@ export const planIntegrationMaintenance = async (
): Promise<IntegrationMaintenance> => {
try {
const scope = await existingMaintenanceScope(home, entryPoint, environment);
const skillDestinations = scope.skillDestinations ?? [];
if (scope.clients.length === 0 && !scope.skill)
return { status: "current", plannedActions: [] };
const command = [
@@ -119,6 +116,7 @@ export const planIntegrationMaintenance = async (
entryPoint,
"setup",
...scope.clients.flatMap((client) => ["--client", client]),
...skillDestinations.flatMap(({ client }) => ["--skill-client", client]),
`--skill=${String(scope.skill)}`,
];
const execution = await execute([...command, "--dry-run", "--json"]);
@@ -132,11 +130,17 @@ export const planIntegrationMaintenance = async (
};
if (plan.data.plannedActions.length === 0)
return { status: "current", plannedActions: [] };
const allowedIds = new Set([
...scope.clients.map((client) => `configure_client:${client}`),
...(scope.skill ? ["install_skill"] : []),
]);
if (plan.data.plannedActions.some(({ id }) => !allowedIds.has(id)))
const allowedIds = new Set(
scope.clients.map((client) => `configure_client:${client}`),
);
const skillTargets = new Set(skillDestinations.map(({ path }) => path));
if (
plan.data.plannedActions.some((action) =>
action.kind === "install_skill"
? !skillTargets.has(action.target)
: !allowedIds.has(action.id),
)
)
return {
status: "unavailable",
remediation:
+6 -2
View File
@@ -27,7 +27,10 @@ import { WebScriptExportError } from "../domain/webScriptExportError.js";
import type { ExecutionOptions } from "./AnalysisProvider.js";
import { WEB_SCRIPT_EXPORT_PROVIDER as PROVIDER } from "./InvestigationProviders.js";
import { readRegularFile } from "./RegularFileRead.js";
import { NonRegularFileReadError } from "../filesystem/RegularFile.js";
import {
NonRegularFileReadError,
RegularFileChangedError,
} from "../filesystem/RegularFile.js";
const OPERATION = "export_web_scripts";
@@ -153,7 +156,7 @@ const readCapture = async (
signal: AbortSignal | undefined,
): Promise<Result<Buffer, AnalysisError>> => {
try {
return ok(await readRegularFile(path, signal));
return ok(await readRegularFile(path, { signal }));
} catch (cause: unknown) {
const code =
cause instanceof Error && "code" in cause ? String(cause.code) : "";
@@ -163,6 +166,7 @@ const readCapture = async (
);
if (
cause instanceof NonRegularFileReadError ||
cause instanceof RegularFileChangedError ||
code === "ENOENT" ||
code === "ENOTDIR" ||
code === "EISDIR" ||
@@ -28,6 +28,7 @@ export interface WebSourceLocationArtifactPort {
/** Replaceable codec receives map data and one explicit generated point. */
export interface WebSourceMapPort {
close?(): Promise<void>;
trace(
input: {
readonly text: string;
@@ -59,6 +59,10 @@ export class WebSourceLocationService {
readonly decoder: WebSourceMapPort,
) {}
close(): Promise<void> {
return this.decoder.close?.() ?? Promise.resolve();
}
/** Shared CLI/MCP workflow preserves raw declarations and every equal-position match. */
async trace(
rawInput: unknown,
+1 -1
View File
@@ -14,7 +14,7 @@ import type { Result } from "../../domain/result.js";
export interface AdbDeviceAnalysisPort {
/** Probe the selected adb binary without contacting devices. */
inspectAvailability(signal?: AbortSignal): Promise<ProviderAvailability>;
/** The provider holds no long-lived resources; safe to call always. */
/** Terminally stop admission, drain work, and retry retained package rollback. */
close(): Promise<void>;
execute(
request: AdbRequest,
@@ -5,10 +5,8 @@ import { ProviderAdapterError } from "../../domain/providerAdapterError.js";
import { ProviderCleanupError } from "../../domain/providerCleanupError.js";
import { err } from "../../domain/result.js";
import type { AnalysisClient } from "../AnalysisProvider.js";
import {
analysisErrorWithCleanupFailure,
closeAnalysisClient,
} from "./AnalysisClientCleanup.js";
import { closeAnalysisClient } from "./AnalysisClientCleanup.js";
import { analysisErrorWithCleanupFailure } from "../../domain/analysisErrorCleanup.js";
const clientClosingWith = (cause: unknown): AnalysisClient => ({
execute: () => Promise.resolve(err(new ProviderCleanupError("x", [], {}))),
@@ -1,6 +1,4 @@
import { ProviderCleanupError } from "../../domain/providerCleanupError.js";
import { ProviderAdapterError } from "../../domain/providerAdapterError.js";
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
import { err, type Result } from "../../domain/result.js";
import { AnalysisError } from "../../domain/analysisErrorBase.js";
import type { AnalysisClient, ExecutionOptions } from "../AnalysisProvider.js";
@@ -29,34 +27,3 @@ export const closeAnalysisClient = async (
);
}
};
/** Preserve the original lifecycle failure alongside incomplete cleanup. */
export const analysisErrorWithCleanupFailure = (
primary: AnalysisError,
cleanup: AnalysisError,
operation = "close_binary",
): AnalysisError => {
const projected = projectAnalysisError(cleanup);
return new ProviderAdapterError(
cleanup instanceof ProviderAdapterError ? cleanup.providerId : "rea",
operation,
{
cause: primary,
userMessage: projectAnalysisError(primary).message,
...(primary.capturedOutput === undefined
? {}
: { capturedOutput: primary.capturedOutput }),
...(primary.partialObservation === undefined
? {}
: { partialObservation: primary.partialObservation }),
cleanup: {
reason: projected.message,
resources: cleanup.cleanupResources,
},
diagnostics: {
primary_error: projectAnalysisError(primary),
cleanup_error: projected,
},
},
);
};
+2 -4
View File
@@ -44,10 +44,8 @@ import {
commitExecutionProfile,
prepareSessionExecution,
} from "./BinarySessionExecution.js";
import {
analysisErrorWithCleanupFailure,
closeAnalysisClient,
} from "./AnalysisClientCleanup.js";
import { closeAnalysisClient } from "./AnalysisClientCleanup.js";
import { analysisErrorWithCleanupFailure } from "../../domain/analysisErrorCleanup.js";
const OFFICIAL_OPERATIONS: ReadonlySet<string> = new Set(
OFFICIAL_TOOL_CONTRACTS.map(({ name }) => name),
);
+1
View File
@@ -8,6 +8,7 @@ import type { Result } from "../../domain/result.js";
/** Replaceable offline interface-recovery engine, independent of wallets and chain clients. */
export interface EvmInterfacePort {
close?(): Promise<void>;
inspect(
input: InspectEvmInterfaceInput,
options?: ExecutionOptions,
@@ -25,6 +25,10 @@ const diagnosticsSchema = z.object({
export class EvmInterfaceService {
constructor(readonly provider: EvmInterfacePort) {}
close(): Promise<void> {
return this.provider.close?.() ?? Promise.resolve();
}
/** Preserve selected byte identity while clearly labeling recovered ABI candidates as inferences. */
async inspect(
rawInput: unknown,
@@ -8,6 +8,7 @@ import type { Result } from "../../domain/result.js";
/** File-oriented firmware engines; executable admission remains a separate operation. */
export interface FirmwareAnalysisPort {
close?(): Promise<void>;
execute(
request: FirmwareRequest,
options?: ExecutionOptions,
@@ -18,6 +18,10 @@ import { err, ok, type Result } from "../../domain/result.js";
export class FirmwareAnalysisService {
constructor(readonly provider: FirmwareAnalysisPort) {}
close(): Promise<void> {
return this.provider.close?.() ?? Promise.resolve();
}
/** Execute an explicit inspection or extraction and preserve its producing identity. */
async execute(
operation: FirmwareOperation,
@@ -2,9 +2,10 @@ import type { ApplicationNode } from "../../domain/javascript/javascriptApplicat
import { sha256Text } from "../../domain/javascript/javascriptStaticAnalysisHelpers.js";
import type { JavaScriptSourceRange } from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
import type {
JavaScriptArtifactGraphContext,
JavaScriptArtifactGraphCoverage,
import {
sourceNodeFor,
type JavaScriptArtifactGraphContext,
type JavaScriptArtifactGraphCoverage,
} from "./JavaScriptArtifactGraphContext.js";
import {
astObservationEvidence,
@@ -38,10 +39,11 @@ export const electronFindingSourceNode = (
context: JavaScriptArtifactGraphContext,
file: JavaScriptArtifactFile,
moduleKey: string | null,
runtime?: string,
): ApplicationNode | undefined =>
moduleKey === null
? (context.assetNodes.get(file.path) ?? context.fileNodes.get(file.path))
: context.moduleNodes.get(`${file.path}\0${moduleKey}`);
: sourceNodeFor(context, file.path, moduleKey, runtime);
/** Observation-scoped identity shared safely across different artifact files. */
export const electronObservationIdentity = (
@@ -73,6 +73,7 @@ const addIpcRecord = (
context,
record.file,
record.finding.module_key,
record.finding.module_runtime,
);
if (source === undefined) return;
state.sources.set(record.key, source);
@@ -255,6 +256,7 @@ const addValidationObservations = (state: IpcGraphState): void => {
state.context,
file,
validation.module_key,
validation.module_runtime,
)
: state.handlers.get(handlerRecord.key);
if (target !== undefined)
@@ -8,6 +8,7 @@ import {
} from "./JavaScriptArtifactGraphContext.js";
import { astObservationEvidence } from "./JavaScriptArtifactGraphEvidence.js";
import { compositeKey } from "../../domain/unicodeCodePointOrder.js";
import { javascriptDisplayText } from "../../domain/javascript/javascriptAstValues.js";
import {
addElectronInferenceEdge,
electronFindingSourceNode,
@@ -37,7 +38,12 @@ export const addElectronNativeBoundaries = (
const addNativeBinding = (input: NativeBindingInput): void => {
const { context, file, value, coverage } = input;
const source = electronFindingSourceNode(context, file, value.module_key);
const source = electronFindingSourceNode(
context,
file,
value.module_key,
value.module_runtime,
);
if (source === undefined) return;
const resolved = resolveArtifactPath(
value.specifier,
@@ -51,17 +57,27 @@ const addNativeBinding = (input: NativeBindingInput): void => {
identity: electronObservationIdentity(
context,
"javascript-requested-native-export",
// Members are a variable-length source-derived list: compositeKey keeps
// ["a\0b"] distinct from ["a", "b"] inside the digested identity.
`${file.path}:${value.specifier}:${value.binding_kind}:${compositeKey(value.members)}:${electronRangeKey(value.location)}`,
compositeKey([
file.path,
value.specifier,
value.binding_kind,
value.members,
value.namespace_access ?? null,
value.dynamic_member_access ?? null,
electronRangeKey(value.location),
]),
),
observations: [
{
label: value.members.join(", "),
label:
value.members.map(javascriptDisplayText).join(", ") ||
value.specifier,
properties: {
specifier: value.specifier,
binding_kind: value.binding_kind,
requested_members: value.members,
namespace_access: value.namespace_access ?? null,
dynamic_member_access: value.dynamic_member_access ?? null,
resolved_path: resolved,
native_export_verification: "not-performed",
},
@@ -90,6 +106,8 @@ const addNativeBinding = (input: NativeBindingInput): void => {
specifier: value.specifier,
binding_kind: value.binding_kind,
requested_members: value.members,
namespace_access: value.namespace_access ?? null,
dynamic_member_access: value.dynamic_member_access ?? null,
},
limitations: [
"Requested JavaScript members are not verified native exports.",
@@ -107,6 +125,8 @@ const addNativeBinding = (input: NativeBindingInput): void => {
properties: {
resolved_path: resolved,
requested_members: value.members,
namespace_access: value.namespace_access ?? null,
dynamic_member_access: value.dynamic_member_access ?? null,
verified: false,
},
confidence: "medium",
@@ -39,17 +39,32 @@ export const addElectronWindowBoundaries = (
if (javascript === null) continue;
const coverage = javascriptAnalysisCoverage(javascript);
for (const value of javascript.electron.browser_windows) {
const source = electronFindingSourceNode(context, file, value.module_key);
const source = electronFindingSourceNode(
context,
file,
value.module_key,
value.module_runtime,
);
if (source !== undefined)
addBrowserWindow({ context, file, source, value, coverage });
}
for (const value of javascript.electron.context_bridge_apis) {
const source = electronFindingSourceNode(context, file, value.module_key);
const source = electronFindingSourceNode(
context,
file,
value.module_key,
value.module_runtime,
);
if (source !== undefined)
addContextBridge({ context, file, source, value, coverage });
}
for (const value of javascript.electron.utility_processes) {
const source = electronFindingSourceNode(context, file, value.module_key);
const source = electronFindingSourceNode(
context,
file,
value.module_key,
value.module_runtime,
);
if (source !== undefined)
addUtilityProcess({ context, file, source, value, coverage });
}
@@ -31,7 +31,16 @@ const graphIdentities = async (source: string) => {
...(identity.strategy === "artifact-local-key"
? {
namespace: identity.namespace,
key: identity.key.replace(/^[0-9a-f]{64}:/u, ""),
// Module-export identity is a JSON tuple whose first entry
// binds the whole container; only that digest changes here.
key:
identity.namespace === "module-export"
? JSON.stringify(
(JSON.parse(identity.key) as unknown[]).slice(1),
)
: identity.namespace === "source-module"
? identity.key.replace(/^[0-9a-f]{64}:/u, "")
: identity.key,
}
: {}),
}))
@@ -1,6 +1,7 @@
import type { ApplicationNode } from "../../domain/javascript/javascriptApplicationGraphSchemas.js";
import type { JavaScriptBundlerRegistration } from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
import { compositeKey } from "../../domain/unicodeCodePointOrder.js";
import {
addAstContainsEdge,
artifactLocalIdentity,
@@ -80,7 +81,11 @@ const createBundlerChunkNode = (
const chunkKey = `${registration.runtime}:${registration.chunk_keys.join(",")}`;
return context.accumulator.addNode({
kind: "javascript-chunk",
identity: artifactLocalIdentity(file.sha256, "bundler-chunk", chunkKey),
identity: artifactLocalIdentity(
file.sha256,
"bundler-chunk",
compositeKey([registration.runtime, registration.chunk_keys]),
),
observations: [
{
label: chunkKey,
@@ -156,7 +161,7 @@ const addBundlerModuleNodes = (
],
});
context.moduleNodes.set(
moduleLookupKey(file.path, moduleValue.module_key),
moduleLookupKey(file.path, registration.runtime, moduleValue.module_key),
module,
);
addAstContainsEdge(context, {
@@ -190,7 +195,7 @@ const addBundlerRuntimeEdges = (
const { file, registration, chunk, coverage } = record;
for (const moduleKey of registration.entry_module_keys) {
const resolved = context.moduleNodes.get(
moduleLookupKey(file.path, moduleKey),
moduleLookupKey(file.path, registration.runtime, moduleKey),
);
const target =
resolved ?? unresolvedBundlerModuleNode(context, record, moduleKey);
@@ -1,4 +1,5 @@
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
import { compositeKey } from "../../domain/unicodeCodePointOrder.js";
import {
completeApplicationCoverage,
partialApplicationCoverage,
@@ -322,15 +323,18 @@ export const selfReferenceOmissions = (
];
/** Deterministic key for one recovered module inside a bundle asset. */
export const moduleLookupKey = (path: string, moduleKey: string): string =>
`${path}\0${moduleKey}`;
export const moduleLookupKey = (
path: string,
runtime: string,
moduleKey: string,
): string => compositeKey([path, runtime, moduleKey]);
/** Deterministic key for one recovered bundler chunk inside a bundle asset. */
export const chunkLookupKey = (
path: string,
runtime: string,
chunkKey: string,
): string => `${path}\0${runtime}\0${chunkKey}`;
): string => compositeKey([path, runtime, chunkKey]);
/** Resolve a relative static specifier without escaping the artifact root. */
export const resolveArtifactPath = (
@@ -352,7 +356,10 @@ export const sourceNodeFor = (
context: JavaScriptArtifactGraphContext,
path: string,
moduleKey: string | null,
runtime?: string,
): ApplicationNode | undefined =>
moduleKey === null
? context.sourceModuleNodes.get(path)
: context.moduleNodes.get(moduleLookupKey(path, moduleKey));
: runtime === undefined
? undefined
: context.moduleNodes.get(moduleLookupKey(path, runtime, moduleKey));
@@ -78,8 +78,12 @@ const addReference = (
omissions: { selfReferences: number },
): void => {
const source =
sourceNodeFor(context, input.file.path, input.value.module_key) ??
input.asset;
sourceNodeFor(
context,
input.file.path,
input.value.module_key,
input.value.module_runtime,
) ?? input.asset;
const resolved =
input.value.specifier === null
? null
@@ -162,8 +166,12 @@ const addEndpoint = (
input: FindingInput<StaticEndpoint>,
): void => {
const source =
sourceNodeFor(context, input.file.path, input.value.module_key) ??
input.asset;
sourceNodeFor(
context,
input.file.path,
input.value.module_key,
input.value.module_runtime,
) ?? input.asset;
const key = compactStringIdentityKey(input.value.value);
const label = `${input.value.kind} endpoint`;
const node = context.accumulator.addNode({
@@ -204,8 +212,12 @@ const addStorage = (
input: FindingInput<StaticStorage>,
): void => {
const source =
sourceNodeFor(context, input.file.path, input.value.module_key) ??
input.asset;
sourceNodeFor(
context,
input.file.path,
input.value.module_key,
input.value.module_runtime,
) ?? input.asset;
const node = context.accumulator.addNode({
kind: "storage",
identity: artifactLocalIdentity(
@@ -262,8 +274,12 @@ const addDiscoveredRole = (
coverage: input.coverage,
});
const source =
sourceNodeFor(context, input.file.path, input.value.module_key) ??
input.asset;
sourceNodeFor(
context,
input.file.path,
input.value.module_key,
input.value.module_runtime,
) ?? input.asset;
addStaticInferenceEdge(context, {
source,
target: role,
@@ -359,9 +375,12 @@ const resolveReference = (
if (specifier === null) return null;
const chunk = resolveBundlerChunkReference(context, sourcePath, reference);
if (chunk !== null) return chunk;
const module = context.moduleNodes.get(
moduleLookupKey(sourcePath, specifier),
);
const module =
reference.module_runtime === undefined
? undefined
: context.moduleNodes.get(
moduleLookupKey(sourcePath, reference.module_runtime, specifier),
);
if (module !== undefined) {
const file = context.filesByPath.get(sourcePath);
return file === undefined
@@ -407,10 +426,11 @@ const resolveBundlerChunkReference = (
if (file === undefined) return null;
const registrations = context.analysis.files
.find(({ file: analyzedFile }) => analyzedFile.path === sourcePath)
?.javascript?.bundler_registrations.filter(({ modules }) =>
reference.module_key === null
? true
: modules.some(
?.javascript?.bundler_registrations.filter(({ runtime, modules }) =>
reference.module_runtime === undefined
? reference.module_key === null
: runtime === reference.module_runtime &&
modules.some(
({ module_key: moduleKey }) => moduleKey === reference.module_key,
),
);
@@ -1,6 +1,9 @@
import { posix } from "node:path";
import { compareUnicodeCodePoints } from "../../domain/unicodeCodePointOrder.js";
import {
compareUnicodeCodePoints,
compositeKey,
} from "../../domain/unicodeCodePointOrder.js";
import type { ApplicationNode } from "../../domain/javascript/javascriptApplicationGraphSchemas.js";
import {
completeApplicationCoverage,
@@ -177,7 +180,7 @@ const addExportRelationship = (
identity: artifactLocalIdentity(
file.sha256,
"module-export",
`${file.container_sha256}:${file.path}:${link.exportedName}`,
compositeKey([file.container_sha256, file.path, link.exportedName]),
),
observations: [
{
@@ -8,6 +8,7 @@ import type {
/** Typed boundary for upstream source recovery without exposing its protocol. */
export interface JavaScriptRecoveryPort {
close?(): Promise<void>;
recover(
input: JavaScriptRecoveryInput,
options?: ExecutionOptions,
@@ -23,6 +23,10 @@ const OPERATION = "recover_javascript_sources";
export class JavaScriptRecoveryService {
constructor(readonly provider: JavaScriptRecoveryPort) {}
close(): Promise<void> {
return this.provider.close?.() ?? Promise.resolve();
}
/** Recover sources and bind the result to its original immutable input. */
async recover(
input: unknown,
@@ -325,8 +325,7 @@ const resolveUiAction = (
(node) =>
["control", "view", "view_controller", "scene"].includes(node.kind) &&
(node.id === query ||
node.attributes.interface_builder_object_id === query ||
node.id.endsWith(`:object:${query}`)),
node.attributes.interface_builder_object_id === query),
);
const actionIds = new Set(
graph.edges.flatMap((edge) =>
+20 -26
View File
@@ -1,4 +1,3 @@
import { readFile } from "node:fs/promises";
import { z } from "zod";
import { AnalysisError } from "../../domain/analysisErrorBase.js";
@@ -17,9 +16,10 @@ import { compareProcessCaptures } from "../../domain/process/processComparison.j
import { parseProcessCapture } from "../../domain/process/processCaptureParsing.js";
import { createProgressReporter } from "../ProgressReporter.js";
import { captureProcessScenario } from "../../process/capture/ProcessHarness.js";
import { ProcessCaptureResourceScope } from "../../process/capture/ProcessCaptureLifecycle.js";
import { PROCESS_PROVIDER } from "../../domain/process/processEvidenceProvider.js";
import { createProcessCaptureEvidence } from "./ProcessEvidence.js";
import { JSON_BYTE_ORDER_MARK_MESSAGE } from "../Utf8JsonInput.js";
import { readJsonFile } from "../JsonFiles.js";
/** Safe process-command failure returned to the CLI adapter. */
export interface ProcessCliErrorOutput {
@@ -63,6 +63,7 @@ export const captureProcessScenarioFile = async (
},
{ minimumIntervalMs: 100 },
);
const resourceScope = new ProcessCaptureResourceScope();
const captured = await captureProcessScenario(
parsed.data,
signal,
@@ -71,7 +72,14 @@ export const captureProcessScenarioFile = async (
undefined,
undefined,
progress,
resourceScope,
);
try {
await resourceScope.close();
} catch (cause: unknown) {
if (!captured.ok) return cliAnalysisError(captured.error);
throw cause;
}
if (!captured.ok) return cliAnalysisError(captured.error);
return createProcessCaptureEvidence(parsed.data, captured.value);
} catch (cause: unknown) {
@@ -178,30 +186,16 @@ const invalidCaptureEvidence = (cause: unknown): ProcessCliFailure =>
);
const readJson = async (path: string): Promise<unknown> => {
let bytes;
try {
bytes = await readFile(path);
} catch (cause: unknown) {
throw new ProcessCliFailure(
"invalid_input",
`Process input file could not be read: ${path} (${describeValidationFailure(cause)}). Check the reported filesystem constraint and retry.`,
);
}
try {
const text = new TextDecoder("utf-8", {
fatal: true,
ignoreBOM: true,
}).decode(bytes);
if (text.startsWith("\uFEFF"))
throw new Error(JSON_BYTE_ORDER_MARK_MESSAGE);
const parsed: unknown = JSON.parse(text);
return parsed;
} catch (cause: unknown) {
throw new ProcessCliFailure(
"invalid_input",
`Process input file is not valid UTF-8 JSON: ${path} (${describeValidationFailure(cause)}). Repair the file, then try again.`,
);
}
const loaded = await readJsonFile(path);
if (loaded.ok) return loaded.value;
if (loaded.error._tag !== "EvidenceFileError") throw loaded.error;
const detail = describeValidationFailure(loaded.error.cause);
throw new ProcessCliFailure(
"invalid_input",
loaded.error.reason === "invalid-json"
? `Process input file is not valid UTF-8 JSON: ${path} (${detail}). Repair the file, then try again.`
: `Process input file could not be read: ${path} (${detail}). Check the reported filesystem constraint and retry.`,
);
};
class ProcessCliFailure extends Error {
+33
View File
@@ -60,6 +60,30 @@ describe("English Unicode case collisions", () => {
]);
});
it("retains every spelling in a high-cardinality folded sibling group", () => {
const count = 8_192;
const variants = caseVariants("abcdefghijklm", count);
const { limitation, byPath } = annotate(variants);
const ordered = [...variants].sort();
const first = ordered[0];
const second = ordered[1];
const original = variants[0];
if (first === undefined || second === undefined || original === undefined) {
throw new Error("Expected at least two case variants");
}
expect(limitation).toBe(ENGLISH_UNICODE_CASE_INVENTORY_LIMITATION);
expect(byPath.size).toBe(count);
expect(byPath.get(first)).toEqual([
`Logical path ${first} collides under English (en-US) Unicode case folding with ${second} and ${String(count - 2)} other spellings; ${NOTE}`,
]);
expect(byPath.get(original)?.[0]).toContain(
`with ${first} and ${String(count - 2)} other spellings;`,
);
expect(
[...byPath.values()].every((limitations) => limitations.length === 1),
).toBe(true);
});
it("stays linear for large case-variant subtrees", () => {
const count = 5_000;
const paths = Array.from({ length: count }, (_, index) => [
@@ -81,3 +105,12 @@ describe("English Unicode case collisions", () => {
expect(byPath.get(deep)).toEqual([]);
});
});
const caseVariants = (lowercase: string, count: number): string[] =>
Array.from({ length: count }, (_, mask) =>
[...lowercase]
.map((character, index) =>
(mask & (1 << index)) === 0 ? character : character.toUpperCase(),
)
.join(""),
);
+4 -4
View File
@@ -177,12 +177,12 @@ const englishUnicodeCaseCollisions = (
if (path.length === 0 || path === ".") continue;
let node = root;
for (const part of path.split("/")) {
const folded = englishCaseFold(part);
const spellings = node.spellingsByFold.get(folded);
if (spellings === undefined) node.spellingsByFold.set(folded, [part]);
else if (!spellings.includes(part)) spellings.push(part);
let child = node.bySpelling.get(part);
if (child === undefined) {
const folded = englishCaseFold(part);
const spellings = node.spellingsByFold.get(folded);
if (spellings === undefined) node.spellingsByFold.set(folded, [part]);
else spellings.push(part);
child = emptyCaseNode();
node.bySpelling.set(part, child);
}
+12
View File
@@ -27,6 +27,7 @@ import {
type ArtifactReader,
} from "./ArtifactReader.js";
import { closeAsarHandle, readValidatedAsarEntry } from "./AsarEntryStream.js";
import { admitAsarHeader } from "./AsarHeader.js";
/**
* Official Electron ASAR adapter with range-streamed member reads.
@@ -57,6 +58,17 @@ export class AsarArtifactReader implements ArtifactReader {
const snapshotPath = this.#snapshotPath();
this.#resetArchiveState();
this.#entries.clear();
const headerHandle = await open(
snapshotPath,
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
);
try {
const admitted = await admitAsarHeader(headerHandle);
this.#archiveSize = admitted.archiveBytes;
} finally {
await headerHandle.close();
}
abortIfNeeded(signal);
paths = listPackage(snapshotPath, { isPack: false }).sort((left, right) =>
left.localeCompare(right, "en"),
);
+48
View File
@@ -0,0 +1,48 @@
import type { FileHandle } from "node:fs/promises";
import { ArtifactReaderFailure } from "./ArtifactReader.js";
/** Shared resource budget for the eagerly decoded ASAR header, not member bytes. */
export const MAX_ASAR_HEADER_BYTES = 16 * 1024 * 1024;
/** Admit both Chromium pickle extents before an upstream decoder allocates them. */
export const admitAsarHeader = async (
handle: FileHandle,
): Promise<{
readonly archiveBytes: number;
readonly headerBytes: number;
readonly jsonBytes: number;
}> => {
const observed = await handle.stat();
if (!observed.isFile())
throw new ArtifactReaderFailure(
"format",
"ASAR container is not a regular file",
);
const prefix = Buffer.alloc(16);
const read = await handle.read(prefix, 0, prefix.length, 0);
if (read.bytesRead !== prefix.length || prefix.readUInt32LE(0) !== 4)
throw new ArtifactReaderFailure(
"format",
"Truncated or invalid ASAR size pickle",
);
const headerBytes = prefix.readUInt32LE(4);
const payloadBytes = prefix.readUInt32LE(8);
const jsonBytes = prefix.readUInt32LE(12);
if (
headerBytes < 8 ||
headerBytes > observed.size - 8 ||
payloadBytes !== headerBytes - 4 ||
jsonBytes > payloadBytes - 4
)
throw new ArtifactReaderFailure(
"format",
"ASAR header leaves its container or has invalid pickle extents",
);
if (headerBytes > MAX_ASAR_HEADER_BYTES)
throw new ArtifactReaderFailure(
"limit",
`ASAR header exceeds the ${String(MAX_ASAR_HEADER_BYTES)} byte decode budget`,
);
return { archiveBytes: observed.size, headerBytes, jsonBytes };
};
+42 -44
View File
@@ -3,15 +3,13 @@ import { lstat, open, opendir, realpath } from "node:fs/promises";
import { isAbsolute, join, relative, sep } from "node:path";
import type { Readable } from "node:stream";
import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js";
import {
ArtifactReaderFailure,
type ArtifactEntry,
type ArtifactReader,
} from "./ArtifactReader.js";
/** Deterministic, non-symlink-following directory reader. */
/** Non-symlink-following directory reader. */
export class DirectoryArtifactReader implements ArtifactReader {
readonly format = "directory" as const;
readonly #rootPromise: Promise<string>;
@@ -38,13 +36,52 @@ export class DirectoryArtifactReader implements ArtifactReader {
await withDirectoryIoContext("inspect directory", directory, () =>
assertContainedDirectory(root, directory),
);
const children: string[] = [];
const handle = await opendir(directory).catch((cause: unknown) => {
throw directoryIoFailure("open directory", directory, cause);
});
const directories: string[] = [];
try {
for await (const child of handle) children.push(child.name);
for await (const child of handle) {
abortIfNeeded(signal);
const absolute = join(directory, child.name);
const metadata = await lstat(absolute).catch((cause: unknown) => {
throw directoryIoFailure("inspect entry", absolute, cause);
});
const path = relative(root, absolute).split(sep).join("/");
const kind = metadata.isSymbolicLink()
? "symlink"
: metadata.isDirectory()
? "directory"
: metadata.isFile()
? "file"
: undefined;
if (kind === undefined) continue;
yield {
path,
kind,
declaredSize: kind === "file" ? metadata.size : null,
compressedSize: null,
executable: (metadata.mode & 0o111) !== 0,
encrypted: false,
byteOffset: null,
declaredSha256: null,
unpacked: false,
limitations:
kind === "symlink"
? ["Symlink target was not followed or disclosed."]
: [],
adapterKey: absolute,
...(kind === "file"
? {
sourceIdentity: { device: metadata.dev, inode: metadata.ino },
}
: {}),
};
if (kind === "directory" && (descendInto?.(path) ?? true))
directories.push(absolute);
}
} catch (cause: unknown) {
if (cause instanceof ArtifactReaderFailure) throw cause;
throw directoryIoFailure("read directory", directory, cause);
} finally {
// best-effort cleanup: directory-handle close must not mask traversal.
@@ -53,45 +90,6 @@ export class DirectoryArtifactReader implements ArtifactReader {
await withDirectoryIoContext("inspect directory", directory, () =>
assertContainedDirectory(root, directory),
);
children.sort(compareUnicodeCodePoints);
const directories: string[] = [];
for (const name of children) {
abortIfNeeded(signal);
const absolute = join(directory, name);
const metadata = await lstat(absolute).catch((cause: unknown) => {
throw directoryIoFailure("inspect entry", absolute, cause);
});
const path = relative(root, absolute).split(sep).join("/");
const kind = metadata.isSymbolicLink()
? "symlink"
: metadata.isDirectory()
? "directory"
: metadata.isFile()
? "file"
: undefined;
if (kind === undefined) continue;
if (kind === "directory" && (descendInto?.(path) ?? true))
directories.push(absolute);
yield {
path,
kind,
declaredSize: kind === "file" ? metadata.size : null,
compressedSize: null,
executable: (metadata.mode & 0o111) !== 0,
encrypted: false,
byteOffset: null,
declaredSha256: null,
unpacked: false,
limitations:
kind === "symlink"
? ["Symlink target was not followed or disclosed."]
: [],
adapterKey: absolute,
...(kind === "file"
? { sourceIdentity: { device: metadata.dev, inode: metadata.ino } }
: {}),
};
}
directories.reverse();
pending.push(...directories);
}
+101 -24
View File
@@ -1,5 +1,5 @@
import { createReadStream } from "node:fs";
import { open as openFile, stat } from "node:fs/promises";
import { lstat } from "node:fs/promises";
import type { FileHandle } from "node:fs/promises";
import { Readable } from "node:stream";
import type { ArtifactCommand } from "../domain/artifactGraph.js";
@@ -19,11 +19,21 @@ import {
type ArtifactReader,
} from "./ArtifactReader.js";
import { streamChunkToBuffer } from "./StreamBytes.js";
import {
openRegularFile,
sameRegularFileState,
type StableRegularFileDescriptor,
} from "../filesystem/RegularFile.js";
/** Read-only universal Mach-O slice reader backed by native lipo metadata. */
export class MachOSliceArtifactReader implements ArtifactReader {
readonly format = "file" as const;
#command: ArtifactCommand | undefined;
#source: StableRegularFileDescriptor | undefined;
#pendingSource: Promise<StableRegularFileDescriptor> | undefined;
readonly #ownsSource: boolean;
#closed = false;
#closePromise: Promise<void> | undefined;
constructor(
private readonly path: string,
@@ -31,9 +41,15 @@ export class MachOSliceArtifactReader implements ArtifactReader {
private readonly runner: NativeCommandRunner = new XcrunCommandRunner(
environment,
),
) {}
source?: StableRegularFileDescriptor,
) {
this.#source = source;
this.#ownsSource = source === undefined;
}
async *entries(signal?: AbortSignal): AsyncIterable<ArtifactEntry> {
const source = await this.#ensureSource();
await this.#verifySource(source);
const captured = await this.runner.run(
"lipo",
["-detailed_info", this.path],
@@ -53,9 +69,11 @@ export class MachOSliceArtifactReader implements ArtifactReader {
exit_code: captured.value.exitCode,
effects: ["read"],
};
const fileSize = (await stat(this.path)).size;
await this.#verifySource(source);
const fileSize = source.initial.size;
const architectures = parseLipoArchitectures(captured.value.stdout);
const structural = await readStructuralSlices(this.path, fileSize);
const structural = await readStructuralSlices(source.handle, fileSize);
await this.#verifySource(source);
if (structural.status === "malformed")
throw new ArtifactReaderFailure(
"integrity",
@@ -168,22 +186,28 @@ export class MachOSliceArtifactReader implements ArtifactReader {
"integrity",
"Invalid Mach-O slice byte range",
);
const fileSize = (await stat(this.path)).size;
if (!Number.isSafeInteger(offset + size) || offset + size > fileSize)
const source = await this.#ensureSource();
await this.#verifySource(source);
if (
!Number.isSafeInteger(offset + size) ||
offset + size > source.initial.size
)
throw new ArtifactReaderFailure(
"integrity",
`Mach-O slice range is outside the artifact: ${entry.path}`,
);
const source = createReadStream(this.path, {
const sourceStream = source.handle.createReadStream({
start: offset,
end: offset + size - 1,
autoClose: false,
...(signal === undefined ? {} : { signal }),
});
const verifySource = () => this.#verifySource(source);
return Readable.from(
(async function* () {
let observedBytes = 0;
try {
for await (const raw of source) {
for await (const raw of sourceStream) {
const chunk = streamChunkToBuffer(raw);
observedBytes += chunk.byteLength;
if (observedBytes > size)
@@ -212,6 +236,7 @@ export class MachOSliceArtifactReader implements ArtifactReader {
"integrity",
`Mach-O slice size disagrees with lipo metadata: ${entry.path}`,
);
await verifySource();
})(),
);
}
@@ -220,28 +245,80 @@ export class MachOSliceArtifactReader implements ArtifactReader {
return this.#command === undefined ? [] : [structuredClone(this.#command)];
}
close(): Promise<void> {
return Promise.resolve();
async close(): Promise<void> {
this.#closed = true;
this.#closePromise ??= this.#closeSource().catch((cause: unknown) => {
this.#closePromise = undefined;
throw cause;
});
return this.#closePromise;
}
async #ensureSource(): Promise<StableRegularFileDescriptor> {
if (this.#closed)
throw new ArtifactReaderFailure(
"unavailable",
"Universal Mach-O reader is closed",
);
if (this.#source !== undefined) return this.#source;
this.#pendingSource ??= this.#acquireSource();
const source = await this.#pendingSource;
if (this.#closed)
throw new ArtifactReaderFailure(
"unavailable",
"Universal Mach-O reader closed during source acquisition",
);
return source;
}
async #acquireSource(): Promise<StableRegularFileDescriptor> {
const handle = await openRegularFile(this.path, { symlinks: "reject" });
try {
const initial = await handle.stat();
const source = { handle, initial };
this.#source = source;
return source;
} catch (cause: unknown) {
await handle.close().catch(() => undefined);
throw cause;
}
}
async #closeSource(): Promise<void> {
await this.#pendingSource?.catch(() => undefined);
if (!this.#ownsSource || this.#source === undefined) return;
const source = this.#source;
await source.handle.close();
this.#source = undefined;
}
async #verifySource(source: StableRegularFileDescriptor): Promise<void> {
const [opened, currentPath] = await Promise.all([
source.handle.stat(),
lstat(this.path),
]);
if (
!sameRegularFileState(source.initial, opened) ||
!sameRegularFileState(source.initial, currentPath)
)
throw new ArtifactReaderFailure(
"integrity",
`Universal Mach-O source changed during inventory: ${this.path}`,
);
}
}
/** Compare lipo's independent ranges with structurally parsed slice identity. */
const readStructuralSlices = async (
path: string,
file: FileHandle,
size: number,
): Promise<Awaited<ReturnType<typeof readMachoImage>>> => {
const file = await openFile(path, "r");
try {
const readAt: ReadAt = async (offset, length) => {
const bytes = Buffer.alloc(length);
const { bytesRead } = await file.read(bytes, 0, length, offset);
return bytes.subarray(0, bytesRead);
};
const facts = await readMachoImage(readAt, size);
return facts;
} finally {
await file.close();
}
const readAt: ReadAt = async (offset, length) => {
const bytes = Buffer.alloc(length);
const { bytesRead } = await file.read(bytes, 0, length, offset);
return bytes.subarray(0, bytesRead);
};
return readMachoImage(readAt, size);
};
const lipoMatchesSlice = (
+93 -46
View File
@@ -105,7 +105,26 @@ export class SafeOutputTree {
}
return new SafeOutputTree(canonicalOutput, platform);
} catch (cause: unknown) {
await rm(canonicalOutput, { recursive: true, force: true });
let removalFailure: unknown;
try {
await rm(canonicalOutput, { recursive: true, force: true });
} catch (cleanupCause: unknown) {
removalFailure = cleanupCause;
}
let absent = false;
try {
absent = await isAbsent(canonicalOutput);
} catch (cleanupCause: unknown) {
removalFailure ??= cleanupCause;
}
if (!absent)
throw ArtifactReaderFailure.withCleanup(cause, {
reason:
removalFailure instanceof Error
? removalFailure.message
: "Extraction output root remains after setup failure",
resources: [canonicalOutput],
});
throw cause;
}
}
@@ -122,56 +141,84 @@ export class SafeOutputTree {
async write(
relativePath: string,
source: Readable,
expectedSha256: string,
expected: { readonly sha256: string; readonly bytes: number },
signal?: AbortSignal,
): Promise<SafeOutputFile> {
this.#assertWritable();
const path = normalizeArtifactPath(relativePath);
this.#registry.add(path, "file");
const destination = await this.#prepareParent(path);
const handle = await open(
destination,
constants.O_CREAT |
constants.O_EXCL |
constants.O_WRONLY |
constants.O_NOFOLLOW,
0o600,
).catch(async (cause: unknown) => {
await throwIfDestinationCaseCollision(dirname(destination), path, cause);
throw new ArtifactReaderFailure(
"path",
`Could not exclusively create extraction path: ${path}`,
{ cause },
);
});
const hash = createHash("sha256");
let bytes = 0;
try {
for await (const raw of source) {
abortIfNeeded(signal);
const chunk = streamChunkToBuffer(raw);
bytes += chunk.length;
hash.update(chunk);
await writeAll(handle, chunk);
this.#assertWritable();
if (!Number.isSafeInteger(expected.bytes) || expected.bytes < 0)
throw new ArtifactReaderFailure(
"format",
`Invalid expected extraction size for ${relativePath}`,
);
const path = normalizeArtifactPath(relativePath);
this.#registry.add(path, "file");
const destination = await this.#prepareParent(path);
const handle = await open(
destination,
constants.O_CREAT |
constants.O_EXCL |
constants.O_WRONLY |
constants.O_NOFOLLOW,
0o600,
).catch(async (cause: unknown) => {
await throwIfDestinationCaseCollision(
dirname(destination),
path,
cause,
);
throw new ArtifactReaderFailure(
"path",
`Could not exclusively create extraction path: ${path}`,
{ cause },
);
});
const hash = createHash("sha256");
let bytes = 0;
try {
for await (const raw of source) {
abortIfNeeded(signal);
const chunk = streamChunkToBuffer(raw);
if (chunk.length > expected.bytes - bytes)
throw new ArtifactReaderFailure(
"integrity",
`Extracted content exceeds the inventoried size: ${path}`,
);
bytes += chunk.length;
hash.update(chunk);
await writeAll(handle, chunk);
}
if (bytes !== expected.bytes)
throw new ArtifactReaderFailure(
"integrity",
`Extracted content size disagrees with inventory: ${path}`,
);
const sha256 = hash.digest("hex");
if (sha256 !== expected.sha256)
throw new ArtifactReaderFailure(
"integrity",
`Extracted content disagrees with inventory: ${path}`,
);
await handle.sync();
await handle.close();
const readback = await hashFile(destination, bytes, signal);
if (readback.sha256 !== sha256 || readback.bytes !== bytes)
throw new ArtifactReaderFailure(
"integrity",
`Durable readback verification failed: ${path}`,
);
return { relativePath: path, sha256, bytesWritten: bytes };
} catch (cause: unknown) {
// best-effort cleanup: file-handle close must not mask the write failure.
await handle.close().catch(() => undefined);
throw cause;
}
const sha256 = hash.digest("hex");
if (sha256 !== expectedSha256)
throw new ArtifactReaderFailure(
"integrity",
`Extracted content disagrees with inventory: ${path}`,
);
await handle.sync();
await handle.close();
const readback = await hashFile(destination, bytes, signal);
if (readback.sha256 !== sha256 || readback.bytes !== bytes)
throw new ArtifactReaderFailure(
"integrity",
`Durable readback verification failed: ${path}`,
);
return { relativePath: path, sha256, bytesWritten: bytes };
} catch (cause: unknown) {
// best-effort cleanup: file-handle close must not mask the write failure.
await handle.close().catch(() => undefined);
try {
source.destroy();
} catch {
// Preserve the write refusal or failure as the caller-visible error.
}
throw cause;
}
}
+47 -6
View File
@@ -1,4 +1,4 @@
import { open, type FileHandle } from "node:fs/promises";
import type { FileHandle } from "node:fs/promises";
import { PassThrough, type Readable } from "node:stream";
import { once } from "node:events";
@@ -10,20 +10,57 @@ import {
type ArtifactReader,
} from "./ArtifactReader.js";
import type { ZipPackageFormat } from "../domain/zipPackageFormat.js";
import {
NonRegularFileReadError,
openRegularFile,
sameRegularFileState,
type StableRegularFileDescriptor,
} from "../filesystem/RegularFile.js";
class NodeFileReader extends Reader<string> {
#handle: FileHandle | undefined;
#initPromise: Promise<void> | undefined;
readonly #ownsHandle: boolean;
constructor(
private readonly path: string,
private readonly maximumReadBytes?: number,
private readonly admitted?: StableRegularFileDescriptor,
) {
super(path);
this.#ownsHandle = admitted === undefined;
}
override async init(): Promise<void> {
this.#handle = await open(this.path, "r");
this.size = (await this.#handle.stat()).size;
override init(): Promise<void> {
this.#initPromise ??= this.#initialize();
return this.#initPromise;
}
async #initialize(): Promise<void> {
try {
this.#handle =
this.admitted?.handle ??
(await openRegularFile(this.path, { symlinks: "reject" }));
} catch (cause: unknown) {
if (cause instanceof NonRegularFileReadError)
throw new ArtifactReaderFailure(
"format",
`ZIP source is not a regular file: ${this.path}`,
{ cause },
);
throw cause;
}
const metadata = await this.#handle.stat();
if (
!metadata.isFile() ||
(this.admitted !== undefined &&
!sameRegularFileState(this.admitted.initial, metadata))
)
throw new ArtifactReaderFailure(
"integrity",
`ZIP source identity changed before reading: ${this.path}`,
);
this.size = metadata.size;
}
override async readUint8Array(
@@ -47,7 +84,10 @@ class NodeFileReader extends Reader<string> {
}
async closeHandle(): Promise<void> {
await this.#handle?.close();
if (!this.#ownsHandle) return;
await this.#initPromise?.catch(() => undefined);
if (this.#handle === undefined) return;
await this.#handle.close();
this.#handle = undefined;
}
}
@@ -64,9 +104,10 @@ export class ZipArtifactReader implements ArtifactReader {
path: string,
format: ZipPackageFormat,
maximumMetadataReadBytes?: number,
admitted?: StableRegularFileDescriptor,
) {
this.format = format;
this.#source = new NodeFileReader(path, maximumMetadataReadBytes);
this.#source = new NodeFileReader(path, maximumMetadataReadBytes, admitted);
this.#reader = new ZipReader(this.#source, {
checkSignature: true,
checkOverlappingEntry: true,
+65 -10
View File
@@ -25,6 +25,7 @@ import {
type NibHierarchyNode,
} from "./NibViewHierarchy.js";
import { jsonValueSchema, type JsonValue } from "../../domain/jsonValue.js";
import { compareUnicodeCodePoints } from "../../domain/unicodeCodePointOrder.js";
import { decodeXmlPlistText } from "../../domain/propertyListXmlText.js";
@@ -49,7 +50,6 @@ export const analyzeInterfaceBuilderBundle = async (input: {
const prototypeKeyOmissions = new Map<string, number>();
let omitted = 0;
let attempted = 0;
let candidates = 0;
let aggregateInputBytes = 0;
const decodeBudget = new InterfaceBuilderDecodeBudget(
MAX_AGGREGATE_DECODE_BYTES,
@@ -60,15 +60,16 @@ export const analyzeInterfaceBuilderBundle = async (input: {
else unreportedOmittedArchives += 1;
};
try {
for await (const entry of reader.entries(input.signal)) {
if (entry.kind !== "file" || !isInterfaceBuilderArchive(entry.path))
continue;
candidates += 1;
if (candidates > limits.max_documents) {
omitted += 1;
recordOmittedArchive(entry.path, "max_documents");
continue;
}
const selection = await selectInterfaceBuilderCandidates(
reader,
input.signal,
limits.max_documents,
);
omitted = selection.omitted;
for (const entry of selection.omittedSamples)
recordOmittedArchive(entry.path, "max_documents");
unreportedOmittedArchives += selection.unreportedOmitted;
for (const entry of selection.selected) {
try {
const remainingInputBytes =
MAX_AGGREGATE_INPUT_BYTES - aggregateInputBytes;
@@ -167,6 +168,60 @@ export const analyzeInterfaceBuilderBundle = async (input: {
});
};
/** Select lexically earliest document paths independent of host traversal order. */
const selectInterfaceBuilderCandidates = async (
reader: DirectoryArtifactReader,
signal: AbortSignal | undefined,
maximumDocuments: number,
): Promise<{
readonly selected: readonly ArtifactEntry[];
readonly omittedSamples: readonly ArtifactEntry[];
readonly omitted: number;
readonly unreportedOmitted: number;
}> => {
const capacity = maximumDocuments + MAX_REPORTED_OMITTED_ARCHIVES;
const candidates: ArtifactEntry[] = [];
let total = 0;
for await (const entry of reader.entries(signal)) {
if (entry.kind !== "file" || !isInterfaceBuilderArchive(entry.path))
continue;
total += 1;
retainCandidate(candidates, entry, capacity);
}
const omittedSamples = candidates.slice(maximumDocuments, capacity);
return {
selected: candidates.slice(0, maximumDocuments),
omittedSamples,
omitted: Math.max(0, total - maximumDocuments),
unreportedOmitted: Math.max(
0,
total - maximumDocuments - omittedSamples.length,
),
};
};
/** Keep the earliest bounded path set; directory iteration order is host-defined. */
const retainCandidate = (
candidates: ArtifactEntry[],
entry: ArtifactEntry,
maximum: number,
): void => {
let low = 0;
let high = candidates.length;
while (low < high) {
const middle = Math.floor((low + high) / 2);
const existing = candidates[middle];
if (
existing !== undefined &&
compareUnicodeCodePoints(existing.path, entry.path) < 0
)
low = middle + 1;
else high = middle;
}
candidates.splice(low, 0, entry);
if (candidates.length > maximum) candidates.pop();
};
const finalizeHierarchyCoverage = (
result: ReturnType<typeof buildInterfaceBuilderAnalysis>,
counts: {
+12 -13
View File
@@ -147,29 +147,28 @@ export const inspectBundleKeyedArchive = async (input: {
try {
// Match inventory spelling while retaining the raw path for I/O and evidence.
const wanted = selected.path.normalize("NFC");
const matches: ArtifactEntry[] = [];
for await (const entry of reader.entries(input.signal, (directory) =>
let entry: ArtifactEntry | undefined;
for await (const candidate of reader.entries(input.signal, (directory) =>
wanted.startsWith(`${directory.normalize("NFC")}/`),
)) {
if (entry.path.normalize("NFC") !== wanted) continue;
if (entry.kind !== "file")
if (candidate.path.normalize("NFC") !== wanted) continue;
if (candidate.kind !== "file")
throw archivePathError(
"invalid_value",
`Archive path selects a ${entry.kind}, not a regular file: ${selected.path}`,
`Archive path selects a ${candidate.kind}, not a regular file: ${selected.path}`,
);
if ((entry.declaredSize ?? 0) > MAX_BYTES)
if ((candidate.declaredSize ?? 0) > MAX_BYTES)
throw new ArtifactReaderFailure(
"limit",
"Keyed archive exceeds 64 MiB",
);
matches.push(entry);
if (entry !== undefined)
throw archivePathError(
"invalid_value",
`Archive path matches multiple Unicode-equivalent bundle entries; selection is ambiguous: ${selected.path}`,
);
entry = candidate;
}
if (matches.length > 1)
throw archivePathError(
"invalid_value",
`Archive path matches ${String(matches.length)} Unicode-equivalent bundle entries; selection is ambiguous: ${selected.path}`,
);
const entry = matches[0];
if (entry === undefined)
throw archivePathError(
"invalid_value",
@@ -215,7 +215,10 @@ const materializeSelection = async ({
const written = await output.write(
path,
stream,
selectedItem.node.sha256,
{
sha256: selectedItem.node.sha256,
bytes: selectedItem.node.size,
},
signal,
);
extracted.push({
@@ -295,23 +295,53 @@ export const createArtifactEdges = (
return edges;
};
export const nearestParent = (
path: string,
occurrences: ReadonlyMap<string, MutableOccurrence>,
interface ArtifactParentIndex {
readonly children: Map<string, ArtifactParentIndex>;
occurrence?: MutableOccurrence;
}
/** Resolve strict observed ancestors without rebuilding every path prefix. */
export const resolveOccurrenceParents = (
occurrences: readonly MutableOccurrence[],
byPath: ReadonlyMap<string, MutableOccurrence>,
expandedContainerIds: ReadonlySet<string>,
): MutableOccurrence | undefined => {
const parts = path.split("/");
while (parts.length > 1) {
parts.pop();
const candidate = occurrences.get(parts.join("/"));
): void => {
const root: ArtifactParentIndex = { children: new Map() };
for (const occurrence of byPath.values()) {
if (
candidate !== undefined &&
(candidate.entry_kind === "directory" ||
expandedContainerIds.has(candidate.occurrence_id))
occurrence.entry_kind !== "directory" &&
!expandedContainerIds.has(occurrence.occurrence_id)
)
return candidate;
continue;
let branch = root;
for (const segment of occurrence.logical_path.split("/")) {
let child = branch.children.get(segment);
if (child === undefined) {
child = { children: new Map() };
branch.children.set(segment, child);
}
branch = child;
}
branch.occurrence = occurrence;
}
for (const occurrence of occurrences) {
const path = occurrence.logical_path;
let branch = root;
let parent: MutableOccurrence | undefined;
// Walk parent segments only. Stop scanning the path once no observed
// directory or expanded container can be an ancestor.
for (let start = 0; ;) {
const slash = path.indexOf("/", start);
if (slash < 0) break;
const segment = path.slice(start, slash);
const child = branch.children.get(segment);
if (child === undefined) break;
branch = child;
parent = branch.occurrence ?? parent;
start = slash + 1;
}
occurrence.parent_occurrence_id = parent?.occurrence_id ?? null;
}
return undefined;
};
/**
+49 -7
View File
@@ -5,6 +5,7 @@ import {
} from "../../filesystem/RegularFile.js";
import type { FileHandle } from "node:fs/promises";
import type { Stats } from "node:fs";
import type { StableRegularFileDescriptor } from "../../filesystem/RegularFile.js";
import { classifyArtifactContent } from "./ArtifactGraphConstruction.js";
import { ARTIFACT_CLASSIFICATION_PREFIX_BYTES } from "../ArtifactHash.js";
@@ -17,18 +18,46 @@ import { ArtifactReaderFailure } from "../ArtifactReader.js";
import type { HashResult } from "../ArtifactHash.js";
import { hashStableRootArtifactHandle } from "./hashStableRootArtifact.js";
interface RootClassification {
readonly format: ArtifactOccurrence["artifact_format"];
readonly digest: HashResult | null;
}
type RootInventoryClassification = RootClassification & {
readonly rootSource: StableRegularFileDescriptor | undefined;
};
/** Classify and hash one file root through the same stable open descriptor. */
export const classifyAndHashRoot = async (
path: string,
directory: boolean,
expectedMetadata: Stats,
signal?: AbortSignal,
): Promise<{
readonly format: ArtifactOccurrence["artifact_format"];
readonly digest: HashResult | null;
}> => {
if (directory) return { format: "directory", digest: null };
): Promise<RootClassification> => {
const result = await classifyAndHashRootForInventory(
path,
directory,
expectedMetadata,
signal,
);
try {
return { format: result.format, digest: result.digest };
} finally {
await result.rootSource?.handle.close();
}
};
/** Classify a root and retain its admitted ZIP descriptor for child inventory. */
export const classifyAndHashRootForInventory = async (
path: string,
directory: boolean,
expectedMetadata: Stats,
signal?: AbortSignal,
): Promise<RootInventoryClassification> => {
if (directory)
return { format: "directory", digest: null, rootSource: undefined };
const handle = await openRootFile(path, signal);
let transferHandle = false;
try {
const initial = await handle.stat();
if (!sameRegularFileState(expectedMetadata, initial))
@@ -43,12 +72,25 @@ export const classifyAndHashRoot = async (
initial,
signal,
);
return { format, digest };
const retainSource = isZipFormat(format) || format === "mach-o-universal";
if (retainSource) transferHandle = true;
return {
format,
digest,
rootSource: retainSource ? { handle, initial } : undefined,
};
} finally {
await handle.close();
if (!transferHandle) await handle.close();
}
};
const isZipFormat = (format: ArtifactOccurrence["artifact_format"]): boolean =>
format === "zip" ||
format === "ipa" ||
format === "apk" ||
format === "msix" ||
format === "appx";
const openRootFile = async (
path: string,
signal?: AbortSignal,
@@ -1,4 +1,5 @@
import { lstat } from "node:fs/promises";
import type { Stats } from "node:fs";
import type { FileHandle } from "node:fs/promises";
import { ArtifactReaderFailure } from "../ArtifactReader.js";
@@ -45,11 +46,13 @@ export const hashStableRootArtifact = async (
export const hashStableRootArtifactHandle = async (
path: string,
handle: FileHandle,
initial: Awaited<ReturnType<FileHandle["stat"]>>,
initial: Stats,
signal?: AbortSignal,
): Promise<HashResult> => {
const digest = await hashReadable(
handle.createReadStream({ autoClose: false }),
// Read at most the admitted extent plus one byte to detect growth without
// following an actively appended file indefinitely.
handle.createReadStream({ autoClose: false, start: 0, end: initial.size }),
signal,
);
abortIfNeeded(signal);
+4 -2
View File
@@ -6,12 +6,14 @@ import { NativeDmgArtifactReader } from "../NativeDmgArtifactReader.js";
import { ZipArtifactReader } from "../ZipArtifactReader.js";
import type { ArtifactOccurrence } from "../../domain/artifactGraph.js";
import { ZIP_NON_ENTRY_TAIL_LIMITATION } from "../../domain/zipPackageFormat.js";
import type { StableRegularFileDescriptor } from "../../filesystem/RegularFile.js";
export const createReader = async (
path: string,
format: ArtifactOccurrence["artifact_format"],
environment: Readonly<NodeJS.ProcessEnv>,
signal?: AbortSignal,
rootSource?: StableRegularFileDescriptor,
): Promise<ArtifactReader | undefined> => {
switch (format) {
case "directory":
@@ -21,12 +23,12 @@ export const createReader = async (
case "apk":
case "msix":
case "appx":
return new ZipArtifactReader(path, format);
return new ZipArtifactReader(path, format, undefined, rootSource);
case "asar":
return new AsarArtifactReader(path);
case "mach-o-universal":
return process.platform === "darwin"
? new MachOSliceArtifactReader(path, environment)
? new MachOSliceArtifactReader(path, environment, undefined, rootSource)
: undefined;
case "dmg":
if (process.platform !== "darwin") return undefined;
+58 -12
View File
@@ -5,6 +5,7 @@ import { AsarArtifactReader } from "../AsarArtifactReader.js";
import { lstat } from "node:fs/promises";
import type { Stats } from "node:fs";
import type { StableRegularFileDescriptor } from "../../filesystem/RegularFile.js";
import {
ArtifactReaderFailure,
@@ -31,9 +32,12 @@ import {
artifactGraphDigest,
artifactManifestId,
} from "../../domain/artifactIdentity.js";
import { classifyAndHashRoot } from "./classify.js";
import { classifyAndHashRootForInventory } from "./classify.js";
import { type HashResult } from "../ArtifactHash.js";
import { hashStableRootArtifact } from "./hashStableRootArtifact.js";
import {
hashStableRootArtifact,
hashStableRootArtifactHandle,
} from "./hashStableRootArtifact.js";
import { createReader, inventoryLimitations } from "./reader.js";
import {
scanReader,
@@ -51,18 +55,29 @@ export const scanCanonicalArtifactInventory = async (
): Promise<ArtifactInventorySnapshot> => {
const integrity = options.integrity ?? STRICT_INTEGRITY_POLICY;
const metadata = await lstat(path);
const { format: rootFormat, digest: rootDigest } = await classifyAndHashRoot(
const {
format: rootFormat,
digest: rootDigest,
rootSource,
} = await classifyAndHashRootForInventory(
path,
metadata.isDirectory(),
metadata,
options.signal,
);
const reader = await readerFactory(
path,
rootFormat,
options.environment ?? {},
options.signal,
);
let reader: ArtifactReader | undefined;
let readerCreationFailure: { readonly cause: unknown } | undefined;
try {
reader = await readerFactory(
path,
rootFormat,
options.environment ?? {},
options.signal,
rootSource,
);
} catch (cause: unknown) {
readerCreationFailure = { cause };
}
const ownedReaders: ArtifactReader[] = reader === undefined ? [] : [reader];
let outcome:
| {
@@ -71,6 +86,7 @@ export const scanCanonicalArtifactInventory = async (
}
| { readonly kind: "failed"; readonly cause: unknown };
try {
if (readerCreationFailure !== undefined) throw readerCreationFailure.cause;
if (reader instanceof AsarArtifactReader)
await reader.prepareContainer(rootDigest?.sha256, options.signal);
const {
@@ -86,6 +102,7 @@ export const scanCanonicalArtifactInventory = async (
metadata,
rootFormat,
rootDigest,
rootSource,
reader,
signal: options.signal,
nodes,
@@ -116,6 +133,24 @@ export const scanCanonicalArtifactInventory = async (
);
}
}
if (rootSource !== undefined) {
try {
await rootSource.handle.close();
} catch (cleanupCause: unknown) {
const cleanup = ArtifactReaderFailure.cleanupObservation(
cleanupCause,
`root artifact descriptor for ${path}`,
);
cleanupFailure = ArtifactReaderFailure.withCleanup(
cleanupFailure ??
(outcome.kind === "failed" ? outcome.cause : cleanupCause),
cleanup,
outcome.kind === "completed"
? { kind: "artifact-inventory", inventory: outcome.snapshot }
: undefined,
);
}
}
if (cleanupFailure !== undefined) throw cleanupFailure;
if (outcome.kind === "failed") throw outcome.cause;
return outcome.snapshot;
@@ -126,6 +161,7 @@ interface SnapshotBuildInput {
readonly metadata: Stats;
readonly rootFormat: ArtifactOccurrence["artifact_format"];
readonly rootDigest: HashResult | null;
readonly rootSource: StableRegularFileDescriptor | undefined;
readonly reader: ArtifactReader | undefined;
readonly signal: AbortSignal | undefined;
readonly nodes: Map<string, ArtifactNode>;
@@ -137,7 +173,8 @@ interface SnapshotBuildInput {
const buildInventorySnapshot = async (
input: SnapshotBuildInput,
): Promise<ArtifactInventorySnapshot> => {
const { path, metadata, rootFormat, rootDigest, reader, signal } = input;
const { path, metadata, rootFormat, rootDigest, rootSource, reader, signal } =
input;
materializeDirectoryNodes(input.occurrences, input.nodes);
const rootNode = createRootNode({
digest: rootDigest,
@@ -181,7 +218,7 @@ const buildInventorySnapshot = async (
ordinal,
}));
await verifyRootDigest(path, rootDigest, signal);
await verifyRootDigest(path, rootDigest, rootSource, signal);
const graphSha256 = artifactGraphDigest({
nodes: orderedNodes,
@@ -256,10 +293,19 @@ const buildIntegrityContradictions = (
const verifyRootDigest = async (
path: string,
rootDigest: HashResult | null,
rootSource: StableRegularFileDescriptor | undefined,
signal: AbortSignal | undefined,
): Promise<void> => {
if (rootDigest === null) return;
const verified = await hashStableRootArtifact(path, signal);
const verified =
rootSource === undefined
? await hashStableRootArtifact(path, signal)
: await hashStableRootArtifactHandle(
path,
rootSource.handle,
rootSource.initial,
signal,
);
if (
verified.sha256 !== rootDigest.sha256 ||
verified.bytes !== rootDigest.bytes
+6 -8
View File
@@ -18,7 +18,7 @@ import {
createArtifactNode,
createOccurrence,
noteUnreadContainerSuffix,
nearestParent,
resolveOccurrenceParents,
type MutableOccurrence,
} from "./ArtifactGraphConstruction.js";
import {
@@ -91,13 +91,11 @@ export const scanReader = async (
await visitArtifactEntries(context, reader, "");
// Archive directories may appear after their children. Resolve containment
// against the complete index before directory identities are materialized.
for (const occurrence of occurrences)
occurrence.parent_occurrence_id =
nearestParent(
occurrence.logical_path,
context.occurrenceByPath,
context.expandedContainerIds,
)?.occurrence_id ?? null;
resolveOccurrenceParents(
occurrences,
context.occurrenceByPath,
context.expandedContainerIds,
);
return {
nodes,
occurrences,
+4 -1
View File
@@ -38,5 +38,8 @@ export interface BrowserScenarioSessionPort {
/** Acquire one scenario session whose lifecycle the caller owns. */
export type BrowserScenarioSessionOpener = (
scenario: BrowserScenario,
options: { readonly signal?: AbortSignal },
options: {
readonly signal?: AbortSignal;
readonly retainCleanup?: (close: () => Promise<unknown>) => void;
},
) => Promise<BrowserScenarioSessionPort>;
+10 -2
View File
@@ -33,6 +33,7 @@ export class CdpConnection {
>();
#nextId = 1;
#closed = false;
#closePromise: Promise<void> | undefined;
#protocolFailed = false;
#transportFailure: "disconnected" | "payload_limit" = "disconnected";
@@ -151,8 +152,15 @@ export class CdpConnection {
}
/** Close only REA's socket; never close the browser or selected page. */
async close(): Promise<void> {
if (this.#closed) return;
close(): Promise<void> {
this.#closePromise ??= this.#closeTransport().catch((cause: unknown) => {
this.#closePromise = undefined;
throw cause;
});
return this.#closePromise;
}
async #closeTransport(): Promise<void> {
this.#closed = true;
this.#failPending("disconnected");
if (this.socket.readyState === WebSocket.CLOSED) return;
+34 -46
View File
@@ -50,54 +50,42 @@ export const closeCdpTargetSession = async (
signal?: AbortSignal,
): Promise<void> => {
const { connection, sessionId } = targetSession;
for (const [index, domain] of enabledDomains.entries()) {
if (signal?.aborted === true) {
await closeCancelledTargetSession(
targetSession,
enabledDomains.slice(index),
);
return;
}
try {
await connection.send(`${domain}.disable`, {}, sessionId, signal);
} catch (cause: unknown) {
// best-effort cleanup: domain disable continues to detach/close boundary.
void cause;
}
}
if (signal?.aborted === true) {
await closeCancelledTargetSession(targetSession, []);
return;
}
if (sessionId !== undefined)
try {
await connection.send(
"Target.detachFromTarget",
{ sessionId },
undefined,
signal,
);
} catch (cause: unknown) {
// best-effort cleanup: closing REA's socket is the final cleanup boundary.
void cause;
}
await connection.close();
};
const closeCancelledTargetSession = async (
{ connection, sessionId }: CdpTargetSession,
enabledDomains: readonly string[],
): Promise<void> => {
const commands = enabledDomains.map((domain) =>
connection.send(`${domain}.disable`, {}, sessionId),
);
if (sessionId !== undefined)
commands.push(connection.send("Target.detachFromTarget", { sessionId }));
const settled = Promise.allSettled(commands);
const cleanup = new AbortController();
const timeout = setTimeout(() => cleanup.abort(), 1_000);
timeout.unref();
try {
await connection.close();
if (signal?.aborted !== true) {
for (const domain of enabledDomains) {
if (cleanup.signal.aborted) break;
try {
await connection.send(
`${domain}.disable`,
{},
sessionId,
cleanup.signal,
);
} catch {
// Best-effort domain cleanup; transport close is the definitive boundary.
}
}
}
// Cancellation skips optional domain shutdown but still releases the
// attached session using its independent, bounded cleanup signal.
if (!cleanup.signal.aborted && sessionId !== undefined)
try {
await connection.send(
"Target.detachFromTarget",
{ sessionId },
undefined,
cleanup.signal,
);
} catch {
// Best-effort detach; transport close is the definitive boundary.
}
} finally {
await settled;
clearTimeout(timeout);
cleanup.abort();
await connection.close();
}
};
+40 -11
View File
@@ -365,20 +365,30 @@ const ingestWebMcpEvent = (options: WebMcpIngestOptions): void => {
transientFrames,
resolvedTransientFrames,
} = options;
if (
event.method !== "WebMCP.toolsAdded" &&
event.method !== "WebMCP.toolsRemoved"
)
return;
const params = recordValue(event.params);
if (params === undefined) return;
if (params === undefined) {
completeness.exclude("webmcp_tools", "invalid_protocol_value");
return;
}
if (event.method === "WebMCP.toolsRemoved") {
for (const removed of recordsValue(params.tools)) {
for (const removed of webMcpEventTools(params.tools, completeness)) {
const name = cdpStringValue(removed.name);
const frameId = cdpStringValue(removed.frameId);
if (name === undefined || frameId === undefined) continue;
if (name === undefined || frameId === undefined) {
completeness.exclude("webmcp_tools", "invalid_protocol_value");
continue;
}
for (const [key, tool] of tools)
if (tool.name === name && tool.frame_id === frameId) tools.delete(key);
}
return;
}
if (event.method !== "WebMCP.toolsAdded") return;
for (const declared of recordsValue(params.tools)) {
for (const declared of webMcpEventTools(params.tools, completeness)) {
const frameId = cdpStringValue(declared.frameId);
if (frameId !== undefined && transientFrames.has(frameId)) {
if (!resolvedTransientFrames.has(frameId)) continue;
@@ -391,6 +401,26 @@ const ingestWebMcpEvent = (options: WebMcpIngestOptions): void => {
}
};
const webMcpEventTools = (
value: unknown,
completeness: CdpCaptureCompleteness,
): UnknownRecord[] => {
if (!Array.isArray(value)) {
completeness.exclude("webmcp_tools", "invalid_protocol_value");
return [];
}
const records: UnknownRecord[] = [];
for (const item of value) {
const record = recordValue(item);
if (record === undefined) {
completeness.exclude("webmcp_tools", "invalid_protocol_value");
continue;
}
records.push(record);
}
return records;
};
const normalizeTool = (
value: UnknownRecord,
input: DiscoverWebMcpToolsInput,
@@ -403,12 +433,11 @@ const normalizeTool = (
const frameId = cdpStringValue(value.frameId);
const name = cdpStringValue(value.name);
const frame = frameId === undefined ? undefined : frames.get(frameId);
if (
frameId === undefined ||
name === undefined ||
frame?.origin === null ||
frame === undefined
) {
if (frameId === undefined || name === undefined) {
completeness.exclude("webmcp_tools", "invalid_protocol_value");
return undefined;
}
if (frame?.origin === null || frame === undefined) {
completeness.exclude("webmcp_tools", "out_of_target_scope");
return undefined;
}
+8 -19
View File
@@ -3,6 +3,11 @@ import { open, realpath, stat } from "node:fs/promises";
import { isAbsolute, join, sep } from "node:path";
import { fileURLToPath } from "node:url";
import {
admitAsarHeader,
MAX_ASAR_HEADER_BYTES,
} from "../artifacts/AsarHeader.js";
/** Resolve one local file URL while rejecting remote hosts and encoded separators. */
export const authorizedElectronFile = async (
value: string,
@@ -77,7 +82,6 @@ const authorizedAsarMember = async (
type ArchiveIdentity = Pick<Stats, "dev" | "ino" | "size" | "mtimeMs">;
// This is a retained-memory budget for JSON headers, shared across all archives.
const MAX_HEADER_BYTES = 16 * 1024 * 1024;
const MAX_CACHED_HEADERS = 8;
const headerCache = new Map<
string,
@@ -105,23 +109,8 @@ const asarFiles = async (
const opened = await handle.stat();
if (!opened.isFile() || archiveIdentity(opened) !== identity)
throw new Error("ASAR identity changed before header read");
const prefix = Buffer.alloc(16);
const read = await handle.read(prefix, 0, prefix.length, 0);
if (read.bytesRead !== prefix.length || prefix.readUInt32LE(0) !== 4)
throw new Error("Truncated or invalid ASAR size pickle");
const headerBytes = prefix.readUInt32LE(4);
const payloadBytes = prefix.readUInt32LE(8);
jsonBytes = prefix.readUInt32LE(12);
if (
headerBytes < 8 ||
headerBytes > opened.size - 8 ||
payloadBytes !== headerBytes - 4 ||
jsonBytes > payloadBytes - 4 ||
jsonBytes > MAX_HEADER_BYTES
)
throw new Error(
"ASAR JSON header leaves the file or exceeds its memory budget",
);
const admitted = await admitAsarHeader(handle);
jsonBytes = admitted.jsonBytes;
const json = Buffer.alloc(jsonBytes);
let offset = 0;
while (offset < json.length) {
@@ -151,7 +140,7 @@ const asarFiles = async (
[...headerCache.values()].reduce((sum, entry) => sum + entry.bytes, 0);
while (
headerCache.size > MAX_CACHED_HEADERS ||
retainedBytes() > MAX_HEADER_BYTES
retainedBytes() > MAX_ASAR_HEADER_BYTES
)
headerCache.delete(headerCache.keys().next().value ?? archive);
return files;
@@ -229,14 +229,15 @@ const runScenario = async (
openSession: BrowserScenarioSessionOpener,
scenario: BrowserScenario,
options: ExecutionOptions,
retainCleanup: (close: () => Promise<unknown>) => void,
): Promise<BrowserScenarioCapture> => {
if (options.signal?.aborted === true)
throw new BrowserObservationError(OPERATION, "cancelled");
const startedAt = Date.now();
const session = await openSession(
scenario,
options.signal === undefined ? {} : { signal: options.signal },
);
const session = await openSession(scenario, {
...(options.signal === undefined ? {} : { signal: options.signal }),
retainCleanup,
});
const steps: BrowserScenarioStep[] = [];
let operationOutcome:
| { readonly ok: true }
@@ -279,6 +280,7 @@ const runScenario = async (
cleanupOutcome = { ok: true, cleanup: await session.close() };
} catch (cause: unknown) {
cleanupOutcome = { ok: false, cause };
retainCleanup(() => session.close());
}
if (!cleanupOutcome.ok) {
const cleanup = browserScenarioCleanupObservation(cleanupOutcome.cause);
@@ -337,18 +339,46 @@ const runScenario = async (
/** Controlled Playwright/CDP scenario driver with exact process ownership. */
export class PlaywrightBrowserScenarioProvider implements BrowserScenarioCapturePort {
#tail: Promise<void> = Promise.resolve();
#closing = false;
#closePromise: Promise<void> | undefined;
readonly #pendingCleanup = new Set<() => Promise<unknown>>();
constructor(private readonly openSession: BrowserScenarioSessionOpener) {}
identity(): ProviderIdentity {
return PLAYWRIGHT_BROWSER_SCENARIO_PROVIDER_IDENTITY;
}
async captureScenario(
captureScenario(
scenario: BrowserScenario,
options: ExecutionOptions = {},
): Promise<Result<BrowserScenarioCapture, AnalysisError>> {
if (this.#closing) return Promise.resolve(err(this.#closedError()));
const operation = this.#tail.then(() =>
this.#closing
? err(this.#closedError())
: this.#captureScenario(scenario, options),
);
this.#tail = operation.then(
() => undefined,
() => undefined,
);
return operation;
}
async #captureScenario(
scenario: BrowserScenario,
options: ExecutionOptions,
): Promise<Result<BrowserScenarioCapture, AnalysisError>> {
const pending = await this.#retryCleanup();
if (pending !== undefined) return err(pending);
try {
return ok(await runScenario(this.openSession, scenario, options));
return ok(
await runScenario(this.openSession, scenario, options, (close) => {
this.#pendingCleanup.add(close);
}),
);
} catch (cause: unknown) {
if (cause instanceof AnalysisError) return err(cause);
return err(
@@ -360,4 +390,43 @@ export class PlaywrightBrowserScenarioProvider implements BrowserScenarioCapture
);
}
}
async close(): Promise<void> {
this.#closing = true;
this.#closePromise ??= this.#tail
.then(async () => {
const failure = await this.#retryCleanup();
if (failure !== undefined) throw failure;
})
.catch((cause: unknown) => {
this.#closePromise = undefined;
throw cause;
});
return this.#closePromise;
}
#closedError(): AnalysisError {
return new BrowserObservationError(OPERATION, "cleanup_failed", {
detail: "Browser scenario provider is closing.",
cleanup: { reason: "Provider is closing", resources: [] },
});
}
async #retryCleanup(): Promise<AnalysisError | undefined> {
for (const pending of this.#pendingCleanup) {
try {
await pending();
this.#pendingCleanup.delete(pending);
} catch (cause: unknown) {
return cause instanceof AnalysisError
? cause
: new ProviderAdapterError(
PLAYWRIGHT_BROWSER_SCENARIO_PROVIDER_IDENTITY.id,
OPERATION,
{ cause },
);
}
}
return undefined;
}
}
+30 -7
View File
@@ -97,6 +97,9 @@ export class PlaywrightScenarioBrowserCleanupOwner {
try {
await eventFinalization;
} catch (cause: unknown) {
// A failed finalization stays failed: the memoized attempt rejects on
// every retry so cleanup can never report success for unfinalized
// events.
failures.push(cause);
}
if (!this.#browserClosed) {
@@ -380,6 +383,7 @@ const openAttachedScenarioBrowser = async (
environment: BrowserScenario["environment"],
launcher: ScenarioBrowserLauncher,
signal: AbortSignal | undefined,
retainCleanup: ((close: () => Promise<unknown>) => void) | undefined,
): Promise<OpenedScenarioBrowser> => {
const lease = await acquireAttachedScenario(browserRequest, signal);
let cleanupOwner: PlaywrightScenarioBrowserCleanupOwner | undefined;
@@ -431,7 +435,13 @@ const openAttachedScenarioBrowser = async (
throw cause;
}
const cleanup = cleanupOwner;
return failBrowserScenarioOperation(() => cleanup.close(), cause);
const close = () => cleanup.close();
try {
return await failBrowserScenarioOperation(close, cause);
} catch (failure: unknown) {
if (failure !== cause) retainCleanup?.(close);
throw failure;
}
}
};
@@ -442,6 +452,7 @@ export const openPlaywrightScenarioBrowser = async (
options: {
readonly launcher?: ScenarioBrowserLauncher;
readonly signal?: AbortSignal | undefined;
readonly retainCleanup?: (close: () => Promise<unknown>) => void;
} = {},
): Promise<OpenedScenarioBrowser> => {
const launcher = options.launcher ?? chromium;
@@ -451,6 +462,7 @@ export const openPlaywrightScenarioBrowser = async (
scenario.environment,
launcher,
options.signal,
options.retainCleanup,
);
const profilePath = await mkdtemp(join(tmpdir(), "rea-browser-scenario-"));
@@ -478,11 +490,16 @@ export const openPlaywrightScenarioBrowser = async (
timeout: 0,
});
} catch (cause: unknown) {
return failBrowserScenarioOperation(
() => rm(profilePath, { recursive: true, force: true, maxRetries: 3 }),
cause,
["browser_profile"],
);
const cleanup = () =>
rm(profilePath, { recursive: true, force: true, maxRetries: 3 });
try {
return await failBrowserScenarioOperation(cleanup, cause, [
"browser_profile",
]);
} catch (failure: unknown) {
if (failure !== cause) options.retainCleanup?.(cleanup);
throw failure;
}
}
const cleanup = new PlaywrightScenarioBrowserCleanupOwner({
closeBrowser: () => context.close(),
@@ -502,6 +519,12 @@ export const openPlaywrightScenarioBrowser = async (
cleanup,
};
} catch (cause: unknown) {
return failBrowserScenarioOperation(() => cleanup.close(), cause);
const close = () => cleanup.close();
try {
return await failBrowserScenarioOperation(close, cause);
} catch (failure: unknown) {
if (failure !== cause) options.retainCleanup?.(close);
throw failure;
}
}
};
+30 -18
View File
@@ -127,6 +127,7 @@ export class PlaywrightScenarioSession implements BrowserScenarioSessionPort {
environment: Readonly<Record<string, string | undefined>>,
options: {
readonly signal?: AbortSignal;
readonly retainCleanup?: (close: () => Promise<unknown>) => void;
},
): Promise<PlaywrightScenarioSession> {
if (options.signal?.aborted === true)
@@ -137,6 +138,9 @@ export class PlaywrightScenarioSession implements BrowserScenarioSessionPort {
const startedAt = Date.now();
const opening = openPlaywrightScenarioBrowser(scenario, environment, {
signal: options.signal,
...(options.retainCleanup === undefined
? {}
: { retainCleanup: options.retainCleanup }),
});
let opened: OpenedScenarioBrowser;
let events: PlaywrightScenarioEvents | undefined;
@@ -147,14 +151,18 @@ export class PlaywrightScenarioSession implements BrowserScenarioSessionPort {
options.signal,
);
} catch (cause: unknown) {
void opening
.then((lateOpened) =>
lateOpened.cleanup.close(undefined, options.signal),
)
.catch((cause: unknown) => {
// best-effort cleanup: late-open cleanup must not mask the boundary failure.
void cause;
});
const lateCleanup = () =>
opening.then(
(lateOpened) => lateOpened.cleanup.close(undefined, options.signal),
() => undefined,
);
options.retainCleanup?.(() =>
withPlaywrightExecutionBoundary(lateCleanup, 1_000),
);
void lateCleanup().catch((cleanupCause: unknown) => {
// best-effort cleanup: retainCleanup keeps the retry capability.
void cleanupCause;
});
throw cause;
}
let session: PlaywrightScenarioSession;
@@ -179,16 +187,19 @@ export class PlaywrightScenarioSession implements BrowserScenarioSessionPort {
eventCapture: events,
});
} catch (cause: unknown) {
return failBrowserScenarioOperation(
() =>
opened.cleanup.close(
events === undefined
? undefined
: () => events?.finish() ?? Promise.resolve(),
options.signal,
),
cause,
);
const cleanup = () =>
opened.cleanup.close(
events === undefined
? undefined
: () => events?.finish() ?? Promise.resolve(),
options.signal,
);
try {
return await failBrowserScenarioOperation(cleanup, cause);
} catch (failure: unknown) {
if (failure !== cause) options.retainCleanup?.(cleanup);
throw failure;
}
}
try {
await withPlaywrightExecutionBoundary(
@@ -212,6 +223,7 @@ export class PlaywrightScenarioSession implements BrowserScenarioSessionPort {
} catch (failure: unknown) {
cleanup = "incomplete";
cleanupFailure = { cause: failure };
options.retainCleanup?.(() => session.close());
}
throw browserScenarioOperationFailure(
cause,
+7 -2
View File
@@ -76,7 +76,12 @@ export const publishWebScripts = async (
bytes: bytes.length,
},
});
await tree.write("manifest.json", Readable.from([bytes]), digest, signal);
await tree.write(
"manifest.json",
Readable.from([bytes]),
{ sha256: digest, bytes: bytes.byteLength },
signal,
);
signal?.throwIfAborted();
await tree.commit();
return result;
@@ -109,7 +114,7 @@ const publishPlannedScript = async (
tree.write(
`files/${relativePath}`,
Readable.from([content.bytes]),
content.sha256,
{ sha256: content.sha256, bytes: content.bytes.byteLength },
signal,
);
try {
+18 -8
View File
@@ -24,14 +24,24 @@ export const registerEvmCommands = (
.describe("Selected carrier representation"),
}),
run: ({ args }) =>
withCommandCancellation((signal) =>
logCliCommand(logger, CLI_COMMANDS.inspectEvmInterface, async () => {
const result = await service.inspect(
{ path: resolve(args.path), encoding: args.encoding },
{ signal },
withCommandCancellation(async (signal) => {
try {
return await logCliCommand(
logger,
CLI_COMMANDS.inspectEvmInterface,
async () => {
const result = await service.inspect(
{ path: resolve(args.path), encoding: args.encoding },
{ signal },
);
return result.ok
? result.value
: projectAnalysisError(result.error);
},
);
return result.ok ? result.value : projectAnalysisError(result.error);
}),
),
} finally {
await service.close();
}
}),
});
};
+6 -2
View File
@@ -26,8 +26,12 @@ export const registerFirmwareCommands = (
) =>
withCommandCancellation((signal) =>
logCliCommand(logger, name, async () => {
const result = await service.execute(operation, input, { signal });
return result.ok ? result.value : projectAnalysisError(result.error);
try {
const result = await service.execute(operation, input, { signal });
return result.ok ? result.value : projectAnalysisError(result.error);
} finally {
await service.close();
}
}),
);
const path = z.string().describe("Local firmware path; never executed");
+28 -24
View File
@@ -41,29 +41,33 @@ export const registerJavaScriptRecoveryCommands = (
),
}),
run: ({ args, options }) =>
withCommandCancellation((signal) =>
logCliCommand(
logger,
CLI_COMMANDS.recoverJavaScriptSources,
async () => {
const result = await service.recover(
{
path: args.path,
output_directory: args.outputDirectory,
...(options.extractionMode === undefined
? {}
: { extraction_mode: options.extractionMode }),
...(options.rewriteLevel === undefined
? {}
: { rewrite_level: options.rewriteLevel }),
},
{ signal },
);
return result.ok
? result.value
: projectAnalysisError(result.error);
},
),
),
withCommandCancellation(async (signal) => {
try {
return await logCliCommand(
logger,
CLI_COMMANDS.recoverJavaScriptSources,
async () => {
const result = await service.recover(
{
path: args.path,
output_directory: args.outputDirectory,
...(options.extractionMode === undefined
? {}
: { extraction_mode: options.extractionMode }),
...(options.rewriteLevel === undefined
? {}
: { rewrite_level: options.rewriteLevel }),
},
{ signal },
);
return result.ok
? result.value
: projectAnalysisError(result.error);
},
);
} finally {
await service.close();
}
}),
});
};
+17 -2
View File
@@ -39,6 +39,7 @@ const supportedClientSchema = z
(candidate) => supportedClientIds.some((name) => name === candidate),
"Unsupported agent integration",
);
const skillClientSchema = z.union([z.literal("shared"), supportedClientSchema]);
/** Register setup, doctor, uninstall, and update CLI commands. */
export const registerSetupCommands = (
@@ -72,6 +73,10 @@ const registerSetupCommand = (
.array(supportedClientSchema)
.default([])
.describe("Agent integration to configure; repeat for multiple agents"),
skillClient: z
.array(skillClientSchema)
.default([])
.describe("Existing client skill destination to update; repeatable"),
allDetected: z
.boolean()
.default(false)
@@ -186,6 +191,7 @@ interface SetupCommandOptions {
readonly yes: boolean;
readonly installHopper: boolean;
readonly client: readonly string[];
readonly skillClient: readonly string[];
readonly allDetected: boolean;
readonly skill?: boolean | undefined;
readonly dryRun: boolean;
@@ -224,8 +230,13 @@ const setupRunOptions = (
): SetupOptions => {
const { options } = input;
const agentIntegrationSelected =
options.allDetected || options.client.length > 0;
const hasSelectedClients = hasExplicitScope && !options.allDetected;
options.allDetected ||
options.client.length > 0 ||
options.skillClient.length > 0;
const hasSelectedClients =
hasExplicitScope &&
!options.allDetected &&
(options.client.length > 0 || options.skillClient.length > 0);
return {
approved: options.yes && !options.dryRun,
installHopper: options.installHopper,
@@ -234,6 +245,9 @@ const setupRunOptions = (
allDetectedClients: options.allDetected,
proposeHopper: interactive || options.installHopper,
...(hasSelectedClients ? { clientIds: options.client } : {}),
...(options.skillClient.length > 0
? { skillClientIds: options.skillClient }
: {}),
...(hasExplicitScope
? { installSkill: options.skill ?? agentIntegrationSelected }
: {}),
@@ -264,6 +278,7 @@ const setupIsInteractive = (
const setupHasExplicitScope = (options: SetupCommandOptions): boolean =>
options.allDetected ||
options.client.length > 0 ||
options.skillClient.length > 0 ||
options.skill !== undefined ||
options.installHopper;
+25 -12
View File
@@ -38,19 +38,32 @@ export const registerWebSourceLocationCommands = (
.describe("Zero-based UTF-16 generated column"),
}),
run: ({ args }) =>
withCommandCancellation((signal) =>
logCliCommand(logger, CLI_COMMANDS.traceWebSourceLocation, async () => {
const result = await service.trace(
{
manifest_path: args.manifestPath,
script_index: args.scriptIndex,
source_map: { path: args.sourceMapPath, url: args.sourceMapUrl },
generated_position: { line: args.line, column: args.column },
withCommandCancellation(async (signal) => {
try {
return await logCliCommand(
logger,
CLI_COMMANDS.traceWebSourceLocation,
async () => {
const result = await service.trace(
{
manifest_path: args.manifestPath,
script_index: args.scriptIndex,
source_map: {
path: args.sourceMapPath,
url: args.sourceMapUrl,
},
generated_position: { line: args.line, column: args.column },
},
{ signal },
);
return result.ok
? result.value
: projectAnalysisError(result.error);
},
{ signal },
);
return result.ok ? result.value : projectAnalysisError(result.error);
}),
),
} finally {
await service.close();
}
}),
});
};
+2 -2
View File
@@ -1,6 +1,6 @@
import { resolve } from "node:path";
import { readCliJsonFile } from "./cliJsonFile.js";
import { readJsonInputFile } from "./application/JsonInputFile.js";
import { NonRegularFileReadError } from "./filesystem/RegularFile.js";
import {
AnalysisAccessDeniedError,
@@ -21,7 +21,7 @@ export const parseCliJsonInput = async (
const inline = safeParseJson(value);
if (inline.ok) return { ok: true, value: inline.value };
try {
const parsed = await readCliJsonFile(value, operation);
const parsed = await readJsonInputFile(value, operation);
return parsed.ok
? { ok: true, value: parsed.value }
: {
+82
View File
@@ -0,0 +1,82 @@
import { expect, it } from "vitest";
import { adbInputSchemas } from "./adbDeviceAnalysis.js";
it("preserves ADB key, action and component admission when making advertised patterns portable", () => {
const serial = "emulator-5554";
const variants = [
{
previous: /^[A-Za-z0-9._-]+$/u,
max: 256,
accepts: (key: string) =>
adbInputSchemas.read_adb_setting.safeParse({
serial,
namespace: "global",
key,
}).success,
},
{
previous: /^[A-Za-z0-9._-]+$/u,
max: 256,
accepts: (key: string) =>
adbInputSchemas.start_adb_activity.safeParse({
serial,
action: "android.intent.action.VIEW",
extras: [{ key, type: "string", value: "payload" }],
}).success,
},
{
previous: /^[A-Za-z][A-Za-z0-9._-]*$/u,
max: 256,
accepts: (action: string) =>
adbInputSchemas.start_adb_activity.safeParse({ serial, action })
.success,
},
{
previous: /^[A-Za-z0-9._-]+(\/[A-Za-z0-9._-]*)?$/u,
max: 512,
accepts: (component: string) =>
adbInputSchemas.start_adb_activity.safeParse({
serial,
action: "android.intent.action.VIEW",
component,
}).success,
},
];
const samples = [
"",
"screen_brightness",
"0",
"_",
".",
"-",
"android.intent.action.VIEW",
"com.example/.MainActivity",
"com.example/",
"com.example/a/b",
"a\n",
"a\r\n",
"é",
"aé",
"a b",
"a\\b",
"a".repeat(256),
"a".repeat(257),
"a".repeat(512),
"a".repeat(513),
];
for (let code = 0; code < 128; code += 1) {
const character = String.fromCharCode(code);
samples.push(character, `a${character}`, `a${character}b`);
}
for (const variant of variants)
for (const sample of samples)
expect(
variant.accepts(sample),
JSON.stringify({ pattern: variant.previous.source, sample }),
).toBe(
sample.length > 0 &&
sample.length <= variant.max &&
variant.previous.test(sample),
);
});
+46 -15
View File
@@ -1,5 +1,6 @@
import { z } from "zod";
import { localPathStringSchema } from "../localPath.js";
import { IDENTIFIER_PATTERN } from "../stringPatterns.js";
/**
* Device serials as adb prints them: emulator-5554, USB identifiers, and
@@ -29,6 +30,9 @@ const packageNameSchema = z
"Package name must be dot-separated Java identifiers",
);
const keyStringSchema = z.string().min(1).max(256);
const keyPattern = /^[A-Za-z0-9._\x2d]+$/u;
/**
* Caller intent for ADB-backed device inspection and acquisition.
*
@@ -174,14 +178,8 @@ export const adbInputSchemas = {
namespace: z
.enum(["system", "secure", "global"])
.describe("Settings namespace to read"),
key: z
.string()
.min(1)
.max(256)
.regex(
/^[A-Za-z0-9._\-]+$/u,
"Setting keys are restricted to identifiers",
)
key: keyStringSchema
.regex(keyPattern, "Setting keys are restricted to identifiers")
.describe("Setting key to read"),
}),
collect_adb_bugreport: z.strictObject({
@@ -241,7 +239,7 @@ export const adbInputSchemas = {
.min(1)
.max(256)
.regex(
/^[A-Za-z][A-Za-z0-9._\-]*$/u,
IDENTIFIER_PATTERN,
"Intent action must be a dotted identifier such as android.intent.action.VIEW",
)
.describe("Intent action to start"),
@@ -257,7 +255,7 @@ export const adbInputSchemas = {
.min(1)
.max(512)
.regex(
/^[A-Za-z0-9._\-]+(\/[A-Za-z0-9._\-]*)?$/u,
/^[A-Za-z0-9._\x2d]+(\/[A-Za-z0-9._\x2d]*)?$/u,
"Component must be package/activity form",
)
.optional()
@@ -265,11 +263,8 @@ export const adbInputSchemas = {
extras: z
.array(
z.strictObject({
key: z
.string()
.min(1)
.max(256)
.regex(/^[A-Za-z0-9._\-]+$/u, "Extra keys are identifiers")
key: keyStringSchema
.regex(keyPattern, "Extra keys are identifiers")
.describe("Extra key"),
type: z
.enum(["string", "boolean", "int", "long", "float"])
@@ -704,3 +699,39 @@ export const adbResultSchemas = {
package_name: packageNameSchema,
}),
} as const;
/** Acquired package facts retained when cancellation leaves caller-owned output. */
export type AdbPackagePullPartialObservation = {
readonly provider_id: "adb";
readonly operation: "pull_adb_package";
readonly result: z.infer<typeof adbResultSchemas.pull_adb_package>;
};
/** A completed acquisition whose digest read was cancelled before verification. */
export type AdbAcquiredFilePartialObservation =
| {
readonly provider_id: "adb";
readonly operation: "pull_adb_file";
readonly result: {
readonly client: z.infer<
typeof adbResultSchemas.pull_adb_file
>["client"];
readonly serial: string;
readonly local_path: string;
readonly device_path: string;
readonly digest_status: "unknown";
};
}
| {
readonly provider_id: "adb";
readonly operation: "collect_adb_bugreport";
readonly result: {
readonly client: z.infer<
typeof adbResultSchemas.collect_adb_bugreport
>["client"];
readonly serial: string;
readonly local_path: string;
readonly adb_reported_path: string;
readonly digest_status: "unknown";
};
};
+9 -1
View File
@@ -99,6 +99,11 @@ import type { BrowserScenarioPartialObservation } from "./browserScenarioCapture
import type { Evidence } from "./evidence.js";
import type { AndroidPartialObservation } from "./android/androidPartialObservation.js";
import type { FirmwarePartialObservation } from "./firmware/firmwareAnalysis.js";
import type {
AdbAcquiredFilePartialObservation,
AdbPackagePullPartialObservation,
} from "./adb/adbDeviceAnalysis.js";
import type { NativeUiObservationResult } from "./native/nativeUiObservation.js";
/** Provider-neutral evidence collected before a typed analysis failure. */
export type AnalysisPartialObservation =
@@ -111,4 +116,7 @@ export type AnalysisPartialObservation =
| ArtifactInventoryPartialObservation
| Evidence
| AndroidPartialObservation
| FirmwarePartialObservation;
| FirmwarePartialObservation
| NativeUiObservationResult
| AdbAcquiredFilePartialObservation
| AdbPackagePullPartialObservation;
+39
View File
@@ -0,0 +1,39 @@
import { ProviderAdapterError } from "./providerAdapterError.js";
import { projectAnalysisError } from "./analysisErrorProjection.js";
import type { AnalysisError } from "./analysisErrorBase.js";
/** Preserve the original lifecycle failure alongside incomplete cleanup. */
export const analysisErrorWithCleanupFailure = (
primary: AnalysisError,
cleanup: AnalysisError,
operation = "close_binary",
): AnalysisError => {
const projected = projectAnalysisError(cleanup);
return new ProviderAdapterError(
cleanup instanceof ProviderAdapterError ? cleanup.providerId : "rea",
operation,
{
cause: primary,
userMessage: projectAnalysisError(primary).message,
...(primary.capturedOutput === undefined
? {}
: { capturedOutput: primary.capturedOutput }),
...(primary.partialObservation === undefined
? {}
: { partialObservation: primary.partialObservation }),
cleanup: {
reason: projected.message,
resources: [
...new Set([
...primary.cleanupResources,
...cleanup.cleanupResources,
]),
],
},
diagnostics: {
primary_error: projectAnalysisError(primary),
cleanup_error: projected,
},
},
);
};
+20 -18
View File
@@ -66,7 +66,9 @@ export const buildInterfaceBuilderAnalysis = (input: {
let omittedHierarchyReferences = 0;
for (const document of input.documents.slice(0, input.limits.max_documents)) {
const parsed = parseInterfaceBuilderRecords(document.raw);
const prefix = `ib:${document.relativePath}:`;
const prefix = `ib:${JSON.stringify(document.relativePath)}:`;
const graphId = (kind: string, parts: readonly unknown[] = []): string =>
`${prefix}${kind}:${JSON.stringify(parts)}`;
const rootId = `${prefix}root`;
const objectIds = new Set(parsed.objects.map(({ id }) => id));
const evidenceFor = (description: string) => [
@@ -104,7 +106,7 @@ export const buildInterfaceBuilderAnalysis = (input: {
addNode(rootNode);
let documentObjectCount = 0;
for (const object of parsed.objects.slice(0, input.limits.max_objects)) {
const id = `${prefix}object:${object.id}`;
const id = graphId("object", [object.id]);
const kind = object.kind === "other" ? "unknown" : object.kind;
if (
!addNode({
@@ -129,7 +131,7 @@ export const buildInterfaceBuilderAnalysis = (input: {
let documentOmittedConnections = 0;
let documentOmittedHierarchyEdges = 0;
const objectNodeId = (objectId: string): string =>
`${prefix}object:${objectId}`;
graphId("object", [objectId]);
const addEdge = (
edge: (typeof edges)[number],
isHierarchy: boolean,
@@ -238,7 +240,7 @@ export const buildInterfaceBuilderAnalysis = (input: {
const known = objectIds.has(objectId);
const childId = known
? objectNodeId(objectId)
: `${prefix}unknown:${objectId}`;
: graphId("unknown", [objectId]);
if (
!known &&
!addNode({
@@ -252,7 +254,7 @@ export const buildInterfaceBuilderAnalysis = (input: {
)
hierarchyTruncated = true;
connect(
`${prefix}hierarchy:${parentId}:${childId}`,
graphId("hierarchy", [parentId, childId]),
parentId,
childId,
"contains",
@@ -281,11 +283,11 @@ export const buildInterfaceBuilderAnalysis = (input: {
const destinationId =
item.destination_id === null ? null : objectNodeId(item.destination_id);
if (item.kind === "action") {
const actionId = `${prefix}action:${item.id}`;
const actionId = graphId("action", [item.id]);
const selectorId =
item.label === null
? null
: `${prefix}selector:${item.destination_id ?? "unknown"}:${item.label}`;
: graphId("selector", [item.destination_id, item.label]);
addNode({
id: actionId,
kind: "action",
@@ -299,7 +301,7 @@ export const buildInterfaceBuilderAnalysis = (input: {
evidence: evidenceFor(`action ${item.id}`),
});
connect(
`${actionId}:source`,
graphId("action-edge", [item.id, "source"]),
sourceId,
actionId,
"target_action",
@@ -315,7 +317,7 @@ export const buildInterfaceBuilderAnalysis = (input: {
evidence: evidenceFor(`selector ${item.label}`),
});
connect(
`${actionId}:selector`,
graphId("action-edge", [item.id, "selector"]),
actionId,
selectorId,
"target_action",
@@ -324,7 +326,7 @@ export const buildInterfaceBuilderAnalysis = (input: {
}
if (destinationId === null || !nodes.has(destinationId))
connect(
`${actionId}:destination`,
graphId("action-edge", [item.id, "destination"]),
actionId,
null,
"target_action",
@@ -333,14 +335,14 @@ export const buildInterfaceBuilderAnalysis = (input: {
);
else
connect(
`${actionId}:destination`,
graphId("action-edge", [item.id, "destination"]),
actionId,
destinationId,
"target_action",
`action destination ${item.destination_id}`,
);
} else if (item.kind === "outlet") {
const outletId = `${prefix}outlet:${item.id}`;
const outletId = graphId("outlet", [item.id]);
addNode({
id: outletId,
kind: "outlet",
@@ -350,7 +352,7 @@ export const buildInterfaceBuilderAnalysis = (input: {
evidence: evidenceFor(`outlet ${item.id}`),
});
connect(
`${outletId}:source`,
graphId("outlet-edge", [item.id, "source"]),
sourceId,
outletId,
"contains",
@@ -358,7 +360,7 @@ export const buildInterfaceBuilderAnalysis = (input: {
);
if (destinationId === null || !nodes.has(destinationId))
connect(
`${outletId}:destination`,
graphId("outlet-edge", [item.id, "destination"]),
outletId,
null,
"outlet_to",
@@ -367,7 +369,7 @@ export const buildInterfaceBuilderAnalysis = (input: {
);
else
connect(
`${outletId}:destination`,
graphId("outlet-edge", [item.id, "destination"]),
outletId,
destinationId,
"outlet_to",
@@ -376,7 +378,7 @@ export const buildInterfaceBuilderAnalysis = (input: {
} else if (item.kind === "segue") {
if (destinationId === null || !nodes.has(destinationId))
connect(
`${prefix}segue:${index}`,
graphId("segue", [index]),
sourceId,
null,
"segue_to",
@@ -385,7 +387,7 @@ export const buildInterfaceBuilderAnalysis = (input: {
);
else
connect(
`${prefix}segue:${index}`,
graphId("segue", [index]),
sourceId,
destinationId,
"segue_to",
@@ -393,7 +395,7 @@ export const buildInterfaceBuilderAnalysis = (input: {
);
} else {
connect(
`${prefix}unknown-connection:${index}`,
graphId("unknown-connection", [index]),
sourceId,
null,
"contains",
@@ -15,7 +15,7 @@ import {
} from "./javascriptStaticAnalysisHelpers.js";
import {
propertyName,
semanticStaticPropertyName,
semanticStaticPropertyKey,
} from "./javascriptAstValues.js";
import type { JavaScriptFindingContext } from "./javascriptStaticAnalysisState.js";
@@ -26,6 +26,8 @@ interface NativeBindingInput {
readonly kind: ElectronNativeAddonBindingFinding["binding_kind"];
readonly moduleKind: ElectronNativeAddonBindingFinding["module_kind"];
readonly members: readonly string[];
readonly namespaceAccess: boolean;
readonly dynamicMemberAccess: boolean;
}
/** Inspect JavaScript-side imports and re-exports of native .node addons. */
@@ -45,10 +47,10 @@ const inspectImport = (
context: JavaScriptFindingContext,
): void => {
if (!isNativeSpecifier(node.source.value, "import")) return;
const members = node.specifiers.map((specifier) => {
if (t.isImportDefaultSpecifier(specifier)) return "default";
if (t.isImportNamespaceSpecifier(specifier)) return "*";
return propertyName(specifier.imported) || "[dynamic-import]";
const members = node.specifiers.flatMap((specifier) => {
if (t.isImportDefaultSpecifier(specifier)) return ["default"];
if (t.isImportNamespaceSpecifier(specifier)) return [];
return [propertyName(specifier.imported)];
});
addBinding({
context,
@@ -57,6 +59,10 @@ const inspectImport = (
kind: "import",
moduleKind: "import",
members,
namespaceAccess: node.specifiers.some((value) =>
t.isImportNamespaceSpecifier(value),
),
dynamicMemberAccess: false,
});
};
@@ -70,9 +76,10 @@ const inspectNamedExport = (
!isNativeSpecifier(node.source.value, "import")
)
return;
const members = node.specifiers.map((specifier) => {
if (t.isExportSpecifier(specifier)) return propertyName(specifier.local);
return "*";
const members = node.specifiers.flatMap((specifier) => {
if (t.isExportSpecifier(specifier)) return [propertyName(specifier.local)];
if (t.isExportDefaultSpecifier(specifier)) return ["default"];
return [];
});
addBinding({
context,
@@ -81,6 +88,10 @@ const inspectNamedExport = (
kind: "re-export",
moduleKind: "import",
members,
namespaceAccess: node.specifiers.some((value) =>
t.isExportNamespaceSpecifier(value),
),
dynamicMemberAccess: false,
});
};
@@ -99,7 +110,9 @@ const inspectExportAll = (
specifier: node.source.value,
kind: "re-export",
moduleKind: "import",
members: ["*"],
members: [],
namespaceAccess: true,
dynamicMemberAccess: false,
});
};
@@ -115,8 +128,7 @@ const inspectRequireBinding = (
specifier: required.specifier,
kind: "require",
moduleKind: "require",
members:
required.member === null ? bindingMembers(node.id) : [required.member],
...(required.hasMemberAccess ? required.access : bindingMembers(node.id)),
});
};
@@ -132,27 +144,33 @@ const inspectReExport = (
specifier: required.specifier,
kind: "re-export",
moduleKind: "require",
members: [required.member ?? exportedMember(node.left) ?? "*"],
...required.access,
});
};
const addBinding = (input: NativeBindingInput): void => {
const { context, node, specifier, kind } = input;
const unique = [
...new Set(input.members.filter((member) => member !== "")),
].sort(compareUnicodeCodePoints);
const members = unique.length === 0 ? ["*"] : unique;
const members = [...new Set(input.members)].sort(compareUnicodeCodePoints);
addLocatedFinding(context, {
collection: context.accumulator.nativeAddonBindings,
// Members are a variable-length source-derived list: compositeKey keeps
// ["a\0b"] distinct from ["a", "b"] instead of collapsing one binding.
key: compositeKey(["native-addon-binding", kind, specifier, members]),
key: compositeKey([
"native-addon-binding",
kind,
specifier,
members,
input.namespaceAccess,
input.dynamicMemberAccess,
]),
node,
value: {
specifier,
binding_kind: kind,
module_kind: input.moduleKind,
members,
namespace_access: input.namespaceAccess,
dynamic_member_access: input.dynamicMemberAccess,
module_key: null,
location: range(node),
},
@@ -162,13 +180,20 @@ const addBinding = (input: NativeBindingInput): void => {
const nativeRequire = (
node: t.Node | null | undefined,
):
| { readonly specifier: string; readonly member: string | null }
| {
readonly specifier: string;
readonly hasMemberAccess: boolean;
readonly access: NativeMemberAccess;
}
| undefined => {
let member: string | null = null;
let hasMemberAccess = false;
while (t.isMemberExpression(node) || t.isOptionalMemberExpression(node)) {
if (!t.isNode(node.object)) return undefined;
const property = semanticStaticPropertyName(node.property, node.computed);
if (member === null && property !== "") member = property;
const property = semanticStaticPropertyKey(node.property, node.computed);
// Walking inward leaves the first member consumed from the addon itself.
member = property;
hasMemberAccess = true;
node = node.object;
}
if (!t.isCallExpression(node)) return undefined;
@@ -181,34 +206,52 @@ const nativeRequire = (
return undefined;
const specifier = stringValue(argumentNode(node.arguments[0]));
return specifier !== undefined && isNativeSpecifier(specifier, "require")
? { specifier, member }
? {
specifier,
hasMemberAccess,
access: {
members: member === null ? [] : [member],
namespaceAccess: !hasMemberAccess,
dynamicMemberAccess: hasMemberAccess && member === null,
},
}
: undefined;
};
const bindingMembers = (pattern: t.Node): string[] => {
if (t.isIdentifier(pattern)) return ["*"];
if (!t.isObjectPattern(pattern)) return ["*"];
return pattern.properties.map((property) => {
if (t.isRestElement(property)) return "*";
return propertyName(property.key) || "[dynamic-import]";
});
interface NativeMemberAccess {
readonly members: readonly string[];
readonly namespaceAccess: boolean;
readonly dynamicMemberAccess: boolean;
}
const bindingMembers = (pattern: t.Node): NativeMemberAccess => {
if (!t.isObjectPattern(pattern))
return { members: [], namespaceAccess: true, dynamicMemberAccess: false };
const members: string[] = [];
let namespaceAccess = false;
let dynamicMemberAccess = false;
for (const property of pattern.properties) {
if (t.isRestElement(property)) {
namespaceAccess = true;
continue;
}
const key = semanticStaticPropertyKey(property.key, property.computed);
if (key === null) dynamicMemberAccess = true;
else members.push(key);
}
return { members, namespaceAccess, dynamicMemberAccess };
};
const isModuleExport = (node: t.Node): boolean => {
const name = calleeName(node);
return (
name === "module.exports" ||
name.startsWith("module.exports.") ||
name.startsWith("exports.")
);
};
const exportedMember = (node: t.Node): string | undefined => {
const name = calleeName(node);
if (name.startsWith("module.exports."))
return name.slice("module.exports.".length);
if (name.startsWith("exports.")) return name.slice("exports.".length);
return undefined;
const keys: (string | null)[] = [];
while (t.isMemberExpression(node) || t.isOptionalMemberExpression(node)) {
keys.unshift(semanticStaticPropertyKey(node.property, node.computed));
node = node.object;
}
if (!t.isIdentifier(node)) return false;
return node.name === "exports"
? keys.length > 0
: node.name === "module" && keys[0] === "exports";
};
const isNativeSpecifier = (
@@ -39,6 +39,8 @@ export type ElectronBrowserWindowFinding = ElectronBrowserWindowPreload & {
readonly web_preferences_status: "object-literal" | "dynamic" | "missing";
readonly web_preferences: readonly ElectronWebPreference[];
readonly module_key: string | null;
/** Recovered factory runtime; absent in older or unbundled findings. */
readonly module_runtime?: string;
readonly location: JavaScriptSourceRange;
};
@@ -56,6 +58,8 @@ export type ElectronContextBridgeFinding = ElectronContextBridgeApiKey & {
readonly members: readonly string[];
readonly unknown_members: number;
readonly module_key: string | null;
/** Recovered factory runtime; absent in older or unbundled findings. */
readonly module_runtime?: string;
readonly location: JavaScriptSourceRange;
};
@@ -109,6 +113,8 @@ type ElectronIpcHandler =
interface ElectronIpcFindingContext {
readonly module_key: string | null;
/** Recovered factory runtime; absent in older or unbundled findings. */
readonly module_runtime?: string;
readonly location: JavaScriptSourceRange;
}
@@ -134,12 +140,16 @@ export interface ElectronSenderValidationFinding {
readonly expected: ElectronStaticValue;
readonly enforcement: "unknown";
readonly module_key: string | null;
/** Recovered factory runtime; absent in older or unbundled findings. */
readonly module_runtime?: string;
readonly location: JavaScriptSourceRange;
}
interface ElectronUtilityProcessFindingState {
readonly service_name: string | null;
readonly module_key: string | null;
/** Recovered factory runtime; absent in older or unbundled findings. */
readonly module_runtime?: string;
readonly location: JavaScriptSourceRange;
}
@@ -163,8 +173,14 @@ export interface ElectronNativeAddonBindingFinding {
readonly specifier: string;
readonly binding_kind: "import" | "require" | "re-export";
readonly module_kind: "import" | "require";
/** Exact requested addon member keys; namespace and dynamic access are separate. */
readonly members: readonly string[];
/** Omitted in legacy findings; omission means unknown, not false. */
readonly namespace_access?: boolean;
readonly dynamic_member_access?: boolean;
readonly module_key: string | null;
/** Recovered factory runtime; absent in older or unbundled findings. */
readonly module_runtime?: string;
readonly location: JavaScriptSourceRange;
}

Some files were not shown because too many files have changed in this diff Show More