leefer/rea
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into feat/claude-code-plugin

Browse Source
This commit is contained in:
GamerDubz
2026-10-12 05:18:13 +02:00
parent 15a5e08338 ee50ed121e
commit d206a19c7d
389 changed files with 22916 additions and 2950 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/reverse-engineer-anything/SKILL.md
+8 -2
View File
@@ -115,8 +115,14 @@ ask when matches are ambiguous. Never choose an example app on the user's behalf
In a target-free session, use `open_binary` to bind any archive/package or
native target whose analysis tool operates on the active target. Do not call an
unadvertised tool; inspect `binary_session` with `{}` and its
`result.tool_availability` for availability reasons and remediation. The tool
list stays complete; availability depends on the operation, target, and host.
`result.tool_availability` for availability reasons and remediation. Expand each
group's `names` into entries, inheriting the inventory-level client requirements
unless the group overrides them. In its closed effect and annotation vocabularies,
listed flags are true and omitted known flags are false. The grouped name union
is the complete tool inventory; use `tools/list` for canonical tool schemas.
Provider candidate facts describe possible providers, while top-level session
capabilities describe the current route and `analysis_provider_binding` names the
selected provider. Do not treat those scopes as interchangeable.
## Work summary-first
.agents/skills/reverse-engineer-anything/references/connection-and-recovery.md
+9 -9
View File
@@ -11,14 +11,14 @@ main and an older release expose the same arguments.
## Choose the remedy from the observed failure
| Observation | Meaning and next step |
| --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| No REA tools in the active session | Inspect registration with `rea doctor --client codex --json`. A skill installation alone does not register MCP. |
| Missing, malformed or stale registration | Prepare a scoped setup plan; apply only authorized changes with backups. |
| Aligned registration, but no connected tools | Restart/reconnect the client. Doctor checks registration; it cannot establish the current agent's connection. Inspect the client's launch error if reconnection fails. |
| `MCP tool call requires approval, but approval policy is never` | Codex rejected the call before REA executed it. Do not install an engine or repeat setup to solve this client-policy decision. See the approval section below. |
| REA returns `provider_unavailable` or an unavailable tool entry | Read that operation's provider diagnostics or `binary_session.result.tool_availability`. Resolve only the prerequisite needed for the task; a complete tool catalog does not imply every operation is available. |
| `rea` is missing in a child shell, or Node is incompatible | Inspect the actual shell PATH and launcher/runtime. A prefix installation may be absent from a login shell's PATH; this does not establish a broken MCP registration. |
| Observation | Meaning and next step |
| --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| No REA tools in the active session | Inspect registration with `rea doctor --client codex --json`. A skill installation alone does not register MCP. |
| Missing, malformed or stale registration | Prepare a scoped setup plan; apply only authorized changes with backups. |
| Aligned registration, but no connected tools | Restart/reconnect the client. Doctor checks registration; it cannot establish the current agent's connection. Inspect the client's launch error if reconnection fails. |
| `MCP tool call requires approval, but approval policy is never` | Codex rejected the call before REA executed it. Do not install an engine or repeat setup to solve this client-policy decision. See the approval section below. |
| REA returns `provider_unavailable` or an unavailable tool entry | Read that operation's provider diagnostics or `binary_session.result.tool_availability`. Expand each group's tool names and inherit the shared client requirements unless overridden. Use the operation's availability, reason, and remediation; a complete tool catalog does not imply every operation is available. |
| `rea` is missing in a child shell, or Node is incompatible | Inspect the actual shell PATH and launcher/runtime. A prefix installation may be absent from a login shell's PATH; this does not establish a broken MCP registration. |
If the client cannot launch its shell/sandbox helper or read the installed skill,
inspect its launch diagnostic. For example, Codex can refuse helper aliases when
@@ -46,7 +46,7 @@ rea setup --client codex --dry-run --json
Use the current supported client's ID: `claude_code`, `claude_desktop`, `codex`,
`cursor`, `gemini_cli`, `windsurf`, `devin`, `opencode`, `antigravity`,
`copilot_cli`, `commandcode`, `qwen_code`, `vscode`, `grok_build`, `omp`, `pi`,
`hermes`, or `grok_bot`. If unknown, inspect `doctor --json` before choosing a
`hermes`, `dsh`, or `grok_bot`. If unknown, inspect `doctor --json` before choosing a
scope. `grok_bot` uses a chat registration step rather than local `mcp.json`.
For unsupported clients, use version-pinned manual stdio registration or CLI.
.agents/skills/reverse-engineer-anything/references/evidence-workflows.md
+6
View File
@@ -46,3 +46,9 @@ observed exit signal stays in `exit.signal`, `exit.code` is null for every
deadline exit, and the code of a finalizer that exited on its own is in
`exit.finalization.exit_code` (null on a signal exit or when no exit was
observed).
Use `inspect_analysis_view` with `view: {kind: "api-surface"}` on retained
`inspect_web_network_capture` HAR Evidence to group observed request lines by
method, origin and path template. Format matches are inferences, not handler
identity or schemas. Group pages cite parent record ordinals; consult record
coverage and exclusions before claiming completeness. No requests are replayed.
.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md
+13 -4
View File
@@ -50,10 +50,19 @@ runs and may change files or use the network. Hardened-runtime targets need the
`get-task-allow` entitlement.
Use `binary_session` with no arguments to check the open target, selected
provider, and alignment. Its default `result.tool_availability` includes the
complete tool inventory with availability reasons and remediation. When choosing
a tool, use its entry in that result; `tools/list` retains the complete catalog
when the target or provider state changes.
provider, and alignment. Its default `result.tool_availability` carries a
complete inventory grouped by shared facts. Expand each group's `names` and
inherit the inventory-level client requirements unless a group overrides them;
availability, reasons, remediation, modes, and flags remain inline. The
`effects` and `annotations` arrays name true flags from fixed vocabularies, so an
omitted known flag is false. `tools/list` remains the canonical source for full
tool schemas when target or provider state changes.
Provider capability groups carry provider identity and ordered limitations.
Prepend each set's `shared_limitations` to a group's `limitations` to recover the
operation's full ordered list. Candidate capabilities describe possible
providers. Top-level session capabilities describe the current route, and
`analysis_provider_binding` identifies the selected provider.
## DOS MZ with Ghidra
AGENTS.md
+3 -1
View File
@@ -14,7 +14,9 @@ Preserve caller-selected inputs and local evidence, including output, URLs, path
Dependencies flow inward: domain semantics and contracts, provider adapters, shared application workflows, then CLI/MCP adapters. Keep engine protocols and provider-specific code out of domain/application layers. See [docs/architecture.mermaid](docs/architecture.mermaid) when changing composition.
Give each interpretation and resource a clear owner. Preserve identity, omission, uncertainty, and source evidence across boundaries. Retain ownership of resources whose cleanup failed. Reuse `src/process/` supervision and identity primitives; a PID and executable path alone do not establish ownership after exit or reuse.
Give each interpretation and resource a clear owner. Consumers must use that owner's parsed facts rather than reinterpret raw source. Preserve identity, omission, uncertainty, and source evidence across boundaries. Retain ownership of resources whose cleanup failed. Reuse `src/process/` supervision and identity primitives; a PID and executable path alone do not establish ownership after exit or reuse.
Keep proof of ownership separate from validity, editability, and readiness. Malformed owned data may still be safely removable. Unresolved identity or applicability must remain unknown rather than be treated as absent, foreign, or aligned.
Assess brittleness in touched code during implementation, debugging, review, and maintenance. Fix the producing representation or ownership rule before adding downstream repairs. Normalize once at the owning boundary using format-aware parsers. Remove superseded representations, aliases, wrappers, and redundant tests after verifying affected consumers. Introduce abstractions only when they remove observed complexity.
bridge/ghidra/ReaGhidraBridge.java
+13
View File
@@ -908,6 +908,19 @@ public final class ReaGhidraBridge extends HeadlessScript {
result.add("procedure", function == null ? JsonNull.INSTANCE : GSON.toJsonTree(canonicalAddress(function.getEntryPoint())));
result.addProperty("architecture", currentProgram.getLanguageID().getIdAsString());
result.addProperty("mode", currentProgram.getLanguage().getLanguageDescription().getVariant());
if (currentProgram.getLanguageID().getIdAsString().startsWith("MIPS:")) {
JsonArray decoderContext = new JsonArray();
var modeRegister = currentProgram.getRegister("ISA_MODE");
if (modeRegister != null) {
var modeValue = currentProgram.getProgramContext().getValue(modeRegister, address, false);
JsonObject context = new JsonObject();
context.addProperty("register", modeRegister.getName());
context.add("value", modeValue == null ? JsonNull.INSTANCE : GSON.toJsonTree(modeValue.toString()));
context.addProperty("source", "ghidra-program-context");
decoderContext.add(context);
}
result.add("decoder_context", decoderContext);
}
JsonArray operands = new JsonArray();
JsonArray references = new JsonArray();
JsonArray destinations = new JsonArray();
bridge/pwntools/layout_impl.py
+5
View File
@@ -206,6 +206,11 @@ def inspect_elf(path, cache):
"visibility": raw.st_other.visibility, "section_index": raw.st_shndx,
})
if h.sh_type in ("SHT_REL", "SHT_RELA"):
if h.sh_size % h.sh_entsize != 0:
raise LayoutFailure(
"format",
f"ELF relocation section {index} size is not a multiple of its {h.sh_entsize}-byte entries.",
)
if image.header.e_type == "ET_REL" and h.sh_info >= len(all_sections):
raise LayoutFailure("format", f"ELF relocation section {index} references missing target section {h.sh_info}.")
symbol_table = None
docs/adb-device-analysis.md
+6 -6
View File
@@ -109,12 +109,12 @@ existing analysis families directly, for example
- **Device paths are not host paths.** `base_apk_device_path` values live on
the device filesystem. Only pulls and captures produce host-local paths,
each paired with byte size and SHA-256 digest. A pulled file's host name is
the device path's last segment, accepted only when every host can store it:
a `pull_adb_file` whose device path ends in a separator, `.`/`..`, a
control or reserved character, a Windows reserved device name, a trailing
dot or space, or a `:` alternate-data-stream marker is refused as
`invalid_request` before any `adb pull` runs, and REA never renames a
device-reported file.
the device path's final POSIX segment. REA preserves the original device
path and filename; a missing final segment or `.`/`..` is refused before
output creation. Windows hosts follow [Windows filename rules](https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file),
while POSIX hosts retain valid POSIX filenames such as names with spaces or
colons. A package pull checks reported names against the destination
filesystem before pulling, and refuses names that would share a local path.
- **Splits are part of the answer.** `pm path` enumerates the base APK and
every installed split; pulls cover the whole set. System packages whose
APKs keep other names report the `unknown` role rather than guessing.
docs/android-analysis.md
+5 -1
View File
@@ -107,7 +107,11 @@ the same source byte limit and explicit truncation reporting.
upstream engine and should be checked against the retained XML when needed.
Signature verification is explicitly `not_performed`.
- Class search consumes all pages and checks their counts. An empty `query`
inventories all class names; class names are decompiler representations.
inventories all class-name rows; class names are decompiler representations.
Repeated names remain repeated in `matches` so the inventory row count and
producer evidence stay visible. `duplicate_names` identifies matching names
with multiple rows and their occurrence counts; a name alone does not
distinguish which provider class node a later lookup selects.
- Class inventories read parsed metadata without generating source and return
full reported display types, fields and inner class names. Synthetic members
that code generation hides can appear in the inventory; it stays consistent
docs/ghidra-mips.md
+65 -5
View File
@@ -16,9 +16,10 @@ PE, managed and Mach-O architecture sets are unchanged.
The Ghidra lane admits ELF32 `ET_EXEC` and `ET_REL`, little or big endian, with
explicit o32 and `EF_MIPS_ARCH_32R2` flags and no machine-specific or ASE
encoding flags. These are header declarations, not proof that every instruction
conforms to that ISA. ELF64, n32/n64, unspecified ABI, MIPS16/microMIPS,
machine-specific variants, shared objects and PSP PRX are not covered by this
lane. Their family may be identified, but provider selection returns the
conforms to that ISA. ELF64, n32/n64, unspecified ABI, microMIPS,
machine-specific variants and PSP PRX are not covered by this standard-instruction
lane. MIPS16 shared images have the separate bounded profile below; other shared
or compressed images remain unsupported. Their family may be identified, but provider selection returns the
specific unsupported constraint rather than guessing a compatible profile.
Explicit `EF_MIPS_FP64` and `EF_MIPS_NAN2008` declarations are refused. Their
@@ -49,6 +50,63 @@ values are retained for diagnosis but do not gain provider support. These checks
bind the declared interpretation; the fixtures do not establish full
floating-point behavioral conformance.
### MIPS16 shared images
A separate profile admits ELF32 `ET_DYN`, either byte order, declaring explicit
o32, MIPS32r2 and exactly the MIPS16 ASE. Machine variants, n32, FP64 and NAN2008
remain refused. Source inspection must examine the complete dynamic relocation
declarations through uniquely mapped file-backed `PT_LOAD` ranges. This profile
verifies `R_MIPS_NONE` and symbol-index-zero `R_MIPS_REL32`; section declarations
must describe that same dynamic table. Unavailable section headers and additional
section relocation tables (including `--emit-relocs` output) remain unsupported.
Symbol-bearing REL32,
other relocation types, RELA and PLT declarations remain unsupported. A real
negative control shows that Ghidra can normalize a symbol-bearing MIPS16 function
pointer to an even address, losing the source mode tag. REA refuses that shape
rather than rewriting the source or silently repairing the analysis result.
The profile pins the endian-matched `MIPS:<LE|BE>:32:default` processor and default
compiler specification, and checks the bridge-reported language after import.
The generic processor preserves mixed MIPS32/MIPS16 code through per-address
`ISA_MODE` context; a global `16e` processor would impose one mode across the
image. A caller-selected incompatible processor or compiler specification is
explicitly refused without replacing the requested choice.
An observed ABI flags record must agree with the header and MIPS16 ASE, and
must declare the same verified floating-point combinations as the standard
lane. With no observed record, every ABI declaration table must have been
completely examined. The header still declares o32, but floating-point ABI
semantics remain unknown and are stated in Evidence; selecting the default
compiler specification does not establish floating-point storage. Partial
section tables do not prove record absence. The existing soft-float prototype
policy applies when the source explicitly declares soft-float.
Returned addresses are analysis coordinates, not runtime load addresses.
`inspect_native_load_image` preserves original relocation bytes, current loaded
memory bytes and provider status. External GOT bindings remain unverified
without their defining images. The provider's `modified_sha256` describes its
ephemeral relocated image and can differ from `original_sha256`; it does not
mean that the caller's artifact was written. `inspect_native_instruction` retains the program
language variant in `mode` and reports actual per-address `ISA_MODE` separately
in `decoder_context`, including null when unavailable. Artifact bytes and digests
remain unchanged; this workflow does not execute the shared library.
The optional real-provider check builds eight source-owned GNU controls, covering
both byte orders, mixed MIPS32/MIPS16 code, declared and absent ABI records, and
the rejected symbol-bearing and additional section relocations. It compares source instruction bytes,
function symbols and the local function-pointer tag with actual CLI/MCP results,
checks context through a snapshot round trip, and verifies process cleanup.
It requires caller-supplied GNU MIPS cross-binutils and Ghidra; no fixture target
is executed or patched after linking. Floating-point behavior of compressed
functions is not established by these integer controls.
```bash
REA_MIPS_AS=/absolute/path/to/mips-linux-gnu-as \
REA_MIPS_LD=/absolute/path/to/mips-linux-gnu-ld \
GHIDRA_INSTALL_DIR=/absolute/path/to/ghidra \
npm run verify:ghidra:mips-shared
```
MIPS header and ABI record facts are committed in the analysis profile so
interpretation changes invalidate profile-bound snapshots. The support-lane
identity is versioned independently of file identity. Evidence, lifecycle
@@ -186,8 +244,10 @@ from the original file bytes. HI16/LO16 are checked as the effective address
each LO16 forms with each HI16 it resolves, including several HI16s before one
LO16, so a missing signed-low carry fails and a HI16 without a following LO16
fails. The lane also requires matching Ghidra relocation entries and statuses,
section file offsets and sizes, coverage of a shared HI16, a LO16 carry,
external code and data and `R_MIPS_32` words, direct calls from the entry
section file offsets and sizes, coverage of a shared HI16, a LO16 resolving
several pending HI16s (from an assembly block, since Clang does not emit that
from C), a LO16 carry, external code and data and `R_MIPS_32` words, direct
calls from the entry
function, the relocatable support lane in the snapshot profile, the
relocatable limitation in Evidence and an unchanged source object. The fixture
also calls `powf`, an external whose float prototype comes from Ghidra's C
docs/installation.md
+61 -17
View File
@@ -89,8 +89,11 @@ running server or its registration.
The npm release checked on **2026-10-11** was **6.4.0**, published from
[`a56f7bb6`](https://github.com/morluto/rea/commit/a56f7bb668ab23fec8d33f7af353ac7ed2fe1895).
The actual registry package accepted dry-run setup plans for all 17 client IDs
listed below, including Qwen Code, Pi and Hermes. OpenCode's plan selected
The actual registry package accepted dry-run setup plans for the 17 client IDs
in that release, including Qwen Code, Pi and Hermes. DeepSeek Harness (`dsh`)
is on main and is not in npm 6.4.0; `rea setup --client dsh` against `@latest`
reports an unsupported client until a release containing it is published.
OpenCode's plan selected
`OPENCODE_CONFIG_DIR` even when that directory did not yet exist. These checks
used an isolated home on macOS with Node 24; they verify planning and argument
admission, without writing client configuration or establishing live integration
@@ -162,6 +165,10 @@ listed after the table because its connector is not one of these files:
| OMP | `omp` |
| Pi | `pi` |
| Hermes | `hermes` |
| DeepSeek Harness | `dsh` |
`dsh` is main-only until a release after 6.4.0. The registry check above does
not cover it.
For GitHub Copilot CLI 1.0.95, `gpt-4.1` model metadata can block ordinary
chat with REA's full input-schema profile before sending a model request
@@ -259,11 +266,15 @@ configuration and skill destinations; doctor and uninstall use those same paths.
### DeepSeek Harness
DeepSeek Harness uses a Cordis YAML patch sequence rather than a `mcpServers`
object. REA's `setup --client` does not edit this format. For Harness
`0.2.0-rc.2`, append the following entry to `$DSH_HOME/cordis.patch.yml`
(`~/.dsh/cordis.patch.yml` when `DSH_HOME` is unset). Preserve existing patches
and replace the executable paths with absolute paths from your installation:
DeepSeek Harness registers MCP clients through a Cordis YAML patch sequence
rather than a `mcpServers` object. `rea setup --client dsh` appends (or
replaces) the REA entry in the home patch `$DSH_HOME/cordis.patch.yml`
(`~/.dsh/cordis.patch.yml` when `DSH_HOME` is unset), preserving unrelated
patches and comments. `rea doctor --client dsh` checks that same entry id.
`rea uninstall` removes it by scanning supported clients; uninstall takes no
`--client` flag. The home patch applies to every Harness profile, so the
registration is written once and no per-profile edit is needed. The written
entry is:
```yaml
- insert:
@@ -274,17 +285,20 @@ and replace the executable paths with absolute paths from your installation:
transport: stdio
command: /absolute/path/to/node
args: ["/absolute/path/to/rea/scripts/rea.mjs", mcp]
cwd: !!js process.cwd()
```
Setup preserves caller-owned Cordis expressions and replaces launcher fields
with literal values that doctor can verify. It refuses to edit an expression
that produces the entire REA config, or a group-targeted REA insertion. Use a
literal config and a root-level insertion as shown above.
For a global npm installation, the launcher is under
`$(npm root -g)/rea-agents/scripts/rea.mjs`. Check the Node executable with
`command -v node`. Register `serverName: rea` only once in each active profile,
then restart Harness. It exposes tools as `mcp__rea__<tool_name>`.
Run `rea setup --skill --dry-run`, review the destination, then
`rea setup --skill --yes` to install the workflow in `~/.agents/skills`.
Harness discovers that directory, or `$DSH_AGENTS_HOME/skills` when configured;
its native `skill` tool can load `reverse-engineer-anything`.
`$(npm root -g)/rea-agents/scripts/rea.mjs`. Restart Harness after setup; it
exposes tools as `mcp__rea__<tool_name>`. Run `rea setup --skill --dry-run`,
review the destination, then `rea setup --skill --yes` to install the workflow
in `~/.agents/skills` (`$DSH_AGENTS_HOME/skills` when configured); Harness
discovers that directory, and its native `skill` tool can load
`reverse-engineer-anything`.
REA's optional [native client verification](testing.md#end-to-end-integration-and-golden-evidence)
checks headless chat, skill loading, all tool input schemas and actual analysis
@@ -779,6 +793,32 @@ is no fixed per-operation or response-size ceiling. Unresolved computed calls
remain unknown, reference-kind provenance is preserved, and provider-specific
pseudocode is never treated as original source or Hopper-equivalent text.
### Missing macOS native decompiler
If `rea doctor --json` reports `native_decompiler` as missing on macOS x64,
the extracted Ghidra archive may include the decompiler source without the
macOS executable. For Ghidra 12.1.4, you can build it manually with an existing
Xcode Command Line Tools installation, `make`, and `bison`:
```bash
cd "$GHIDRA_INSTALL_DIR/Ghidra/Features/Decompiler/src/decompile/cpp"
make ghidra_opt
decompiler_dir="$GHIDRA_INSTALL_DIR/Ghidra/Features/Decompiler/os/mac_x86_64"
mkdir -p "$decompiler_dir"
test ! -e "$decompiler_dir/decompile" && test ! -L "$decompiler_dir/decompile" &&
install -m 755 ghidra_opt "$decompiler_dir/decompile"
rea doctor --json
```
The copy step requires the destination to be absent and preserves an existing
decompiler. This recipe is specific to macOS x64: the
[Ghidra 12.1.4 Makefile](https://github.com/NationalSecurityAgency/ghidra/blob/Ghidra_12.1.4_build/Ghidra/Features/Decompiler/src/decompile/cpp/Makefile)
explicitly selects `-arch x86_64` on Darwin. For other architectures or a Gradle
build, follow the matching release's
[native component instructions](https://github.com/NationalSecurityAgency/ghidra/blob/Ghidra_12.1.4_build/GhidraDocs/GettingStarted.md#building-native-components).
REA accepts the matching executable under either `os/<platform>/` or
`build/os/<platform>/`; it does not build the component during setup or diagnosis.
### Ghidra import language and analysis seeds
Ghidra detects the language from the executable header. To set it (for example, Borland C++ PE images):
@@ -882,8 +922,12 @@ verification workflows.
Unexpected request failures retain their original internal cause. CLI and MCP
errors expose Error names, messages, and available codes under
`details.diagnostics.failure_cause`; other rejection values retain their
primitive value or an explicit type. Shutdown warnings include the failure
`details.diagnostics.failure_cause`, including bounded nested causes and explicit
truncation markers. Darwin ownership-helper compilation failures also retain
the compiler outcome, exit code or signal, and bounded stderr; a timed-out
compiler remains distinguishable from a compiler that exited with an error.
Other rejection values retain their primitive value or an explicit type.
Shutdown warnings include the failure
kind, message, and diagnostics, while process and temporary-project cleanup
continues. These messages redact known bridge authentication tokens and
preserve local paths and other analysis context.
docs/javascript-artifact-reconstruction.md
+41 -2
View File
@@ -174,6 +174,24 @@ default). Reaching it fails the current source with the same memory resource
failure and reports `worker_process_memory_limit_bytes`, also a coverage limit;
the next source starts a new analyzer. Bun elsewhere enforces no analyzer limit.
The default semantic capacities are 100000 nodes across the application and
20000 per source. Large bundles may explicitly select `max_semantic_nodes` and
`max_semantic_nodes_per_file` (CLI: `--max-semantic-nodes` and
`--max-semantic-nodes-per-file`), each from 1 to 1000000. The per-file offer is
also bounded by the remaining application capacity. Raising these controls does
not bypass worker heap, execution deadline or parent result admission. Truncated
coverage reports both effective node ceilings; omitted relationships remain
unknown.
While sources remain, the sequential worker reserves one next compact static
checkpoint at its 8 MiB bound with the same 6x retention expansion used at
admission. Earlier checkpoints are already charged to retained bytes; the
remaining inventory does not multiply the reserve. Streamed file transfers keep the default 512 MiB
allowance, and can grow with the parent's offered retention budget (8x byte
expansion), up to 2 GiB. Individual records remain bounded to 8 MiB. The parent
still independently checks actual bytes, record overhead, digests and source
identities before adopting results.
A source that exhausts the analyzer heap does not terminate the CLI/MCP
process. Completed static facts are retained, other files can still be analyzed,
and unexamined semantic relationships remain explicit unknowns. The parent
@@ -366,9 +384,30 @@ Source-map contents are parsed as local input. The graph stores original source
Only explicitly present BrowserWindow values are observations. REA does not
substitute version-dependent Electron defaults for omitted `webPreferences`.
Dynamic option objects, bridge keys, API objects, and IPC channel expressions
Unresolved option objects, bridge keys, API objects, and IPC channel expressions
remain unknown and make coverage partial.
Recognized literal module-table bootstraps also support bounded lexical
substitution through module exports, import helper bodies, local wrapper calls,
returned closures, object spreads, and callback parameters. Recovered Electron
paths retain source ranges, module identities, literal channels and branch
conditions as static candidates. Unknown spreads, reassigned bindings, accessors,
dynamic channels and unsupported syntax do not become exact implementations.
This pass neither executes the bundle nor proves that runtime guards pass.
An ambiguous or unresolved handler retains its channel registration but does
not acquire a path to one arbitrarily selected callback. Unknown factory
conditions retain both return branches, including fallthrough. Short-circuit
selection distinguishes truthiness from nullishness; uncertain `??` paths
retain a nullish guard rather than a falsy guard.
These paths appear as contextual `javascript-expression` nodes. An exact `api`
seed such as `desktop.copy` can trace an exposed member through a unique IPC
pairing to an external API call. `addListener` is normalized to `on`. Semantic
projection retains these boundary paths before bulk function definitions, within
the existing per-file node budget; omitted general semantics remain explicitly
partial. Wrapper expansion is bounded by one million visits/substitutions and
128 nested substitutions, with exhaustion reported as unknown coverage.
Electron API identity follows proven lexical module aliases. Namespace interop
also preserves identity when the complete Rolldown helper operations, lexical
roles, live getters, and captured intrinsic origins match, or when a stable
@@ -415,7 +454,7 @@ do not establish that an application has no Electron boundary usage.
IPC pairing is an inference, not an observation. A renderer `invoke` pairs only
with one unique `ipcMain.handle`/`handleOnce` candidate on the same exact literal
channel; a renderer send pairs only with one unique `ipcMain.on`/`once`
channel; a renderer send pairs only with one unique `ipcMain.on`/`once`/`addListener`
candidate. Dynamic channels are never paired, and multiple compatible handlers
are reported as ambiguous without adding a caller-to-handler edge. A matching
channel still does not prove registration order, reachability, or runtime use.
docs/managed-code-analysis.md
+28
View File
@@ -58,6 +58,34 @@ reports `resource-limit` with unavailable or partial coverage; omitted facts do
not become empty observations. Ambiguous PE section mappings are rejected
rather than interpreted according to section order.
### Selected member inspection
When whole-assembly member decoding reaches the representation budget, select
known declaring types before their signatures and CIL bodies are decoded:
```bash
rea inspect-managed-members /absolute/path/Assembly.dll --type Example.Parser
```
MCP uses `inspect_managed_members` with
`{"path":"/absolute/path/Assembly.dll","type_names":["Example.Parser"]}`.
Repeat `--type` or supply multiple unique `type_names` for several types.
Names match the reported `full_name` exactly. Top-level names are namespace
qualified; nested types currently report bare names, which can select several
types with that name.
The result preserves the requested names in `selection.type_names` and reports
unmatched names separately. If metadata names are unreadable, limitations retain
that uncertainty. With no selection, `selection` is null and whole-assembly
inspection keeps its existing resource policy. Selection restricts types,
fields, methods, call edges and field accesses; member references remain an
assembly-wide inventory, so output size is not guaranteed to be small.
Coverage describes the selected members. Comparisons leave members outside the
opposite observation's selection unknown, and application graphs report their
selected scope as partial. Neither makes an absence claim about the whole
assembly. Older member Evidence without `selection` remains readable.
## Analysis objective
REA's managed-code track is intended to answer five different questions without
docs/mcp-contracts.md
+57 -11
View File
@@ -113,17 +113,48 @@ Schema does not express those cross-field invariants.
Call `binary_session` with `{}` and read `result.tool_availability` to choose a
callable operation for the current target, provider, host, and negotiated client
capabilities. The default result includes the complete inventory with each
tool's availability, reason, and remediation. Each entry also reports required
and optional negotiated client features plus the currently missing features.
The optional inputs `expected_package_version`, `expected_catalog_digest`, and
capabilities. The default result contains the complete inventory in groups that
share status facts. `tool_availability.client_requirements` is the baseline for
each group; a group's optional `client_requirements` replaces that baseline for
the group's tools. Expand each group's `names` into one tool entry, carrying its
surface, availability, reason, remediation, default-mode status, modes, and
client requirements. Compare the union of group names with `tools/list` when
checking that a client received the full catalog. An operation absent from all
groups is missing from the complete inventory; it is not an implicit
unavailable entry.
The `effects` and `annotations` arrays list the true members of their fixed
vocabularies. A known member omitted from either array is false. This applies
only to these closed flag sets; it does not make an omitted tool, provider, mode,
or limitation false. Keep availability reasons, remediation, client-feature
requirements, and mode facts inline with the grouped tool names. The optional
inputs `expected_package_version`, `expected_catalog_digest`, and
`expected_server_path` compare the live session with the caller's expectations.
`server_identity.catalog` contains catalog counts and schema digests. Grouped
tool availability carries the name set and current per-tool facts; `tools/list`
is the canonical source for complete tool definitions, including input and
output schemas. Clients that previously read `server_identity.catalog.tools`
should expand `tool_availability.groups` and use `tools/list` for schemas. The
same catalog identity summary is returned by CLI `capabilities` and `providers`;
MCP tool definitions remain available through `tools/list`.
Provider capabilities are grouped by provider identity and shared facts. In each
set, `shared_limitations` is the ordered common prefix; append a group's ordered
`limitations` suffix to recover each operation's exact limitations. Group
`effects` arrays use the same true-only closed-vocabulary rule as tool
availability. Expand every group operation while retaining its provider
identity, availability, reason, rejection code, effects, and ordered
limitations.
`binary_session.analysis_provider_candidates` is authoritative for deep-engine
discovery. Target-free discovery is sorted by provider ID, reports host
availability and `unknown` target support, and does not create an analysis
client. `open_binary.provider_id` accepts a concrete provider ID or `auto`; it
uses the same parser and selection policy as CLI `--provider` and
discovery. Each candidate's capability sets describe that possible provider;
they do not select it. The top-level `capabilities` describe the current route,
while `analysis_provider_binding` identifies the concrete selected route. These
fields are not interchangeable. Target-free discovery is sorted by provider ID,
reports host availability and `unknown` target support, and does not create an
analysis client. `open_binary.provider_id` accepts a concrete provider ID or
`auto`; it uses the same parser and selection policy as CLI `--provider` and
`REA_ANALYSIS_PROVIDER`. A successful deep open exposes one immutable provider,
concrete version, selection source, and complete analysis profile through
`analysis_provider_binding`. Ambiguity and unknown, unavailable, or unsupported
@@ -165,6 +196,11 @@ Provider calls receive the request cancellation signal. Artifact traversal,
hashing, version comparisons, Hopper requests, and process capture
check the same signal. Cancellation is distinct from timeout. A cleanup failure
uses `cleanup_incomplete` and lists only the owned resource kinds that remain.
An atomic Evidence bundle or snapshot write can publish its complete output
before staging cleanup fails. In that case, `details.published: true`,
`details.published_path`, and `details.bytes` retain the publication receipt.
Use that receipt to distinguish completed publication from a write that never
committed. Staging resources remain owned until cleanup is verified.
Native call tracing and process capture retain available observations in
`details.partial_observation` on failure, including when cleanup succeeds.
The observation reports its partial coverage; cleanup details describe host
@@ -310,9 +346,12 @@ directly to `compare_functions`, and `inspect_artifact` Evidence to
`compare_artifacts`. Use `get_evidence_bundle` when the task needs broader
retained session history or an explicit bundle for transfer.
`get_call_graph` keeps provider-limited relationship reads on their node as
`limitations`, alongside any observed `calls`. An empty call list with such a
limitation is incomplete coverage, not proof that the function has no calls.
`get_call_graph` returns a `graph` keyed by traversal depth and a top-level
`limitations` array with deduplicated execution caveats. Each node keeps its
relationship coverage and read-specific `limitations`, alongside observed
`calls`. When a provider does not classify its caveats, they remain on that
node and coverage is `unknown`. An empty call list with partial or unknown
coverage is not proof that the function has no calls.
`trace_call_path` reports each limited expansion in `residual_unknowns` and
sets `truncated: true`; any path already supported by observed edges remains
visible. Tool errors remain in their existing error or failure fields.
@@ -380,6 +419,13 @@ analysis, so later views and application workflows read it without repeating
analysis or moving the complete record through the response. A server without
session retention refuses summary detail with `capability_unavailable`.
`inspect_analysis_view` also accepts `api-surface` for retained HAR capture Evidence,
grouping request lines by method, origin and format-rule path template. Templated
segments are inferences. Because paging follows the full grouping, every page is
inferred Evidence whenever any group is templated, and derived Evidence only when
no group was templated.
See [derived API surfaces](web-network-captures.md#derived-api-surface).
`inspect_analysis_view` projects a caller-selected view of already completed
`inspect_binary_layout`, `analyze_javascript_application`, or `analyze_function`
Evidence. Native views select procedure, pseudocode, assembly, basic blocks,
docs/native-investigation.md
+26 -7
View File
@@ -6,11 +6,20 @@ coverage and unknowns. The CLI and MCP use the same application workflows.
## Static inspection
`inspect_macho` retains otool segment `file_offset` values relative to the
selected Mach-O slice. Its evidence file-offset ranges address the original
input file and include the observed lipo slice offset for universal binaries.
If that offset is unavailable, segment evidence locations are omitted with an
explicit limitation; architecture inventory locations remain available.
`inspect_macho.selected_architecture` identifies the exact selected lipo slice
and its `container_file_offset` in the input file. A thin Mach-O has a known
offset of zero; an unknown universal-slice offset stays null. The Evidence
subject format describes the input container: `mach-o-universal` for FAT files,
including those with one slice. The normalized result describes the selected
Mach-O image. Architecture
`file_offset` values are container-relative. Entrypoint, segment and section
`file_offset` values retain their source meaning and are relative to the
selected Mach-O slice. Raw load-command fields retain their original values.
Evidence locations map the LC_MAIN entrypoint and nonempty file-backed segment
ranges to container offsets by adding the observed slice offset. Zero-fill
segments do not produce file ranges. When the selected slice's container offset
is unknown, those mapped locations are omitted with an explicit limitation;
architecture inventory locations remain available.
Hopper's `address_to_file_offset` combines the native image-relative mapping with
original-file coordinates only after checking the source against the session's
@@ -53,6 +62,10 @@ complete observation.
classes are preserved. Select `root`, `offset` and `limit` explicitly.
UID 0 cannot distinguish conditional nil from ordinary nil. Limits are
64 MiB of input, 200,000 objects/references and 128 nesting levels.
XML dictionary entries decoded as `__proto__` are omitted with an explicit
count because REA results cannot represent that key. XML tag whitespace and
comments do not change that handling. Other values and the original archive
digest are preserved; key interpretation follows the pinned plist decoder.
- `decode_interface_builder` / `rea decode-interface-builder <app>` reads
compiled storyboard and nib resources into objects, outlets, actions,
connections and controller/class names. `verify:interface-builder` compiles
@@ -192,8 +205,14 @@ rea inspect-native-data-type <binary> --address <typed-data-address> --provider
Instruction facts include bytes, length, decoder text, ordered register/scalar/
address tokens, typed references and flow destinations. Mid-instruction, data,
outside-memory and undecodable addresses have separate outcomes. Effective
memory base/index/displacement roles and per-instruction context mode remain
unavailable; `mode` is the program language variant.
memory base/index/displacement roles remain unavailable. Ghidra reports its program
language variant in `mode`; other providers retain their own mode label. Ghidra
MIPS results additionally expose `decoder_context`
with the observed `ISA_MODE` register at the queried address and its provider
source. A null value is unavailable, not mode zero; an empty array reports no
available context register, and omission preserves older captures or providers
that do not expose this facet. The context is an analysis observation, not proof
of a runtime processor state.
`inspect_native_instruction` also returns `incoming_references` to the exact
queried address, including data, interior-instruction and undecodable addresses.
docs/testing.md
+20 -6
View File
@@ -112,8 +112,12 @@ Set `REA_VERIFY_HERMES_STICKY_PROFILE=1` to exercise native Hermes selection of
a named sticky profile with `HERMES_HOME` still pointing to its root.
`verify:gemini-client` requires an installed Gemini CLI (verified with
`@google/gemini-cli@0.63.0`); select it with `REA_VERIFY_GEMINI_COMMAND`.
The optional POSIX lane uses native `GEMINI_CLI_HOME` discovery in an isolated
`@google/gemini-cli@0.63.0` on POSIX and native Windows); select it with
`REA_VERIFY_GEMINI_COMMAND`. A selected `.js`, `.mjs` or `.cjs` entry point runs
under the verifier's Node. On Windows, select the npm install's
`node_modules/@google/gemini-cli/bundle/gemini.js`, because only a `gemini.cmd`
shim reaches `PATH`.
The optional lane uses native `GEMINI_CLI_HOME` discovery in an isolated
Git project, checks setup plans, backups and idempotence, activates the installed
personal skill, validates all forwarded input JSON Schemas against their declared
dialect, forwards the complete REA catalog and checks full JavaScript
@@ -121,7 +125,7 @@ Evidence for a Unicode path. Use `-- chat` for ordinary chat with REA enabled,
or `REA_VERIFY_RUNTIME_ROOT` for a production-only installed REA package.
The local fixture exercises the real Gemini API adapter, including its
`untrusted_context` tool-result envelope. Token counts are synthetic; live
Google API and native Windows compatibility remain unverified. The lane disables
Google API compatibility remains unverified. The lane disables
the client's memory-based relaunch to preserve the caller's Node heap budget.
`verify:opencode-client` requires an installed OpenCode (verified with
@@ -268,9 +272,13 @@ unverified; skill discovery uses an explicit isolated custom directory.
`verify:deepseek-client` requires an installed DeepSeek Harness (`dsh`;
verified with `@deepseek-ai/dsh@0.2.0-rc.2`) and Git. Set
`REA_VERIFY_DEEPSEEK_COMMAND` to select its executable. This optional POSIX lane
installs the skill through `rea setup --skill`, writes the native Cordis MCP
patch into an isolated `DSH_HOME`, and binds `HOME` and `USERPROFILE` to the
disposable account. The native default `~/.agents/skills` search path discovers
seeds caller model settings and a stale Cordis registration in an isolated
`DSH_HOME`, then runs public `setup --client dsh` dry-run, setup, repeated setup,
and scoped doctor. It checks the backup and unrelated patches before booting
Harness with the generated registration. After native execution it uninstalls
through the public CLI and checks that only caller patches remain. `HOME` and
`USERPROFILE` point to the disposable account. The native default
`~/.agents/skills` search path discovers
the installed skill without a `DSH_AGENTS_HOME` override. A separate Git root prevents inherited project skills from
masking a missing installation. The default `call` mode checks skill loading,
complete catalog discovery, the actual forwarded input schemas and regexes,
@@ -281,6 +289,12 @@ context capacity declared by its default DeepSeek Flash model; it does not
establish live DeepSeek API or native Windows coverage. Receipts include
request artifacts, result digests and owned-process lineage.
The DSH acceptance journey also exercises public setup, approval, doctor,
generated-launcher MCP startup and uninstall without an installed Harness.
That journey proves REA workflow behavior, not native Cordis expression
evaluation. Filesystem regressions retain tagged values and unrelated rows
across alias dependencies and check refusal of unsafe YAML without mutation.
Full E2E tests invoke the production command dispatcher and real providers,
without fake launchers, runners or responses. `verify:keyed-archive` writes
Foundation binary, XML, and integer archives using one run-owned Swift module
docs/tool-design.md
+5
View File
@@ -55,6 +55,11 @@ satisfy. A plan without execution does not establish runtime behavior.
- Keep observed, derived, and inferred relationships distinct. Preserve
unresolved edges and partial provider facts instead of requiring every
provider to supply the same metadata.
- For caller/callee reads, `relationship_coverage` describes only the
provider-observed direct relationship set. `complete` means that set was
returned without a known read limit; it does not claim every possible runtime
edge. Provider and execution caveats stay in `limitations` and do not imply
incomplete relationship coverage. Missing coverage remains `unknown`.
- Declare process, filesystem, network, UI, and mutation effects truthfully;
enforce the requested target and lifecycle.
- Add limits only for real format, protocol, authority, or measured resource
docs/wasm-artifacts.md
+9
View File
@@ -4,6 +4,15 @@
local WASM bytes using caller-supplied WABT 1.0.42. No active binary target is
required. REA does not install tools or fetch artifacts and does not execute WASM.
`open_binary` is not the WASM inspection route. A recognized WASM v1 header
returns `target_unavailable` with category `unsupported_target` and constraint
`wasm_requires_static_inspection`, including the selected path in a valid
`inspect_wasm_artifact` call. Use that target-free tool or
`rea inspect-wasm-artifact <path> --format json` with the toolchain below.
Check `binary_session` for configured capability availability. A recognized
header is not validation of the complete module; missing WABT is reported
separately as an unavailable capability, not as missing native engine support.
Set `REA_WABT_BIN_DIRECTORY` to an absolute directory containing executable
`wasm-validate`, `wasm-objdump` and `wasm2wat` (with `.exe` on Windows). Missing
executables, unsupported banners and inaccessible tools produce actionable
docs/web-network-captures.md
+73
View File
@@ -100,6 +100,11 @@ resources follow the same explicit declarations without changing error types;
sensitive issue pointers identify a real ancestor instead of a fabricated
coordinate. Header/cookie authentication values and known
transport URL userinfo are excluded structurally, including native backups.
HAR redaction and API-surface projection share the same recorded-URL parser.
Canonical userinfo spans are removed without rewriting the remaining source
text. Credential-bearing spellings without a safe raw span, such as slashless
or extra-slash HTTP URLs, are excluded as a whole transport field. Ordinary
extension text and noncredential path/query values remain unchanged.
Generated free-text limitations follow the same literal declarations, including
global, per-record and Evidence explanations added during projection. Contract
@@ -112,6 +117,74 @@ wire fragments. Native dictionary duplicate keys have the unchanged upstream
decoder's last-value semantics; this adapter does not claim to recover discarded
duplicates or the original numeric spelling of parsed native floats.
## Derived API surface
Pass retained HAR Evidence to `inspect_analysis_view` with
`view: {"kind":"api-surface","offset":0,"limit":50}`. Omitted offset and limit
are 0 and 50. The view groups request lines by method, WHATWG URL origin and
path template, sorted by origin, template and method. Query strings, fragments,
credentials, headers and bodies are not copied into groups.
| Rule | Exact segment format | Placeholder |
| ---------- | ---------------------------------------------------- | ------------ |
| `numeric` | ASCII digits only | `{numeric}` |
| `uuid` | RFC 9562 textual UUID, any version, case-insensitive | `{uuid}` |
| `hex` | At least 32 hexadecimal digits | `{hex}` |
| `iso-date` | Valid calendar date `YYYY-MM-DD` | `{iso-date}` |
Rules apply in table order to the recorded, still-encoded request path, read
before WHATWG dot-segment removal, so `/a/%2e%2e/admin` and `/admin` stay distinct. Segment indexes start
at zero after the leading slash. Each matched segment lists its observed literals
and `basis: "inferred"`. `templated_groups` counts templated groups across every
page. Because paging follows the full grouping, every page is inferred Evidence
when that count is nonzero, and derived Evidence only when no group was templated. Version prefixes, short opaque IDs
and digit-bearing route names stay literal. Samples merge only when method,
origin, and exact segment rules match. The view does not infer further
cross-sample patterns.
Groups cite parent record ordinals and distinct statuses and MIME types (lowercase
type/subtype without parameters). A malformed media type is unknown, not a content
type. Flags report single samples, mixed statuses or mixed content types.
`coverage` pages groups; `records` separately reports examined, total and excluded
parent records. Ordinal subsets carry a limitation. Null URLs are excluded as
`url-redacted`; unparseable URLs, URLs without an origin, spellings without
`scheme://authority` (such as `https:host/path`), and authorities containing a
tab or newline as `url-unparseable`.
Unavailable request objects or methods are explicitly excluded. Unavailable
response fields, including HAR's `status: 0` and an empty `mimeType`, remain
unknown, not invented statuses or MIME types.
The view makes no handler-identity, schema or causality claim. It performs no
network activity. Native mitmproxy captures return a typed unsupported error.
Parents inspected with `sensitive_values`, or whose Evidence does not record
`sensitive_value_count`, return a typed unsupported error whose reason is the
canonical token `sensitive_values_unretained`. The view would write path
templates and limitation text that it cannot check against declarations the
parent does not retain, so the refusal does not repeat declaration text.
One accumulator owns group identity, observed-value/status/MIME sets, ordinals,
exclusions and deduplicated limitations. It scans path spans without splitting
literal paths into arrays. Before schema parsing, it reserves the capture clone
and validation scratch by walking the parent without cloning it. Inline requests
borrow the Evidence value through CLI/MCP envelope validation; the application
admits the entire portable envelope before authenticated parsing or hashing.
The envelope clone, hash scratch and capture clone stay charged during grouping.
Temporary URL/path/MIME work and retained state must
fit together within a 96 MiB projection representation budget. A whole group or
merge is admitted before allocating its retained sets, map entries or array
values. Fixed group storage, distinct text, exclusions and limitations count as
well as matched segments. Per-record reservations are released after each
record; repeated values do not accumulate another retained charge. Only the
requested page is materialized as output groups.
The representation budget uses conservative storage and worst-case JSON-escaping
allowances; it is not an enforced V8 heap or process-memory limit. It is separate
from the decoder's actual reply-byte limit and MCP transport admission. Exceeding
it returns `resource_constraint` without a partial view. Limitations on the
parent envelope, capture and examined records are deduplicated onto the view.
Portable captures must retain unique ordinals within their declared total;
caller-selected subsets can preserve any requested ordinal order.
## Resource and lifecycle boundaries
Complete evidence has a 32 MiB input, 96 MiB decoder reply and 64-level nesting
package-lock.json
Generated
+1 -1
View File
@@ -6707,7 +6707,7 @@
},
"node_modules/rolldown": {
"version": "1.2.12",
"resolved": "https://mirrors.cloud.tencent.com/npm/rolldown/-/rolldown-1.2.12.tgz",
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.2.12.tgz",
"integrity": "sha512-8wafseiaG80xmXSfqidUNqZcylTlhmPZZt+za2m+js2sFZ8dTNlhIOV2WcbIPx2hgwPBJpEUGFAMZ9bgBBLTSQ==",
"dev": true,
"license": "MIT",
package.json
+3 -2
View File
@@ -165,6 +165,7 @@
"verify:ghidra": "npm run build:cached && node scripts/verify-real-ghidra.mjs",
"verify:ghidra:seeds": "npm run build:cached && node scripts/verify/ghidra/seeds.mjs",
"verify:ghidra:mips": "npm run build:cached && node scripts/verify-real-ghidra-mips.mjs",
"verify:ghidra:mips-shared": "npm run build:cached && node scripts/verify-real-ghidra-mips-shared.mjs",
"verify:ghidra:mips-relocatable": "npm run build:cached && node scripts/verify-real-ghidra-mips-relocatable.mjs",
"verify:ghidra:swift": "npm run build:cached && node scripts/verify-real-ghidra-swift.mjs",
"verify:ida": "npm run build:cached && node scripts/verify-real-ida.mjs",
@@ -255,8 +256,8 @@
"verify:evm:interface": "npm run build:cached && node scripts/verify-evm-interface.mjs",
"verify:recorded:crash": "npm run build:cached && node scripts/verify-recorded-crash.mjs",
"mcp-catalog:check:built": "node scripts/generate-mcp-tool-catalog.mjs --check",
"compile": "node scripts/clean-build-output.mjs && tsc -p tsconfig.build.json && node scripts/bundle-mcp-runtime.mjs",
"compile:termux": "node scripts/clean-build-output.mjs && node --max-old-space-size=3072 node_modules/typescript-termux/lib/tsc.js -p tsconfig.build.json && node scripts/bundle-mcp-runtime.mjs",
"compile": "node scripts/clean-build-output.mjs && tsc -p tsconfig.build.json && node scripts/bundle-runtime.mjs",
"compile:termux": "node scripts/clean-build-output.mjs && node --max-old-space-size=3072 node_modules/typescript-termux/lib/tsc.js -p tsconfig.build.json && node scripts/bundle-runtime.mjs",
"artifacts:mcp-catalog": "node scripts/generate-mcp-tool-catalog.mjs",
"artifacts:skills": "node scripts/generate-skill-metadata.mjs",
"artifacts:product-catalog": "node scripts/generate-product-catalog.mjs",
scripts/bundle-mcp-runtime.d.mts
-1
View File
@@ -1 +0,0 @@
export const bundleMcpRuntime: (root: string) => Promise<void>;
scripts/bundle-mcp-runtime.mjs
-104
View File
@@ -1,104 +0,0 @@
import { parse } from "@babel/parser";
import { readFile } from "node:fs/promises";
import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
/**
* Reduce filesystem resolution during MCP startup. Keep dependencies external,
* dynamic imports lazy, and the original compiled files for CLI/worker entrypoints.
*/
export const bundleMcpRuntime = async (root) => {
const compiledRoot = join(root, "dist");
const { rolldown, RUNTIME_MODULE_ID } = await import("rolldown");
const { imports = {} } = JSON.parse(
await readFile(join(root, "package.json"), "utf8"),
);
const bundle = await rolldown({
input: { main: join(compiledRoot, "main.js") },
platform: "node",
treeshake: false,
external: (source) => !source.startsWith(".") && !isAbsolute(source),
plugins: [
{
name: "preserve-compiled-module-locations",
transform(code, id) {
if (id === RUNTIME_MODULE_ID) return null;
if (!code.includes("import.meta")) return null;
const sourcePath = relative(compiledRoot, id);
if (
isAbsolute(sourcePath) ||
sourcePath === ".." ||
sourcePath.startsWith(`..${sep}`)
)
throw new Error(`MCP bundle input is outside dist: ${id}`);
// All chunks are directly under dist/mcp. Preserve the original
// module URL, including URL escaping, for bridges and createRequire.
const moduleUrl = `new URL(${JSON.stringify(`../${sourcePath.split(sep).map(encodeURIComponent).join("/")}`)}, import.meta.url).href`;
const edits = [];
const visit = (node) => {
if (node === null || typeof node !== "object") return;
const isMeta = (value) =>
value?.type === "MetaProperty" &&
value.meta.name === "import" &&
value.property.name === "meta";
if (
node.type === "CallExpression" &&
node.callee.type === "MemberExpression" &&
!node.callee.computed &&
isMeta(node.callee.object) &&
node.callee.property.name === "resolve" &&
node.arguments.length === 1 &&
node.arguments[0].type === "StringLiteral" &&
Object.hasOwn(imports, node.arguments[0].value)
)
// Package imports resolve in the same package scope after bundling.
return;
if (
node.type === "MemberExpression" &&
!node.computed &&
isMeta(node.object) &&
node.property.name === "url"
) {
edits.push({ start: node.start, end: node.end });
return;
}
if (isMeta(node))
throw new Error(
`Unsupported import.meta use in MCP bundle: ${id}`,
);
for (const value of Object.values(node)) {
if (Array.isArray(value)) value.forEach(visit);
else visit(value);
}
};
visit(parse(code, { sourceType: "module" }).program);
for (const { start, end } of edits.sort((a, b) => b.start - a.start))
code = code.slice(0, start) + moduleUrl + code.slice(end);
return { code, map: null };
},
},
],
});
try {
await bundle.write({
dir: join(compiledRoot, "mcp"),
format: "esm",
entryFileNames: "[name].js",
chunkFileNames: "[name]-[hash].js",
codeSplitting: true,
strictExecutionOrder: true,
minify: false,
sourcemap: false,
});
} finally {
await bundle.close();
}
};
if (
process.argv[1] !== undefined &&
pathToFileURL(resolve(process.argv[1])).href === import.meta.url
)
await bundleMcpRuntime(
resolve(dirname(fileURLToPath(import.meta.url)), ".."),
);
scripts/bundle-runtime.d.mts
+1
View File
@@ -0,0 +1 @@
export const bundleRuntime: (root: string) => Promise<void>;
scripts/bundle-runtime.mjs
+140
View File
@@ -0,0 +1,140 @@
import { parse } from "@babel/parser";
import { readFile } from "node:fs/promises";
import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
/**
* Reduce filesystem resolution during CLI and MCP startup. Keep dependencies
* external, dynamic imports lazy, and original compiled worker entrypoints.
*/
export const bundleRuntime = async (root) => {
const compiledRoot = resolve(root, "dist");
// CLI, MCP, workers and updater fixtures share one release-identity owner.
const metadataModule = join(compiledRoot, "generatedPackageMetadata.js");
const isMetadataModule = (id) =>
isAbsolute(id) && resolve(id) === metadataModule;
const { rolldown, RUNTIME_MODULE_ID } = await import("rolldown");
const { imports = {} } = JSON.parse(
await readFile(join(root, "package.json"), "utf8"),
);
for (const [directory, entries] of [
["mcp", { main: "main.js" }],
[
"cli-runtime",
{
cli: "cli.js",
cliOutput: "cliOutput.js",
streamedJsonOutput: "cli/streamedJsonOutput.js",
mcpDoctor: "mcpDoctor.js",
},
],
]) {
const bundle = await rolldown({
input: Object.fromEntries(
Object.entries(entries).map(([name, path]) => [
name,
join(compiledRoot, path),
]),
),
platform: "node",
treeshake: false,
external: (source) =>
isMetadataModule(source) ||
(!source.startsWith(".") && !isAbsolute(source)),
plugins: [
{
name: "preserve-compiled-module-locations",
transform(code, id) {
if (id === RUNTIME_MODULE_ID) return null;
if (!code.includes("import.meta")) return null;
const sourcePath = relative(compiledRoot, id);
if (
isAbsolute(sourcePath) ||
sourcePath === ".." ||
sourcePath.startsWith(`..${sep}`)
)
throw new Error(`Runtime bundle input is outside dist: ${id}`);
// Each output directory is directly under dist. Preserve the original
// module URL, including URL escaping, for bridges and createRequire.
const moduleUrl = `new URL(${JSON.stringify(`../${sourcePath.split(sep).map(encodeURIComponent).join("/")}`)}, import.meta.url).href`;
const edits = [];
const visit = (node) => {
if (node === null || typeof node !== "object") return;
const isMeta = (value) =>
value?.type === "MetaProperty" &&
value.meta.name === "import" &&
value.property.name === "meta";
if (
node.type === "CallExpression" &&
node.callee.type === "MemberExpression" &&
!node.callee.computed &&
isMeta(node.callee.object) &&
node.callee.property.name === "resolve" &&
node.arguments.length === 1 &&
node.arguments[0].type === "StringLiteral" &&
Object.hasOwn(imports, node.arguments[0].value)
)
// Package imports resolve in the same package scope after bundling.
return;
if (
node.type === "MemberExpression" &&
!node.computed &&
isMeta(node.object) &&
node.property.name === "url"
) {
edits.push({ start: node.start, end: node.end });
return;
}
if (isMeta(node))
throw new Error(
`Unsupported import.meta use in runtime bundle: ${id}`,
);
for (const value of Object.values(node)) {
if (Array.isArray(value)) value.forEach(visit);
else visit(value);
}
};
visit(parse(code, { sourceType: "module" }).program);
for (const { start, end } of edits.sort(
(a, b) => b.start - a.start,
))
code = code.slice(0, start) + moduleUrl + code.slice(end);
return { code, map: null };
},
},
],
});
try {
await bundle.write({
dir: join(compiledRoot, directory),
format: "esm",
entryFileNames: "[name].js",
chunkFileNames:
directory === "mcp"
? "[name]-[hash].js"
: (chunk) => {
// Keep lazy facades distinguishable when source basenames repeat.
if (typeof chunk.facadeModuleId !== "string")
return "[name]-[hash].js";
const path = relative(compiledRoot, chunk.facadeModuleId);
return `${path.replace(/\.js$/u, "").split(sep).join("__")}-[hash].js`;
},
paths: (id) =>
isMetadataModule(id) ? "../generatedPackageMetadata.js" : id,
minifyInternalExports: directory === "mcp",
codeSplitting: true,
strictExecutionOrder: true,
minify: false,
sourcemap: false,
});
} finally {
await bundle.close();
}
}
};
if (
process.argv[1] !== undefined &&
pathToFileURL(resolve(process.argv[1])).href === import.meta.url
)
await bundleRuntime(resolve(dirname(fileURLToPath(import.meta.url)), ".."));
scripts/lib/mcp-verifier-results.mjs
+56 -2
View File
@@ -1,5 +1,11 @@
import { analysisErrorProjectionSchema } from "../../dist/contracts/errorSchemas.js";
import { toolContract } from "../../dist/contracts/toolContracts.js";
import {
expandProviderCapabilities,
expandToolAvailability,
providerCapabilitySet,
} from "../../dist/contracts/toolOutputSchemaPrimitives.js";
import { compareUnicodeCodePoints } from "../../dist/domain/unicodeCodePointOrder.js";
/** Read the textual JSON projection from one MCP tool call. */
export const mcpTextValue = (result) => {
@@ -10,6 +16,10 @@ export const mcpTextValue = (result) => {
const jsonValue = (result) => JSON.parse(mcpTextValue(result));
/** Sort canonical contract names using REA's code-point ordering. */
export const sortVerifierNames = (names) =>
[...names].sort(compareUnicodeCodePoints);
/** Read the complete canonical diagnostic from an MCP tool failure. */
export const requireMcpToolError = (result) => {
if (result.isError !== true) throw new Error("Expected an MCP tool failure");
@@ -35,6 +45,44 @@ const operationResultKey = (contract) => {
return evidence ? "normalized_result" : "result";
};
/** Expand compact session metadata once at the verifier boundary. */
export const expandMcpSessionStatus = (status) => ({
...status,
capabilities: expandProviderCapabilities(status.capabilities),
analysis_provider_candidates: expandProviderCandidates(
status.analysis_provider_candidates,
),
tool_availability: expandToolAvailability(status.tool_availability),
});
/** Expand provider capability groups while retaining candidate identity/state. */
export const expandProviderCandidates = (candidates) =>
candidates.map((candidate) => ({
...candidate,
capabilities: expandProviderCapabilities(
providerCapabilitySet.array().parse(candidate.capabilities),
),
}));
/** Expand CLI provider capability groups at the same owning boundary. */
export const expandProviderOutput = (providers) => ({
...providers,
capabilities: expandProviderCapabilities(
providerCapabilitySet.array().parse(providers.capabilities),
),
analysis_provider_candidates: expandProviderCandidates(
providers.analysis_provider_candidates,
),
});
/** Expand the CLI capabilities command's compact provider descriptor groups. */
export const expandCapabilitiesOutput = (capabilities) => ({
...capabilities,
capabilities: expandProviderCapabilities(
providerCapabilitySet.array().parse(capabilities.capabilities),
),
});
/** Validate an Evidence-producing tool output and return its normalized result. */
export const requireMcpEvidenceResult = (result, operation) => {
const contract = toolContract(operation);
@@ -48,7 +96,10 @@ export const requireMcpLifecycleResult = (result, operation) => {
const contract = toolContract(operation);
if (operationResultKey(contract) !== "result")
throw new Error(`${operation} requires the Evidence result extractor`);
return requireOutput(result, operation, contract).result;
const projected = requireOutput(result, operation, contract).result;
return operation === "binary_session"
? expandMcpSessionStatus(projected)
: projected;
};
/** Project a mixed verifier call using its exact named canonical output schema. */
@@ -58,7 +109,10 @@ export const requireMcpOperationResult = (
contract = toolContract(operation),
) => {
const key = operationResultKey(contract);
return requireOutput(result, operation, contract)[key];
const projected = requireOutput(result, operation, contract)[key];
return operation === "binary_session"
? expandMcpSessionStatus(projected)
: projected;
};
/** Verify provider identity directly from the tool result. */
scripts/lib/native-value-e2e.mjs
+12 -5
View File
@@ -11,6 +11,7 @@ import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { parseEvidence } from "../../dist/domain/evidence.js";
import { analysisProfileSchema } from "../../dist/domain/analysisProfile.js";
import { nativeValueTraceSchema } from "../../dist/domain/native/nativeValueTrace.js";
import { requireMcpLifecycleResult } from "./mcp-verifier-results.mjs";
import { verifyGhidraBoundaries } from "./real-ghidra-boundaries.mjs";
const defaultEntrypoint = fileURLToPath(new URL("../rea.mjs", import.meta.url));
@@ -172,13 +173,16 @@ async function verifyNativeValueAdapters(
{ timeout: 180000 },
);
assert.notEqual(opened.isError, true, JSON.stringify(opened));
const inventory = await client.callTool({
const inventoryReply = await client.callTool({
name: "binary_session",
arguments: {},
});
assert.notEqual(inventory.isError, true, JSON.stringify(inventory));
const inventory = requireMcpLifecycleResult(
inventoryReply,
"binary_session",
);
assert.ok(
inventory.structuredContent?.result?.tool_availability.some(
inventory.tool_availability.some(
(tool) => tool.name === "trace_native_values" && tool.available,
),
"Native value tracing is unavailable in real provider discovery",
@@ -267,11 +271,14 @@ async function verifyNativeValueAdapters(
arguments: {},
});
assert.notEqual(closed.isError, true, JSON.stringify(closed));
const status = await client.callTool({
const statusReply = await client.callTool({
name: "binary_session",
arguments: {},
});
assert.equal(status.structuredContent?.result?.open, false);
assert.equal(
requireMcpLifecycleResult(statusReply, "binary_session").open,
false,
);
assert.deepEqual(
(await readdir(env.TMPDIR)).filter((name) =>
name.startsWith("rea-ghidra-"),
scripts/lib/real-ghidra-boundaries.mjs
-2
View File
@@ -13,8 +13,6 @@ import { promisify } from "node:util";
import Ajv from "ajv/dist/2020.js";
import addFormats from "ajv-formats";
import { analysisErrorProjectionSchema } from "../../dist/contracts/errorSchemas.js";
import { verifyLegacyGhidraReferenceSnapshot } from "./ghidra-reference-snapshot-e2e.mjs";
import { verifyGhidraSnapshotLifecycle } from "./real-ghidra-snapshot-lifecycle.mjs";
import { verifyGhidraTargetAdmission } from "./real-ghidra-target-admission.mjs";
scripts/lib/real-hopper-workflows.mjs
+1 -1
View File
@@ -37,7 +37,7 @@ export async function verifyHopperWorkflows(call, invalid, procedure) {
);
assert.equal(cycle.length, 2, "fixture omitted its source-owned call cycle");
const cycleGraph = await call("get_call_graph", { address: cycle[0].value });
const cycleNodes = Object.values(cycleGraph).flat();
const cycleNodes = Object.values(cycleGraph.graph).flat();
assert.equal(cycleNodes.length, 2);
assert.deepEqual(
cycleNodes.map((node) => node.address).sort(),
scripts/lib/verify-package-core.mjs
+8 -5
View File
@@ -2,6 +2,10 @@ import { execFile } from "node:child_process";
import { lstat } from "node:fs/promises";
import { promisify } from "node:util";
import { Ajv2020 } from "ajv/dist/2020.js";
import {
requireMcpLifecycleResult,
sortVerifierNames,
} from "./mcp-verifier-results.mjs";
/** Promisified child_process execution helper. */
export const exec = promisify(execFile);
@@ -46,11 +50,10 @@ export const verifyCompleteToolCatalog = async (client, options) => {
{ name: "binary_session", arguments: {} },
options,
);
const availability = status.structuredContent?.result?.tool_availability;
if (!Array.isArray(availability))
throw new Error("packaged MCP omitted tool availability");
const expected = availability.map(({ name }) => name).sort();
const observed = listed.tools.map(({ name }) => name).sort();
const expanded = requireMcpLifecycleResult(status, "binary_session");
const availability = expanded.tool_availability;
const expected = sortVerifierNames(availability.map(({ name }) => name));
const observed = sortVerifierNames(listed.tools.map(({ name }) => name));
if (JSON.stringify(observed) !== JSON.stringify(expected))
throw new Error(
"packaged MCP tool inventory diverged from session metadata",
scripts/prepack.mjs
+4
View File
@@ -47,6 +47,10 @@ if (await pathExists(join(root, "src"))) {
for (const path of [
"dist/main.js",
"dist/mcp/main.js",
"dist/cli-runtime/cli.js",
"dist/cli-runtime/cliOutput.js",
"dist/cli-runtime/streamedJsonOutput.js",
"dist/cli-runtime/mcpDoctor.js",
"dist/mcpDoctor.js",
"dist/cli.js",
"dist/cliOutput.js",
scripts/rea.mjs
+11 -6
View File
@@ -33,8 +33,12 @@ if (
const runtimeFiles = isMcpMode
? ["dist/mcp/main.js"]
: isMcpDoctorMode
? ["dist/mcp/main.js", "dist/mcpDoctor.js"]
: ["dist/cli.js", "dist/cliOutput.js", "dist/cli/streamedJsonOutput.js"];
? ["dist/mcp/main.js", "dist/cli-runtime/mcpDoctor.js"]
: [
"dist/cli-runtime/cli.js",
"dist/cli-runtime/cliOutput.js",
"dist/cli-runtime/streamedJsonOutput.js",
];
if (!(await compiledRuntimeExists(runtimeFiles))) {
process.stderr.write(
@@ -45,7 +49,8 @@ if (!(await compiledRuntimeExists(runtimeFiles))) {
const { runEntrypoint } = await import("../dist/mcp/main.js");
await runEntrypoint();
} else if (isMcpDoctorMode) {
const { runProductionMcpDoctorCli } = await import("../dist/mcpDoctor.js");
const { runProductionMcpDoctorCli } =
await import("../dist/cli-runtime/mcpDoctor.js");
const result = await runProductionMcpDoctorCli(args.slice(2), {
dispatcherPath: fileURLToPath(import.meta.url),
packageRoot,
@@ -53,14 +58,14 @@ if (!(await compiledRuntimeExists(runtimeFiles))) {
process.stdout.write(result.output);
process.exitCode = result.exitCode;
} else {
const { createCli } = await import("../dist/cli.js");
const { createCli } = await import("../dist/cli-runtime/cli.js");
const { createStreamedCliJsonOutput } =
await import("../dist/cli/streamedJsonOutput.js");
await import("../dist/cli-runtime/streamedJsonOutput.js");
const {
renderCliOutputArgumentError,
renderEmptyFilteredCliOutput,
validateCliOutputArguments,
} = await import("../dist/cliOutput.js");
} = await import("../dist/cli-runtime/cliOutput.js");
const outputArguments = validateCliOutputArguments(args);
if (!outputArguments.ok) {
process.stdout.write(renderCliOutputArgumentError(outputArguments));
scripts/verify-apktool.mjs
+80 -67
View File
@@ -1,7 +1,5 @@
import { createHash } from "node:crypto";
import { mkdtemp, readFile, rm, stat } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { readFile, stat } from "node:fs/promises";
import { ApktoolResourceAnalysisService } from "../dist/application/apktool/ApktoolResourceAnalysisService.js";
import { createApktoolResourceAnalysisProvider } from "../dist/composition/apktool.js";
@@ -11,8 +9,8 @@ if (options.help) {
process.stdout.write(
"Usage: npm run verify:apktool [-- --apk PATH]\n" +
"Requires a caller-selected apktool launcher (REA_APKTOOL_COMMAND or\n" +
"PATH) and a Java runtime. Without --apk, a small signed probe APK is\n" +
"expected at the path given by --apk; nothing is downloaded or built.\n",
"PATH) and a Java runtime. Without --apk, only client availability is\n" +
"checked. Supply a local APK to verify decoding; nothing is downloaded or built.\n",
);
process.exit(0);
}
@@ -38,70 +36,80 @@ const run = async (operation, input) => {
return result.value.normalized_result;
};
const client = await run("inspect_apktool_client", {});
if (client !== null) {
check(
"apktool launcher reports a version",
typeof client.client.apktool_version === "string" &&
client.client.apktool_version !== "",
`command=${client.client.command} version=${client.client.apktool_version}`,
);
}
try {
const client = await run("inspect_apktool_client", {});
if (client !== null) {
check(
"apktool launcher reports a version",
typeof client.client.apktool_version === "string" &&
client.client.apktool_version !== "",
`command=${client.client.command} version=${client.client.apktool_version}`,
);
}
if (options.apk !== null) {
const decoded = await run("decode_android_resources", {
path: options.apk,
include_strings: true,
});
if (decoded !== null) {
const bytes = await stat(options.apk).then((info) => info.size);
const digest = createHash("sha256")
.update(await readFile(options.apk))
.digest("hex");
check(
"decoded target identity matches the file on disk",
decoded.target.bytes === bytes && decoded.target.sha256 === digest,
`${bytes} bytes`,
);
check(
"metadata carries version and SDK facts",
typeof decoded.metadata.version_name === "string" &&
decoded.metadata.min_sdk_version !== null &&
decoded.metadata.target_sdk_version !== null,
`version=${String(decoded.metadata.version_name)} sdk=${String(decoded.metadata.min_sdk_version)}..${String(decoded.metadata.target_sdk_version)}`,
);
check(
"decoded manifest names the package",
typeof decoded.metadata.package_name === "string" &&
decoded.manifest !== null &&
decoded.manifest.includes(decoded.metadata.package_name),
String(decoded.metadata.package_name),
);
check(
"string resources projected",
decoded.strings.length > 0,
`${decoded.strings.length} string(s)`,
);
if (decoded.locales.length > 0) {
const locale = await run("decode_android_resources", {
path: options.apk,
include_strings: true,
locale: decoded.locales[0],
});
if (locale !== null)
check(
`locale table projected for ${String(decoded.locales[0])}`,
Array.isArray(locale.strings),
`${locale.strings.length} string(s)`,
);
if (options.apk !== null) {
const decoded = await run("decode_android_resources", {
path: options.apk,
include_strings: true,
});
if (decoded !== null) {
const bytes = await stat(options.apk).then((info) => info.size);
const digest = createHash("sha256")
.update(await readFile(options.apk))
.digest("hex");
check(
"decoded target identity matches the file on disk",
decoded.target.bytes === bytes && decoded.target.sha256 === digest,
`${bytes} bytes`,
);
check(
"metadata carries version and SDK facts",
typeof decoded.metadata.version_name === "string" &&
decoded.metadata.min_sdk_version !== null &&
decoded.metadata.target_sdk_version !== null,
`version=${String(decoded.metadata.version_name)} sdk=${String(decoded.metadata.min_sdk_version)}..${String(decoded.metadata.target_sdk_version)}`,
);
check(
"decoded manifest names the package",
typeof decoded.metadata.package_name === "string" &&
decoded.manifest !== null &&
decoded.manifest.includes(decoded.metadata.package_name),
String(decoded.metadata.package_name),
);
check(
"string resources projected",
decoded.strings.length > 0,
`${decoded.strings.length} string(s)`,
);
if (decoded.locales.length > 0) {
const locale = await run("decode_android_resources", {
path: options.apk,
include_strings: true,
locale: decoded.locales[0],
});
if (locale !== null)
check(
`locale table projected for ${String(decoded.locales[0])}`,
Array.isArray(locale.strings),
`${locale.strings.length} string(s)`,
);
}
}
}
}
try {
await provider.close();
} catch {
// The provider owns per-call workspaces only; closing is best-effort.
} catch (cause) {
failures += 1;
process.stdout.write(
`[FAIL] verifier execution — ${cause instanceof Error ? cause.message : String(cause)}\n`,
);
} finally {
try {
await provider.close();
} catch (cause) {
failures += 1;
process.stdout.write(
`[FAIL] provider cleanup — ${cause instanceof Error ? cause.message : String(cause)}\n`,
);
}
}
if (failures > 0) {
@@ -116,7 +124,12 @@ function parseArguments(argv) {
const argument = argv[index];
if (argument === "--help" || argument === "-h") parsed.help = true;
else if (argument === "--apk") {
parsed.apk = argv[index + 1] ?? null;
const path = argv[index + 1];
if (path === undefined) {
process.stderr.write("--apk requires a local APK path\n");
process.exit(2);
}
parsed.apk = path;
index += 1;
} else {
process.stderr.write(`unknown argument: ${argument}\n`);
scripts/verify-binary-layout.mjs
+23
View File
@@ -662,6 +662,16 @@ try {
bytes.writeBigUInt64LE(8n, header + 56);
return [`undersized-entry-${type}`, bytes, "invalid_input"];
});
const nonintegralSymbolEntries = ["SHT_SYMTAB", "SHT_DYNSYM"].map((type) => {
const section = protectedReport.sections.find((item) => item.type === type);
assert.notEqual(section, undefined);
const bytes = Buffer.from(sectionBearingBytes);
const header = Number(BigInt(section.header_location.offset));
const size = bytes.readBigUInt64LE(header + 32);
assert.ok(size + 1n <= BigInt(bytes.length) - BigInt(section.offset));
bytes.writeBigUInt64LE(size + 1n, header + 32);
return [`nonintegral-entry-count-${type}`, bytes, "invalid_input"];
});
const relocationSection = relocatable.sections.find(
(item) => item.type === "SHT_RELA",
);
@@ -677,6 +687,17 @@ try {
bytes.writeBigUInt64LE(8n, header + 56);
return [`undersized-entry-${type}`, bytes, "invalid_input"];
});
const nonintegralRelocationEntries = [
["RELA", 4, 24],
["REL", 9, 16],
].map(([type, sectionType, entrySize]) => {
const bytes = Buffer.from(object);
const header = Number(BigInt(relocationSection.header_location.offset));
bytes.writeUInt32LE(sectionType, header + 4);
bytes.writeBigUInt64LE(BigInt(entrySize + 1), header + 32);
bytes.writeBigUInt64LE(BigInt(entrySize), header + 56);
return [`nonintegral-entry-count-${type}`, bytes, "invalid_input"];
});
const invalidRelocationReferences = [
["missing-symbol-table", relocatable.sections.length + 100],
["wrong-symbol-table-kind", 1],
@@ -1066,7 +1087,9 @@ try {
],
...zeroSymbolEntries,
...undersizedSymbolEntries,
...nonintegralSymbolEntries,
...undersizedRelocationEntries,
...nonintegralRelocationEntries,
...invalidRelocationReferences,
["unterminated-dynamic-segment", shortDynamic, "invalid_input"],
["unterminated-interpreter", unterminatedInterpreter, "invalid_input"],
scripts/verify-claude-client.mjs
+71 -14
View File
@@ -5,11 +5,16 @@ import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
import Ajv2020 from "ajv/dist/2020.js";
import { CATALOG_IDENTITY } from "../dist/catalogIdentity.js";
import {
TOOL_CONTRACTS,
toolContract,
} from "../dist/contracts/toolContracts.js";
import { exec } from "./lib/verify-package-core.mjs";
import {
expandMcpSessionStatus,
sortVerifierNames,
} from "./lib/mcp-verifier-results.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
import { createAnthropicModelFixture } from "./verify/clients/anthropic-model-fixture.mjs";
@@ -58,6 +63,8 @@ await writeFile(
'export function result() { return { clientCompatibility: "REA_CLIENT_FIXTURE", count: 7 }; }\n',
);
let skillRequested = false;
let sessionRequested = false;
let sessionStatusSeen = false;
let skillLoaded = false;
let expectedSkillBody;
let evidenceSeen = false;
@@ -73,7 +80,7 @@ const containsText = (value, text) => {
return Object.values(value).some((nested) => containsText(nested, text));
return false;
};
const extractEvidence = (value) => {
const extractInlineResult = (value, matches) => {
if (typeof value === "string") {
try {
// Native Claude appends this resource hint to tool results. Original bodies remain intact in request artifacts.
@@ -81,20 +88,16 @@ const extractEvidence = (value) => {
/\n\n<system-reminder>\n<total_tokens>\d+ tokens left<\/total_tokens>\n<\/system-reminder>$/u,
"",
);
return extractEvidence(JSON.parse(json));
return extractInlineResult(JSON.parse(json), matches);
} catch {
return undefined;
}
}
if (value && typeof value === "object") {
if (
typeof value.evidence_id === "string" &&
value.normalized_result?.graph?.nodes?.length > 0
)
return value;
if (matches(value)) return value;
for (const nested of Object.values(value)) {
const evidence = extractEvidence(nested);
if (evidence) return evidence;
const result = extractInlineResult(nested, matches);
if (result) return result;
}
}
return undefined;
@@ -125,10 +128,12 @@ const fixture = await createAnthropicModelFixture({
0,
-"analyze_javascript_application".length,
);
const expected = TOOL_CONTRACTS.map((t) => prefix + t.name).sort();
const expected = sortVerifierNames(
TOOL_CONTRACTS.map((t) => prefix + t.name),
);
const reaTools = tools.filter((t) => t.name.startsWith(prefix));
assert.deepEqual(
reaTools.map((t) => t.name).sort(),
sortVerifierNames(reaTools.map((t) => t.name)),
expected,
"Claude must forward the complete REA catalog after connection",
);
@@ -146,11 +151,61 @@ const fixture = await createAnthropicModelFixture({
schemasChecked++;
}
}
if (mode === "call" && !sessionStatusSeen) {
const observedStatus = extractInlineResult(
body.messages,
(value) =>
Boolean(value.result?.server_identity) &&
Array.isArray(value.result?.tool_availability?.groups),
);
if (observedStatus) {
const parsedStatus =
toolContract("binary_session").outputSchema.parse(observedStatus);
const status = {
...parsedStatus,
result: expandMcpSessionStatus(parsedStatus.result),
};
assert.deepEqual(
status.result.server_identity.catalog,
CATALOG_IDENTITY,
"binary_session must return the exact compact catalog counts and digests",
);
assert.equal(
Object.hasOwn(status.result.server_identity.catalog, "tools"),
false,
"binary_session must not inline the duplicated canonical tool catalog",
);
assert.deepEqual(
sortVerifierNames(
status.result.tool_availability.map((entry) => entry.name),
),
sortVerifierNames(TOOL_CONTRACTS.map((contract) => contract.name)),
"binary_session must retain complete canonical tool availability",
);
sessionStatusSeen = true;
} else {
assert(
!sessionRequested,
"Claude must return the inline binary_session result before continuing",
);
const session = tools.find(
(tool) => tool.name === `${prefix}binary_session`,
);
assert(session, "Claude must expose the REA binary_session tool");
sessionRequested = true;
return call(session.name, {});
}
}
skillLoaded ||=
expectedSkillBody !== undefined &&
containsText(body.messages, expectedSkillBody) &&
containsText(body.messages, skillDirectory);
const observed = extractEvidence(body.messages);
const observed = extractInlineResult(
body.messages,
(value) =>
typeof value.evidence_id === "string" &&
value.normalized_result?.graph?.nodes?.length > 0,
);
if (observed) {
const evidence = toolContract(
"analyze_javascript_application",
@@ -184,7 +239,7 @@ const fixture = await createAnthropicModelFixture({
await writeFile(join(lab, "claude-evidence.json"), bytes);
evidenceSeen = true;
}
if (mode === "chat" || evidenceSeen) {
if (mode === "chat" || (sessionStatusSeen && evidenceSeen)) {
finalSeen = true;
return [{ type: "text", text: "REA_CLIENT_COMPATIBILITY_OK" }];
}
@@ -294,6 +349,7 @@ try {
"Skill",
"WaitForMcpServers",
"mcp__rea__analyze_javascript_application",
...(mode === "call" ? ["mcp__rea__binary_session"] : []),
],
},
}),
@@ -324,7 +380,7 @@ try {
assert.equal(parsed.result, "REA_CLIENT_COMPATIBILITY_OK");
assert.deepEqual(parsed.permission_denials, []);
assert(finalSeen);
if (mode === "call") assert(evidenceSeen && skillLoaded);
if (mode === "call") assert(evidenceSeen && skillLoaded && sessionStatusSeen);
} catch (cause) {
failure = cause;
if (typeof cause?.stdout === "string")
@@ -355,6 +411,7 @@ try {
schemasChecked,
pendingServerWaits,
evidenceSeen,
sessionStatusSeen,
skillLoaded,
artifactDigest,
finalSeen,
scripts/verify-deepseek-client.mjs
+91 -29
View File
@@ -5,11 +5,12 @@ import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import { Ajv2020 } from "ajv/dist/2020.js";
import { parse as parseYaml } from "yaml";
import {
TOOL_CONTRACTS,
toolContract,
} from "../dist/contracts/toolContracts.js";
import { exec } from "./lib/verify-package-core.mjs";
import { exec, pathExists } from "./lib/verify-package-core.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
import { createOpenAiModelFixture } from "./verify/clients/model-fixture.mjs";
@@ -32,6 +33,7 @@ const skillPath = join(
"SKILL.md",
);
const cli = join(runtimeRoot, "scripts", "rea.mjs");
const configPath = join(profile, "cordis.patch.yml");
const verifier = createVerifierRun();
await Promise.all([
mkdir(account),
@@ -66,32 +68,32 @@ const clientVersion = (
timeout: 30_000,
})
).stdout.trim();
for (const args of [
["setup", "--skill", "--dry-run"],
["setup", "--skill", "--yes"],
]) {
const lifecycle = [];
const runRea = async (args, artifact) => {
const result = await exec(process.execPath, [cli, ...args, "--json"], {
env: environment,
cwd: workspace,
timeout: 60_000,
});
const parsed = JSON.parse(result.stdout);
assert.equal(parsed.status, args.includes("--dry-run") ? "planned" : "ready");
if (args.includes("--dry-run")) {
assert.equal(parsed.plannedActions.length, 1);
assert.equal(parsed.plannedActions[0].kind, "install_skill");
assert.equal(parsed.plannedActions[0].target, dirname(skillPath));
}
await writeFile(
join(lab, args.includes("--dry-run") ? "setup-plan.json" : "setup.json"),
result.stdout,
);
}
assert(
(await readFile(skillPath, "utf8")).includes(
"name: reverse-engineer-anything",
),
);
await writeFile(join(lab, `${artifact}.json`), result.stdout);
return JSON.parse(result.stdout);
};
const patches = (text) =>
parseYaml(text, {
customTags: [
{
tag: "tag:yaml.org,2002:js",
resolve: (source) => ({ __jsExpr: source }),
},
],
});
const withoutRea = (text) =>
patches(text).flatMap((op) => {
if (op.id === "rea-mcp") return [];
if (!Array.isArray(op.insert)) return [op];
const insert = op.insert.filter((entry) => entry.id !== "rea-mcp");
return insert.length === 0 ? [] : [{ ...op, insert }];
});
let catalog;
let patternCount = 0;
@@ -247,10 +249,9 @@ const fixture = await createOpenAiModelFixture({
});
let failure;
try {
// Use the client's native Cordis patch DSL; generic `rea setup --client` does not edit it.
await writeFile(
join(profile, "cordis.patch.yml"),
`
// Seed model settings and a stale registration; the public setup workflow
// must produce the registration this native client actually boots.
const original = `
- id: llm-pi-ai
config:
providers:
@@ -275,13 +276,59 @@ try {
config:
serverName: rea
transport: stdio
command: ${JSON.stringify(process.execPath)}
args: ${JSON.stringify([cli, "mcp"])}
command: stale-launcher
args: [mcp]
cwd: !!js process.cwd()
env:
REA_PROCESS_RUN_ID: ${JSON.stringify(verifier.run_id)}
`,
`;
await writeFile(configPath, original);
const planned = await runRea(
["setup", "--client", "dsh", "--dry-run"],
"setup-plan",
);
assert.equal(planned.status, "planned");
assert(planned.plannedActions.some((action) => action.target === configPath));
assert(
planned.plannedActions.some(
(action) => action.target === dirname(skillPath),
),
);
assert(
planned.plannedActions.every(
(action) =>
action.target === configPath || action.target === dirname(skillPath),
),
);
assert.equal(await readFile(configPath, "utf8"), original);
assert.equal(await pathExists(`${configPath}.rea.backup`), false);
lifecycle.push("dry-run");
const setup = await runRea(["setup", "--client", "dsh", "--yes"], "setup");
assert.equal(setup.status, "ready");
assert.equal(await readFile(`${configPath}.rea.backup`, "utf8"), original);
const configured = await readFile(configPath, "utf8");
assert.deepEqual(withoutRea(configured), withoutRea(original));
assert(configured.includes("!!js process.cwd()"));
assert(
(await readFile(skillPath, "utf8")).includes(
"name: reverse-engineer-anything",
),
);
lifecycle.push("setup");
const repeated = await runRea(
["setup", "--client", "dsh", "--yes"],
"setup-repeat",
);
assert.deepEqual(repeated.appliedActions, []);
assert.equal(await readFile(configPath, "utf8"), configured);
assert.equal(await readFile(`${configPath}.rea.backup`, "utf8"), original);
lifecycle.push("idempotent-setup");
const doctor = await runRea(
["doctor", "--client", "dsh", "--skill"],
"doctor",
);
assert.equal(doctor.healthy, true);
lifecycle.push("doctor");
const execution = exec(
command,
[
@@ -304,6 +351,20 @@ try {
assert.equal(fixture.failure, undefined);
assert(finalSeen && result.stdout.includes("REA_CLIENT_COMPATIBILITY_OK"));
if (mode === "call") assert(evidenceSeen && skillLoaded);
lifecycle.push(mode === "call" ? "native-tool-invocation" : "native-chat");
const uninstall = await runRea(["uninstall"], "uninstall");
assert.equal(uninstall.status, "complete");
assert(
uninstall.items.some(
(item) => item.name === "dsh" && item.status === "removed",
),
);
assert.deepEqual(
patches(await readFile(configPath, "utf8")),
withoutRea(original),
);
assert.equal(await pathExists(skillPath), false);
lifecycle.push("uninstall");
} catch (cause) {
failure = cause;
if (typeof cause?.stdout === "string")
@@ -316,6 +377,7 @@ try {
client: "deepseek-harness",
clientVersion,
mode,
lifecycle,
passed: failure === undefined,
artifacts: lab,
runtimeRoot,
scripts/verify-gemini-client.mjs
+19 -12
View File
@@ -2,7 +2,7 @@ import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { join, resolve, sep } from "node:path";
import { fileURLToPath } from "node:url";
import Ajv2020 from "ajv/dist/2020.js";
import {
@@ -13,14 +13,18 @@ import { exec } from "./lib/verify-package-core.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
import { createGeminiModelFixture } from "./verify/clients/gemini-model-fixture.mjs";
assert(
process.platform !== "win32",
"Native Gemini verifier requires POSIX; Windows is unverified",
);
const repo = fileURLToPath(new URL("..", import.meta.url));
const runtimeRoot = process.env.REA_VERIFY_RUNTIME_ROOT ?? repo;
const command = process.env.REA_VERIFY_GEMINI_COMMAND ?? "gemini";
const mode = process.argv[2] ?? "call";
// npm installs of Gemini CLI provide a JavaScript entry point; on Windows only a gemini.cmd shim reaches
// PATH, which Node does not spawn without a shell, so a selected .js entry runs under this Node. It is
// resolved once, because the preflight runs from the caller's directory and the client from the workspace.
const entryPoint = /\.(c|m)?js$/iu.test(command) ? resolve(command) : undefined;
const launch = (args) =>
entryPoint === undefined
? [command, args]
: [process.execPath, [entryPoint, ...args]];
assert(
["chat", "call"].includes(mode),
"Usage: verify-gemini-client.mjs [chat|call]",
@@ -60,11 +64,15 @@ const environment = {
let clientVersion;
try {
clientVersion = (
await exec(command, ["--version"], { env: environment, timeout: 30_000 })
await exec(...launch(["--version"]), { env: environment, timeout: 30_000 })
).stdout.trim();
} catch (cause) {
const windowsHint =
process.platform === "win32"
? " On Windows, select the JavaScript entry point; npm installs provide it as node_modules/@google/gemini-cli/bundle/gemini.js."
: "";
throw new Error(
`Gemini preflight failed for ${command}; install Gemini CLI or set REA_VERIFY_GEMINI_COMMAND. ${String(cause)}`,
`Gemini preflight failed for ${command}; install Gemini CLI or set REA_VERIFY_GEMINI_COMMAND.${windowsHint} ${String(cause)}`,
);
}
const original = JSON.stringify(
@@ -103,11 +111,11 @@ for (const args of [
assert(parsed.plannedActions.some((a) => a.target === configPath));
assert(
parsed.plannedActions.some((a) =>
a.target.startsWith(join(profile, ".agents", "skills") + "/"),
a.target.startsWith(join(profile, ".agents", "skills") + sep),
),
);
assert(
parsed.plannedActions.every((a) => a.target.startsWith(profile + "/")),
parsed.plannedActions.every((a) => a.target.startsWith(profile + sep)),
);
}
}
@@ -300,8 +308,7 @@ const fixture = await createGeminiModelFixture({
let failure;
try {
const response = await exec(
command,
[
...launch([
"--model",
"gemini-2.5-flash",
"--approval-mode",
@@ -314,7 +321,7 @@ try {
mode === "chat"
? "Say hello briefly."
: `Use REA to analyze the JavaScript directory ${target}. Report the result.`,
],
]),
{
env: {
...environment,
scripts/verify-package-capabilities.mjs
+9 -3
View File
@@ -1,13 +1,19 @@
import { json, run } from "./lib/verify-package-core.mjs";
import {
expandCapabilitiesOutput,
expandProviderOutput,
} from "./lib/mcp-verifier-results.mjs";
/** Verify packaged capabilities, providers, and non-Linux search results. */
export async function verifyPackageCapabilitiesAndSearch({ cli, environment }) {
const capabilities = json(
await run(cli, ["capabilities", "--json"], environment),
const capabilities = expandCapabilitiesOutput(
json(await run(cli, ["capabilities", "--json"], environment)),
);
if (!Array.isArray(capabilities.capabilities))
throw new Error("packaged capabilities CLI failed");
const providers = json(await run(cli, ["providers", "--json"], environment));
const providers = expandProviderOutput(
json(await run(cli, ["providers", "--json"], environment)),
);
const candidates = providers.analysis_provider_candidates;
if (
!Array.isArray(candidates) ||
scripts/verify-package-mcp.mjs
+14 -13
View File
@@ -1,4 +1,7 @@
import { requireMcpToolError } from "./lib/mcp-verifier-results.mjs";
import {
requireMcpLifecycleResult,
requireMcpToolError,
} from "./lib/mcp-verifier-results.mjs";
import assert from "node:assert/strict";
import { join } from "node:path";
import { execFile } from "node:child_process";
@@ -21,14 +24,14 @@ const verifyMcpToolsAndPrompts = async (client, mcpOptions) => {
};
const verifyMcpTargetFree = async (client, mcpOptions) => {
const status = await client.callTool(
const result = await client.callTool(
{ name: "binary_session", arguments: {} },
mcpOptions,
);
const currentDocument =
status.structuredContent?.result?.tool_availability?.find(
({ name }) => name === "current_document",
);
const status = requireMcpLifecycleResult(result, "binary_session");
const currentDocument = status.tool_availability.find(
({ name }) => name === "current_document",
);
if (
currentDocument?.available !== false ||
currentDocument.reason !== "target_required"
@@ -67,15 +70,13 @@ const verifyMcpOpenAndBind = async (client, mcpOptions) => {
);
if (opened.isError === true)
throw new Error("packaged MCP could not open a binary");
const providerStatusEnvelope = json(
prompts.mcpText(
await client.callTool(
{ name: "binary_session", arguments: {} },
mcpOptions,
),
const providerStatus = requireMcpLifecycleResult(
await client.callTool(
{ name: "binary_session", arguments: {} },
mcpOptions,
),
"binary_session",
);
const providerStatus = providerStatusEnvelope.result;
if (
providerStatus.analysis_provider_binding?.provider?.id !== "hopper" ||
providerStatus.analysis_provider_binding?.selection_source !== "request"
scripts/verify-packaged-ghidra-windows.mjs
+14 -9
View File
@@ -1,7 +1,10 @@
#!/usr/bin/env node
import assert from "node:assert/strict";
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
import {
expandProviderOutput,
requireMcpOperationResult,
} from "./lib/mcp-verifier-results.mjs";
import { verifyGhidraCompactReferences } from "./lib/real-ghidra-compact-references.mjs";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
@@ -230,14 +233,16 @@ const assertCleanup = async () => {
assert.equal(entries.length, 0, "Provider runtime cleanup left files.");
};
try {
const providers = JSON.parse(
(
await exec(process.execPath, [entry, "providers", "--json"], {
env: environment,
cwd: callerDirectory,
timeout: 30_000,
})
).stdout,
const providers = expandProviderOutput(
JSON.parse(
(
await exec(process.execPath, [entry, "providers", "--json"], {
env: environment,
cwd: callerDirectory,
timeout: 30_000,
})
).stdout,
),
);
assert.equal(providers.analysis_provider_binding, null);
assert.ok(Array.isArray(providers.analysis_provider_candidates));
scripts/verify-published-package.mjs
+26 -9
View File
@@ -10,6 +10,11 @@ import { pathToFileURL } from "node:url";
import { lt, rcompare } from "semver";
import { CATALOG_IDENTITY } from "../dist/catalogIdentity.js";
import { TOOL_CONTRACTS } from "../dist/contracts/toolContracts.js";
import {
requireMcpLifecycleResult,
sortVerifierNames,
} from "./lib/mcp-verifier-results.mjs";
import { completeVerifierRun, createVerifierRun } from "./lib/verifier-run.mjs";
const verifierRun = createVerifierRun();
@@ -54,27 +59,39 @@ try {
await client.connect(transport);
const tools = await client.listTools();
const toolNames = tools.tools.map(({ name }) => name);
const canonicalToolNames = new Set(
CATALOG_IDENTITY.tools.map(({ name }) => name),
);
const canonicalToolNames = new Set(TOOL_CONTRACTS.map(({ name }) => name));
const unknownToolNames = toolNames.filter(
(name) => !canonicalToolNames.has(name),
);
const publishedToolNames = new Set(toolNames);
const missingToolNames = [...canonicalToolNames].filter(
(name) => !publishedToolNames.has(name),
);
publishedToolCount = toolNames.length;
if (
publishedToolCount === 0 ||
new Set(toolNames).size !== publishedToolCount ||
unknownToolNames.length > 0
publishedToolNames.size !== publishedToolCount ||
unknownToolNames.length > 0 ||
missingToolNames.length > 0
)
throw new Error(
`Published MCP exposed an invalid capability-scoped tool projection (${String(publishedToolCount)} tools, unknown: ${unknownToolNames.join(", ") || "none"})`,
`Published MCP exposed an invalid canonical tool catalog (${String(publishedToolCount)} tools, unknown: ${unknownToolNames.join(", ") || "none"}, missing: ${missingToolNames.join(", ") || "none"})`,
);
const status = await client.callTool({
const statusReply = await client.callTool({
name: "binary_session",
arguments: {},
});
if (status.isError === true)
throw new Error("Published target-free MCP session check failed");
const status = requireMcpLifecycleResult(statusReply, "binary_session");
const availabilityNames = sortVerifierNames(
status.tool_availability.map(({ name }) => name),
);
if (
JSON.stringify(availabilityNames) !==
JSON.stringify(sortVerifierNames(canonicalToolNames))
)
throw new Error(
"Published binary_session omitted canonical tool availability",
);
update = await verifyPublishedUpdate({ version, canaryRoot });
} catch (cause) {
throw new Error(`Published package verification failed: ${stderr}`, {
scripts/verify-real-ghidra-com.mjs
+1 -1
View File
@@ -158,7 +158,7 @@ try {
observations: {
...x.observations,
source_files: x.observations.source_files.map(
({ name, ...file }) => file,
({ name: _name, ...file }) => file,
),
},
});
scripts/verify-real-ghidra-mips-relocatable.mjs
+1
View File
@@ -163,6 +163,7 @@ async function verifyFixture(byteOrder, kernelModule) {
});
assert.equal(coverage.unplaced, 0, "Fixture references an empty section");
assert.ok(coverage.shared_hi16, "No HI16 shared by several LO16s");
assert.ok(coverage.multiple_hi16, "No LO16 resolved several pending HI16s");
assert.ok(coverage.lo16_carry, "No LO16 >= 0x8000 forced a HI16 carry");
assert.ok(coverage.external_code, "No external code relocation");
assert.ok(coverage.external_data, "No external data relocation");
scripts/verify-real-ghidra-mips-shared.mjs
+338
View File
@@ -0,0 +1,338 @@
#!/usr/bin/env node
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { promisify } from "node:util";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { parseBinaryTarget } from "../dist/application/BinaryTargetResolver.js";
import { parseEvidence } from "../dist/domain/evidence.js";
import { parseAnalysisSnapshot } from "../dist/domain/analysisSnapshot.js";
import { buildMips16SharedFixture } from "../tests/conformance/ghidra/mips16-shared-fixture.mjs";
import {
requireMcpOperationResult,
requireMcpToolError,
} from "./lib/mcp-verifier-results.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
if (
process.env.GHIDRA_INSTALL_DIR === undefined ||
!["linux", "darwin"].includes(process.platform)
)
throw new Error(
"This optional lane requires BYO Ghidra on Linux/macOS and GNU MIPS cross-binutils.",
);
const execute = promisify(execFile);
const run = createVerifierRun();
const workspace = await mkdtemp(join(tmpdir(), "rea-mips-shared-proof-"));
const runtime = join(workspace, "runtime");
await mkdir(runtime);
const env = {
PATH: process.env.PATH ?? "/usr/bin:/bin",
TMPDIR: runtime,
REA_LOG_LEVEL: "silent",
REA_ANALYSIS_PROVIDER: "ghidra",
REA_PROCESS_RUN_ID: run.run_id,
GHIDRA_INSTALL_DIR: process.env.GHIDRA_INSTALL_DIR,
...(process.env.JAVA_HOME === undefined
? {}
: { JAVA_HOME: process.env.JAVA_HOME }),
...(process.env.GHIDRA_HEADLESS_MAXMEM === undefined
? {}
: { GHIDRA_HEADLESS_MAXMEM: process.env.GHIDRA_HEADLESS_MAXMEM }),
};
const reports = [];
let failure;
try {
for (const byteOrder of ["big", "little"]) {
for (const variant of [
"declared",
"recordless",
"symbol-relative",
"section-relocations",
]) {
const fixture = await buildMips16SharedFixture(
workspace,
byteOrder,
variant,
);
const resolved = await parseBinaryTarget(fixture.path);
if (!resolved.ok) throw resolved.error;
assert.equal(resolved.value.sha256, fixture.sha256);
assert.equal(resolved.value.architecture, "mips");
if (variant === "section-relocations") {
assert.equal(resolved.value.mips.sharedLoad.status, "unavailable");
assert.match(
resolved.value.mips.sharedLoad.reason,
/section relocation declarations/u,
);
} else {
assert.equal(resolved.value.mips.sharedLoad.status, "complete");
assert.deepEqual(
resolved.value.mips.sharedLoad.relocationTypes,
[0, 3],
);
assert.equal(
resolved.value.mips.sharedLoad.symbolRelativeCount,
variant === "symbol-relative" ? 1 : 0,
);
}
if (variant === "recordless")
assert.equal(resolved.value.mips.abiFlags, null);
else {
assert.equal(resolved.value.mips.abiFlags.fpAbi, 5);
assert.equal(resolved.value.mips.abiFlags.ases, 0x400);
}
const transport = new StdioClientTransport({
command: process.execPath,
args: ["scripts/rea.mjs", "mcp"],
cwd: process.cwd(),
env,
stderr: "pipe",
});
const client = new Client({
name: "rea-mips16-shared-conformance",
version: "1",
});
let stderr = "";
transport.stderr?.on("data", (chunk) => {
stderr = (stderr + chunk.toString()).slice(-262144);
});
let calls = 0;
const reply = async (name, args = {}) => {
const response = await client.callTool(
{ name, arguments: args },
{ timeout: 180000 },
);
await writeFile(
join(workspace, `${byteOrder}-${variant}-${++calls}-${name}.json`),
JSON.stringify(response, null, 2),
);
return response;
};
const call = async (name, args) =>
requireMcpOperationResult(await reply(name, args), name);
const original = await readFile(fixture.path);
try {
await client.connect(transport);
await client.listTools();
if (
variant === "symbol-relative" ||
variant === "section-relocations"
) {
const refusal = requireMcpToolError(
await reply("open_binary", {
path: fixture.path,
provider_id: "ghidra",
}),
);
assert.equal(refusal.category, "unsupported_provider");
assert.match(
JSON.stringify(refusal),
variant === "symbol-relative"
? /Symbol-bearing R_MIPS_REL32/u
: /section relocation declarations/u,
);
assert.deepEqual(await readFile(fixture.path), original);
await client.ping();
reports.push({ ...fixture, status: "refused", ping: true });
continue;
}
await call("open_binary", {
path: fixture.path,
provider_id: "ghidra",
});
const loadedReply = await reply("inspect_native_load_image");
const loaded = requireMcpOperationResult(
loadedReply,
"inspect_native_load_image",
);
const evidence = parseEvidence(
JSON.parse(
loadedReply.content.find((item) => item.type === "text").text,
),
);
const language = `MIPS:${byteOrder === "little" ? "LE" : "BE"}:32:default`;
assert.equal(loaded.observations.language_id, language);
assert.equal(loaded.observations.compiler_spec_id, "default");
assert(
loaded.observations.source_files.some(
(file) => file.original_sha256 === fixture.sha256,
),
);
assert(
evidence.limitations.some((item) =>
item.includes("External GOT bindings"),
),
);
if (variant === "recordless")
assert(
evidence.limitations.some((item) =>
item.includes("does not establish the floating-point ABI"),
),
);
const base = BigInt(loaded.observations.image_base);
const address = (source, clearTag = false) =>
`0x${(base + (clearTag ? BigInt(source) & ~1n : BigInt(source))).toString(16)}`;
const compressed = address(fixture.symbols.rea_mips16_probe, true);
const standard = address(fixture.symbols.rea_mips32_probe);
const procedures = await call("list_procedures");
assert(
procedures.some(
(item) =>
item.address === compressed && item.value === "rea_mips16_probe",
),
);
assert(
procedures.some(
(item) =>
item.address === standard && item.value === "rea_mips32_probe",
),
);
let firstInstruction;
const expected =
byteOrder === "little"
? ["5a6a", "8eea", "074a", "a0e8"]
: ["6a5a", "ea8e", "4a07", "e8a0"];
for (let i = 0; i < 4; i += 1) {
const instruction = await call("inspect_native_instruction", {
address: `0x${(BigInt(compressed) + BigInt(2 * i)).toString(16)}`,
});
assert.equal(instruction.status, "decoded");
assert.equal(instruction.mode, "default");
assert.equal(instruction.length, 2);
assert.equal(instruction.bytes, expected[i]);
assert.deepEqual(instruction.decoder_context, [
{
register: "ISA_MODE",
value: "1",
source: "ghidra-program-context",
},
]);
if (i === 0) firstInstruction = instruction;
}
const standardInstruction = await call("inspect_native_instruction", {
address: standard,
});
assert.equal(standardInstruction.length, 4);
assert.equal(
standardInstruction.bytes,
byteOrder === "little" ? "03008224" : "24820003",
);
assert.equal(
standardInstruction.decoder_context[0].register,
"ISA_MODE",
);
assert(
[null, "0"].includes(standardInstruction.decoder_context[0].value),
);
const pointer = await call("read_bytes", {
address: address(fixture.symbols.rea_mips16_pointer),
length: 4,
});
const pointerBytes = Buffer.from(pointer.bytes_hex, "hex");
const pointerValue =
byteOrder === "little"
? pointerBytes.readUInt32LE(0)
: pointerBytes.readUInt32BE(0);
assert.equal(
BigInt(pointerValue),
base + BigInt(fixture.symbols.rea_mips16_probe),
);
const dossier = await call("analyze_function", {
procedure: compressed,
});
assert.match(dossier.pseudocode, /\^\s*0x5a/u);
assert.match(dossier.pseudocode, /\+\s*(?:7|0x7)\b/u);
const standardDossier = await call("analyze_function", {
procedure: standard,
});
assert.match(standardDossier.pseudocode, /\+\s*(?:3|0x3)\b/u);
const snapshotPath = join(
workspace,
`${byteOrder}-${variant}.snapshot.json`,
);
await call("close_binary", { snapshot_path: snapshotPath });
const snapshot = parseAnalysisSnapshot(
JSON.parse(await readFile(snapshotPath, "utf8")),
);
assert.equal(
snapshot.binding.analysis_profile.parameters.language_selection,
"provider-pinned-v1",
);
assert.equal(
snapshot.binding.analysis_profile.parameters.language_id,
language,
);
await call("open_binary", {
path: fixture.path,
provider_id: "ghidra",
snapshot_path: snapshotPath,
});
assert.deepEqual(
await call("inspect_native_instruction", { address: compressed }),
firstInstruction,
);
await call("close_binary");
await client.ping();
const cli = await execute(
process.execPath,
[
"scripts/rea.mjs",
"inspect-native-instruction",
fixture.path,
compressed,
"--provider",
"ghidra",
"--format",
"json",
],
{ env, timeout: 180000, maxBuffer: 8 * 1024 * 1024 },
);
assert.deepEqual(
parseEvidence(JSON.parse(cli.stdout)).normalized_result,
firstInstruction,
);
assert.deepEqual(await readFile(fixture.path), original);
reports.push({
...fixture,
status: "passed",
language_id: language,
mixed_mode: true,
standard_decoder_context: standardInstruction.decoder_context,
pointer_tag_preserved: true,
snapshot_context_preserved: true,
cli_mcp_parity: true,
source_unchanged: true,
ping: true,
});
} finally {
await client.close();
await transport.close();
await writeFile(
join(workspace, `${byteOrder}-${variant}.stderr`),
stderr,
);
}
}
}
} catch (cause) {
failure = cause;
}
const completed = await completeVerifierRun(run);
if (
completed.process_lineage.status !== "verified" ||
completed.process_lineage.descendants.length !== 0
)
throw new Error(`Cleanup unverified; retained workspace: ${workspace}`, {
cause: failure,
});
if (failure !== undefined)
throw new Error(
`MIPS16 shared verification failed; retained workspace: ${workspace}`,
{ cause: failure },
);
await rm(workspace, { recursive: true, force: true });
console.log(JSON.stringify({ ...completed, fixtures: reports }, null, 2));
scripts/verify-real-hopper.mjs
+5 -2
View File
@@ -30,6 +30,7 @@ import {
requireMcpOperationResult,
requireMcpLifecycleResult,
requireMcpEvidenceResult,
sortVerifierNames,
requireWorkflowEvidenceProvider,
} from "./lib/mcp-verifier-results.mjs";
import {
@@ -283,8 +284,10 @@ try {
const availableTools = initialStatus.tool_availability;
if (!Array.isArray(availableTools))
throw new Error("The real session omitted tool availability");
const expectedNames = availableTools.map(({ name }) => name).sort();
const actualNames = listed.tools.map(({ name }) => name).sort();
const expectedNames = sortVerifierNames(
availableTools.map(({ name }) => name),
);
const actualNames = sortVerifierNames(listed.tools.map(({ name }) => name));
if (JSON.stringify(actualNames) !== JSON.stringify(expectedNames)) {
const expected = new Set(expectedNames);
const actual = new Set(actualNames);
scripts/verify-wasm-artifact.mjs
+1
View File
@@ -247,6 +247,7 @@ try {
'"test\nline"',
'"fake"\nSection Details:\n Type start=0x0000000a end=0x0000000b (size=0x00000001) count: 0\n"end"',
'"unicode-é\\\t"',
'"visible"',
],
);
if (debugNames) {
scripts/verify-web-network-captures.mjs
+24
View File
@@ -913,6 +913,30 @@ async function inspect(mode, input, errorCategory) {
}
assert.notEqual(result.isError, true, mcpTextValue(result));
assertSensitiveLimitations(value, input.sensitive_values ?? []);
if (input.format === "har") {
const surface = await client.callTool({
name: "inspect_analysis_view",
arguments: {
source: { kind: "retained-evidence", evidence_id: value.evidence_id },
view: { kind: "api-surface" },
},
});
if ((input.sensitive_values ?? []).length > 0) {
// Derived view text cannot be checked against undisclosed declarations.
assert.equal(surface.isError, true);
const refused = JSON.parse(mcpTextValue(surface));
for (const literal of input.sensitive_values)
assert.ok(!JSON.stringify(refused).includes(literal));
} else {
assert.notEqual(surface.isError, true, mcpTextValue(surface));
const projected = JSON.parse(mcpTextValue(surface));
assert.equal(projected.normalized_result.kind, "api-surface");
assert.equal(
projected.normalized_result.records.examined,
value.normalized_result.records.length,
);
}
}
return value.normalized_result;
}
const args = [
scripts/verify/android/failure-recovery.mjs
+14 -9
View File
@@ -1,4 +1,8 @@
import { requireMcpToolError } from "../../lib/mcp-verifier-results.mjs";
import {
requireMcpLifecycleResult,
requireMcpToolError,
sortVerifierNames,
} from "../../lib/mcp-verifier-results.mjs";
import assert from "node:assert/strict";
import {
access,
@@ -273,22 +277,23 @@ const verifyAndroidReadiness = async ({
try {
await client.connect(transport);
const catalog = await client.listTools();
const catalogNames = catalog.tools.map(({ name }) => name).sort();
const status = await client.callTool({
const catalogNames = sortVerifierNames(
catalog.tools.map(({ name }) => name),
);
const statusReply = await client.callTool({
name: "binary_session",
arguments: {},
});
assert.notEqual(status.isError, true, JSON.stringify(status));
const availability = status.structuredContent?.result?.tool_availability;
assert.ok(Array.isArray(availability));
const status = requireMcpLifecycleResult(statusReply, "binary_session");
const availability = status.tool_availability;
assert.deepEqual(
availability.map(({ name }) => name).sort(),
TOOL_CONTRACTS.map(({ name }) => name).sort(),
sortVerifierNames(availability.map(({ name }) => name)),
sortVerifierNames(TOOL_CONTRACTS.map(({ name }) => name)),
"Readiness must retain one availability record for the complete canonical catalog",
);
assert.deepEqual(
catalogNames,
TOOL_CONTRACTS.map(({ name }) => name).sort(),
sortVerifierNames(TOOL_CONTRACTS.map(({ name }) => name)),
"Readiness must not truncate the SDK tool catalog",
);
const byName = new Map(availability.map((item) => [item.name, item]));
scripts/verify/apple/legacy-simulator.mjs
+1 -2
View File
@@ -1,7 +1,6 @@
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { promisify } from "node:util";
scripts/verify/architecture-guards.mjs
-1
View File
@@ -1,4 +1,3 @@
import assert from "node:assert/strict";
import { readFile, readdir } from "node:fs/promises";
import { join, relative, sep } from "node:path";
import { fileURLToPath } from "node:url";
scripts/verify/firmware/analysis.mjs
+3 -1
View File
@@ -100,7 +100,9 @@ const verifyRegions = (evidence) => {
);
assert.equal(gzip.reported_size, oracle.length);
assert.equal(gzip.size_basis, "provider_reported_validation_unknown");
return result.regions.map(({ provider_id, ...region }) => region);
return result.regions.map(
({ provider_id: _provider_id, ...region }) => region,
);
};
const verifyFiles = async (evidence) => {
const result = firmwareResultSchemas.extract_firmware.parse(
src/adb/AdbDeviceOutput.test.ts
+31
View File
@@ -239,4 +239,35 @@ describe("parsePmPathOutput", () => {
role: "unknown",
});
});
it("preserves whitespace and the exact POSIX device path", () => {
const devicePath = "/data/app/My App/résumé:final\\draft*.apk";
const { paths, rejectedPaths, unparsedLines } = parsePmPathOutput(
`package:${devicePath}\r\n`,
);
expect(paths).toEqual([
{
device_path: devicePath,
file_name: "résumé:final\\draft*.apk",
role: "unknown",
},
]);
expect(rejectedPaths).toEqual([]);
expect(unparsedLines).toEqual([]);
});
it("rejects an empty final component without normalizing the source", () => {
const devicePath = "/data/app/pkg/base.apk/";
const parsed = parsePmPathOutput(`package:${devicePath}\n`);
expect(parsed.paths).toEqual([]);
expect(parsed.rejectedPaths).toEqual([
{ device_path: devicePath, reason: "empty_file_name" },
]);
});
it("retains malformed lines including surrounding whitespace", () => {
expect(
parsePmPathOutput(" package:/data/app/pkg/base.apk \n").unparsedLines,
).toEqual([" package:/data/app/pkg/base.apk "]);
});
});
src/adb/AdbDeviceOutput.ts
+26 -11
View File
@@ -6,6 +6,11 @@
* their evidence basis.
*/
import {
adbHostFileName,
type AdbHostFileNameReason,
} from "./AdbHostFileName.js";
/** One device exactly as `adb devices -l` reports it. */
export interface AdbDeviceEntry {
readonly serial: string;
@@ -205,11 +210,6 @@ export interface AdbPackagePathEntry {
readonly role: "base" | "split" | "unknown";
}
const fileNameOf = (devicePath: string): string => {
const parts = devicePath.split(/\//u);
return parts[parts.length - 1] ?? devicePath;
};
/**
* Parse `pm path PACKAGE` lines. The base/split role is derived from the file
* name (`base.apk`, `split_*.apk`), which is the only signal the output
@@ -219,20 +219,35 @@ export const parsePmPathOutput = (
stdout: string,
): {
paths: readonly AdbPackagePathEntry[];
rejectedPaths: readonly {
device_path: string;
reason: AdbHostFileNameReason;
}[];
unparsedLines: readonly string[];
} => {
const paths: AdbPackagePathEntry[] = [];
const rejectedPaths: {
device_path: string;
reason: AdbHostFileNameReason;
}[] = [];
const unparsedLines: string[] = [];
for (const rawLine of stdout.split(/\r?\n/u)) {
const line = rawLine.trim();
if (line === "") continue;
const match = /^package:(\S+)$/u.exec(line);
if (rawLine === "") continue;
const match = /^package:(.*)$/u.exec(rawLine);
if (match === null) {
unparsedLines.push(line);
unparsedLines.push(rawLine);
continue;
}
const devicePath = match[1]!;
const fileName = fileNameOf(devicePath);
const validated = adbHostFileName(devicePath);
if (!validated.ok) {
rejectedPaths.push({
device_path: devicePath,
reason: validated.reason,
});
continue;
}
const fileName = validated.fileName;
paths.push({
device_path: devicePath,
file_name: fileName,
@@ -244,5 +259,5 @@ export const parsePmPathOutput = (
: "unknown",
});
}
return { paths, unparsedLines };
return { paths, rejectedPaths, unparsedLines };
};
src/adb/AdbHostFileName.test.ts
+30 -8
View File
@@ -14,28 +14,50 @@ describe("adbHostFileName", () => {
],
["/data/local/tmp/résumé.xml", "résumé.xml"],
["/data/local/tmp/a-b_c.d", "a-b_c.d"],
["/data/local/tmp/", "tmp"],
])("accepts the portable device path %s", (devicePath, fileName) => {
["/data/local/tmp/a\\b", "a\\b"],
["/data/local/tmp/report:2024", "report:2024"],
["/data/local/tmp/a*b.txt", "a*b.txt"],
["/data/local/tmp/re\u007fport", "re\u007fport"],
])("accepts the POSIX device path %s", (devicePath, fileName) => {
expect(adbHostFileName(devicePath)).toEqual({ ok: true, fileName });
});
it.each([
["/", "empty_file_name"],
["data/local/tmp/file", "not_absolute_path"],
["/data/local/tmp/", "empty_file_name"],
["/data/local/tmp/..", "dot_segment"],
["/data/local/tmp/.", "dot_segment"],
["/data/local/tmp/a\\b", "path_separator"],
["/data/local/tmp/re\u007fport", "control_character"],
["/data/local/tmp/ta\tb", "control_character"],
["/data\0/local/tmp/file", "unsupported_character"],
["/data/local/tmp/a\0b", "unsupported_character"],
] as const)("refuses %s as %s", (devicePath, reason) => {
expect(adbHostFileName(devicePath)).toEqual({ ok: false, reason });
});
it.each([
["/data/local/tmp/a\\b", "unsupported_character"],
["/data/local/tmp/ta\tb", "unsupported_character"],
["/data/local/tmp/CON", "reserved_name"],
["/data/local/tmp/nul.txt", "reserved_name"],
["/data/local/tmp/com1", "reserved_name"],
["/data/local/tmp/LPT9", "reserved_name"],
["/data/local/tmp/report:2024", "alternate_data_stream"],
["/data/local/tmp/COM¹.txt", "reserved_name"],
["/data/local/tmp/lpt³", "reserved_name"],
["/data/local/tmp/report:2024", "unsupported_character"],
["/data/local/tmp/report.", "trailing_dot_or_space"],
["/data/local/tmp/report ", "trailing_dot_or_space"],
["/data/local/tmp/a*b.txt", "unsupported_character"],
["/data/local/tmp/a?b.txt", "unsupported_character"],
] as const)("refuses %s as %s", (devicePath, reason) => {
expect(adbHostFileName(devicePath)).toEqual({ ok: false, reason });
] as const)("rejects %s on Windows as %s", (devicePath, reason) => {
expect(adbHostFileName(devicePath, "win32")).toEqual({ ok: false, reason });
});
it("does not impose non-Windows control restrictions on Windows filenames", () => {
expect(adbHostFileName("/data/local/tmp/name\u007f.txt", "win32").ok).toBe(
true,
);
expect(adbHostFileName("/data/local/tmp/name\u0085.txt", "win32").ok).toBe(
true,
);
});
});
src/adb/AdbHostFileName.ts
+34 -42
View File
@@ -1,61 +1,53 @@
import { posix } from "node:path";
/** Why a device-reported path cannot be used as a host file name. */
export type AdbHostFileNameReason =
| "empty_file_name"
| "not_absolute_path"
| "dot_segment"
| "path_separator"
| "control_character"
| "unsupported_character"
| "reserved_name"
| "trailing_dot_or_space"
| "alternate_data_stream";
| "trailing_dot_or_space";
/** A portable host file name, or the exact reason the device name was refused. */
export type AdbHostFileNameResult =
| { readonly ok: true; readonly fileName: string }
| { readonly ok: false; readonly reason: AdbHostFileNameReason };
/**
* Win32 resolves these basenames to a device regardless of case and of any
* extension (`NUL`, `nul.txt`), so a portable host file name must avoid them on
* every host. `(?:\.|$)` keeps ordinary names such as `console.txt` admissible.
*/
const WINDOWS_RESERVED_BASENAME =
/^(?:con|prn|aux|nul|com[1-9]|lpt[1-9])(?:\.|$)/iu;
/** Characters that are not portable in a host file name on Windows or POSIX. */
const UNSUPPORTED_CHARACTER = /[*?"<>|]/u;
/** Control characters (`Cc`), including C0 controls and DEL. */
const CONTROL_CHARACTER = /\p{Cc}/u;
/^(?:con|prn|aux|nul|com[1-9¹²³]|lpt[1-9¹²³])(?:\.|$)/iu;
/**
* Derive a portable host file name from a device-reported path.
*
* The device path is a POSIX path on the device filesystem, so its last
* `/`-separated segment is taken with POSIX semantics rather than the host
* `basename` (which would vary between POSIX and Windows hosts). A name is
* accepted only when every host can store it and address it back unambiguously:
* no separator, control, or reserved-device characters, no trailing dot or
* space, no `:` alternate-data-stream marker, and no `.`/`..` segment.
* Derive the final component of a device POSIX path without normalizing the
* source. A trailing slash has no filename and must not silently become the
* preceding component as `basename` would make it.
*/
export const adbHostFileName = (devicePath: string): AdbHostFileNameResult => {
const fileName = posix.basename(devicePath);
if (fileName === "") return { ok: false, reason: "empty_file_name" };
export const adbHostFileName = (
devicePath: string,
platform: NodeJS.Platform = process.platform,
): AdbHostFileNameResult => {
if (devicePath === "" || devicePath.endsWith("/"))
return { ok: false, reason: "empty_file_name" };
if (devicePath.includes("\0"))
return { ok: false, reason: "unsupported_character" };
if (!devicePath.startsWith("/"))
return { ok: false, reason: "not_absolute_path" };
const fileName = devicePath.slice(devicePath.lastIndexOf("/") + 1);
if (fileName === "." || fileName === "..")
return { ok: false, reason: "dot_segment" };
if (fileName.includes("\\") || fileName.includes("/"))
return { ok: false, reason: "path_separator" };
if (CONTROL_CHARACTER.test(fileName))
return { ok: false, reason: "control_character" };
if (fileName.includes(":"))
return { ok: false, reason: "alternate_data_stream" };
if (UNSUPPORTED_CHARACTER.test(fileName))
return { ok: false, reason: "unsupported_character" };
if (WINDOWS_RESERVED_BASENAME.test(fileName))
return { ok: false, reason: "reserved_name" };
if (/[. ]$/u.test(fileName))
return { ok: false, reason: "trailing_dot_or_space" };
if (platform === "win32") {
if (/[\\:*?"<>|]/u.test(fileName) || hasWindowsControl(fileName))
return { ok: false, reason: "unsupported_character" };
if (WINDOWS_RESERVED_BASENAME.test(fileName))
return { ok: false, reason: "reserved_name" };
if (/[. ]$/u.test(fileName))
return { ok: false, reason: "trailing_dot_or_space" };
}
return { ok: true, fileName };
};
const hasWindowsControl = (fileName: string): boolean => {
for (const character of fileName) {
const code = character.codePointAt(0)!;
if (code > 0 && code < 0x20) return true;
}
return false;
};
src/adb/AdbProvider.test.ts
+111 -9
View File
@@ -1,6 +1,7 @@
import { createHash } from "node:crypto";
import {
chmod,
lstat,
mkdir,
mkdtemp,
readFile,
@@ -52,6 +53,28 @@ const refusedBy = async (
return projectAnalysisError(result.error);
};
const namesAliasOnVolume = async (
parent: string,
firstName: string,
secondName: string,
): Promise<boolean> => {
const probe = join(parent, "destination-name-probe");
await mkdir(probe);
try {
await mkdir(join(probe, firstName));
try {
await mkdir(join(probe, secondName));
return false;
} catch (cause) {
if (cause instanceof Error && "code" in cause && cause.code === "EEXIST")
return true;
throw cause;
}
} finally {
await rm(probe, { recursive: true, force: true });
}
};
it("inspects the adb client without contacting the server", async () => {
const { binary } = await writeAdbStub(
() => `if [ "$1" = "version" ]; then ${VERSION_BLOCK}; fi`,
@@ -283,6 +306,82 @@ it("refuses to reuse an existing package output directory", async () => {
expect(result.error.message).toContain("already exists");
});
it.each([
["an empty filename", "package:/data/app/x/"],
[
"duplicate local filenames",
"package:/data/app/x/base.apk\npackage:/data/app/y/base.apk",
],
])("refuses %s before creating output or pulling", async (_, pmOutput) => {
const { binary, root } = await writeAdbStub(
(root) => `if [ "$1" = "version" ]; then ${VERSION_BLOCK}; exit 0; fi
if [ "$1" = "-s" ] && [ "$4" = "pm" ]; then printf '%b' '${pmOutput.replaceAll("'", "'\\''").replaceAll("\n", "\\n")}\\n'; exit 0; fi
if [ "$1" = "-s" ] && [ "$3" = "pull" ]; then : > '${join(root, "pull-ran")}'; exit 0; fi
exit 1`,
);
const output = join(root, "pulls");
const result = await providerOver(binary).execute({
operation: "pull_adb_package",
input: {
serial: "emulator-5554",
package: "com.example",
output_directory: output,
},
});
expect(result.ok).toBe(false);
if (result.ok) return;
expect(result.error.capturedOutput?.stdout).toContain(
pmOutput.split("\n")[0],
);
await expect(readFile(join(root, "pull-ran"))).rejects.toThrow();
await expect(lstat(output)).rejects.toMatchObject({ code: "ENOENT" });
});
it.each([
["case-only names", "base.apk", "BASE.APK"],
["normalization-equivalent names", "caf\u00e9.apk", "cafe\u0301.apk"],
])(
"checks %s against the actual destination filesystem before pulling",
async (_, firstName, secondName) => {
const { binary, root } = await writeAdbStub((root) =>
pullStub(
`'package:/data/app/x/${firstName}' 'package:/data/app/y/${secondName}'`,
`printf 'pulled\\n' >> '${join(root, "pull.log")}'
printf 'apk' > "$5"`,
),
);
const aliases = await namesAliasOnVolume(root, firstName, secondName);
const result = await providerOver(binary).execute({
operation: "pull_adb_package",
input: {
serial: "emulator-5554",
package: "com.example",
output_directory: root,
},
});
const pullLog = join(root, "pull.log");
const packageDirectory = join(root, "com.example");
if (aliases) {
expect(result.ok).toBe(false);
if (result.ok) return;
expect(result.error.message).toContain("unique local destinations");
await expect(readFile(pullLog)).rejects.toThrow();
await expect(lstat(packageDirectory)).rejects.toThrow();
return;
}
expect(result.ok).toBe(true);
if (!result.ok) throw new Error(result.error.message);
expect(
(
result.value.result as { artifacts: { file_name: string }[] }
).artifacts.map((artifact) => artifact.file_name),
).toEqual([firstName, secondName]);
expect((await readFile(pullLog, "utf8")).trim().split("\n")).toHaveLength(
2,
);
},
);
it("reports an uninstalled package from empty pm path output", async () => {
const { binary } = await writeAdbStub(
() => `if [ "$1" = "version" ]; then ${VERSION_BLOCK}; exit 0; fi
@@ -465,14 +564,14 @@ exit 1`,
operation: "pull_adb_file",
input: {
serial: "emulator-5554",
device_path: "/data/local/tmp/notes.txt",
device_path: "/data/local/tmp/My Notes:r*sumé.txt",
output_directory: output,
},
});
const digest = createHash("sha256").update("file-bytes").digest("hex");
expect(result).toMatchObject({
device_path: "/data/local/tmp/notes.txt",
local_path: join(output, "notes.txt"),
device_path: "/data/local/tmp/My Notes:r*sumé.txt",
local_path: join(output, "My Notes:r*sumé.txt"),
bytes: 10,
sha256: digest,
});
@@ -480,10 +579,7 @@ exit 1`,
it.each([
["/data/local/tmp/..", "dot_segment"],
["/data/local/tmp/a\\b", "path_separator"],
["/data/local/tmp/CON", "reserved_name"],
["/data/local/tmp/report.", "trailing_dot_or_space"],
["/data/local/tmp/report:2024", "alternate_data_stream"],
["/data/local/tmp/", "empty_file_name"],
] as const)(
"refuses the unsafe device path %s before pulling",
async (devicePath, reason) => {
@@ -510,13 +606,19 @@ exit 0`,
{
path: ["device_path"],
reason: "invalid_value",
message: expect.stringContaining(reason),
message: expect.stringContaining(
`${JSON.stringify(devicePath)} does not name`,
),
},
],
});
expect(
(projected.details as { issues: { message: string }[] }).issues[0]!
.message,
).toContain(reason);
// No host directory and no adb invocation are effects of a refused pull.
await expect(readFile(join(root, "pull-ran"))).rejects.toThrow();
await expect(readFile(output)).rejects.toThrow();
await expect(lstat(output)).rejects.toMatchObject({ code: "ENOENT" });
},
);
src/adb/AdbProvider.ts
+136 -41
View File
@@ -1,7 +1,15 @@
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import { constants as fsConstants } from "node:fs";
import { access, lstat, mkdir, rm, stat, writeFile } from "node:fs/promises";
import {
access,
lstat,
mkdir,
rmdir,
rm,
stat,
writeFile,
} from "node:fs/promises";
import { join, resolve } from "node:path";
import type {
@@ -72,8 +80,6 @@ export const ADB_PROVIDER_IDENTITY = {
version: null,
} as const;
const VERSION_TIMEOUT_MS = 10_000;
const VERSION_MAX_OUTPUT_BYTES = 64 * 1024;
const DEVICES_TIMEOUT_MS = 20_000;
const DEVICES_MAX_OUTPUT_BYTES = 1024 * 1024;
const SHELL_TIMEOUT_MS = 30_000;
@@ -388,6 +394,68 @@ const removeOwnedPackageDirectory = async (
await rm(path, { recursive: true, force: true });
};
const reservePackageDestinations = async (
operation: string,
packageDirectory: string,
entries: readonly { readonly file_name: string }[],
capturedOutput: {
readonly stdout: string;
readonly stderr: string;
readonly truncated: boolean;
},
): Promise<AnalysisOutputError | undefined> => {
const reservations: {
readonly path: string;
readonly identity: PackageDirectoryIdentity;
}[] = [];
for (const entry of entries) {
const path = join(packageDirectory, entry.file_name);
try {
await mkdir(path);
const created = await lstat(path);
if (!created.isDirectory() || created.isSymbolicLink())
throw new Error(`Could not reserve local filename ${entry.file_name}`);
reservations.push({
path,
identity: {
dev: created.dev,
ino: created.ino,
mode: created.mode,
},
});
} catch (cause: unknown) {
return new AnalysisOutputError(
operation,
`Package Manager paths cannot be written to unique local destinations at ${JSON.stringify(entry.file_name)}; refusing before pulling any files`,
{ cause, capturedOutput },
);
}
}
for (const reservation of reservations) {
try {
const current = await lstat(reservation.path);
if (
!current.isDirectory() ||
current.isSymbolicLink() ||
current.dev !== reservation.identity.dev ||
current.ino !== reservation.identity.ino ||
current.mode !== reservation.identity.mode
)
throw new Error(
`The local filename reservation changed before release: ${reservation.path}`,
);
await rmdir(reservation.path);
} catch (cause: unknown) {
return new AnalysisOutputError(
operation,
`Could not release a local filename reservation before pulling: ${reservation.path}`,
{ cause, capturedOutput },
);
}
}
return undefined;
};
/**
* ADB device inspection and acquisition over a caller-selected adb binary.
*
@@ -1156,6 +1224,19 @@ export class AdbProvider {
);
if (!paths.ok) return paths;
const parsed = parsePmPathOutput(paths.value.stdout);
const capturedOutput = {
stdout: paths.value.stdout,
stderr: paths.value.stderr,
truncated: false,
};
if (parsed.rejectedPaths.length > 0)
return err(
new AnalysisOutputError(
operation,
`Package Manager reported paths that cannot be represented as host files: ${parsed.rejectedPaths.map((entry) => `${JSON.stringify(entry.device_path)} (${entry.reason})`).join(", ")}`,
{ capturedOutput },
),
);
if (parsed.paths.length === 0 && parsed.unparsedLines.length === 0)
return err(
new AnalysisUnsupportedTargetError(
@@ -1168,8 +1249,21 @@ export class AdbProvider {
return err(
new AnalysisProtocolError(
`pm path reported an unrecognized line: ${parsed.unparsedLines[0]}`,
{ capturedOutput },
),
);
const destinationNames = new Set<string>();
for (const entry of parsed.paths) {
if (destinationNames.has(entry.file_name))
return err(
new AnalysisOutputError(
operation,
`Package Manager reported multiple paths with the same local destination ${JSON.stringify(entry.file_name)}; refusing before pulling any files`,
{ capturedOutput },
),
);
destinationNames.add(entry.file_name);
}
const directory = await prepareOutputDirectory(
operation,
input.output_directory,
@@ -1228,6 +1322,43 @@ export class AdbProvider {
);
return err(analysisErrorWithCleanupFailure(primary, cleanup, operation));
}
const failOwnedPackage = async (
primary: AnalysisError,
): Promise<Result<AnalysisExecution, AnalysisError>> => {
const owner: PendingPackageDirectoryCleanup = {
path: packageDirectory,
identity: packageDirectoryIdentity,
primary,
remove: () =>
removeOwnedPackageDirectory(
packageDirectory,
packageDirectoryIdentity,
),
};
try {
await owner.remove();
} catch (cause: unknown) {
this.#pendingPackageDirectoryCleanup.add(owner);
const cleanup = new ProviderCleanupError(
ADB_PROVIDER_IDENTITY.id,
[packageDirectory],
{ reason: cause instanceof Error ? cause.message : String(cause) },
{ operation, cause },
);
return err(
analysisErrorWithCleanupFailure(primary, cleanup, operation),
);
}
return err(primary);
};
const destinationError = await reservePackageDestinations(
operation,
packageDirectory,
parsed.paths,
capturedOutput,
);
if (destinationError !== undefined)
return failOwnedPackage(destinationError);
const artifacts: {
device_path: string;
file_name: string;
@@ -1273,17 +1404,7 @@ export class AdbProvider {
});
return err(error);
};
const seenFileNames = new Set<string>();
for (const entry of parsed.paths) {
if (seenFileNames.has(entry.file_name)) {
failures.push({
device_path: entry.device_path,
stage: "pull",
message: `Duplicate file name ${entry.file_name} in the APK set; refusing to overwrite`,
});
continue;
}
seenFileNames.add(entry.file_name);
const localPath = join(packageDirectory, entry.file_name);
const pulled = await runAdb(
operation,
@@ -1338,33 +1459,7 @@ export class AdbProvider {
const primary = new AnalysisProtocolError(
`Every APK pull failed for ${packageName} on ${serial}: ${failures[0]!.message}`,
);
const owner: PendingPackageDirectoryCleanup = {
path: packageDirectory,
identity: packageDirectoryIdentity,
primary,
remove: () =>
removeOwnedPackageDirectory(
packageDirectory,
packageDirectoryIdentity,
),
};
try {
await owner.remove();
} catch (cause: unknown) {
this.#pendingPackageDirectoryCleanup.add(owner);
const cleanup = new ProviderCleanupError(
ADB_PROVIDER_IDENTITY.id,
[packageDirectory],
{
reason: cause instanceof Error ? cause.message : String(cause),
},
{ operation: operation, cause },
);
return err(
analysisErrorWithCleanupFailure(primary, cleanup, operation),
);
}
return err(primary);
return failOwnedPackage(primary);
}
return ok(
createAnalysisExecution(
@@ -1407,7 +1502,7 @@ export class AdbProvider {
{
path: ["device_path"],
reason: "invalid_value",
message: `The device path does not name a portable host file (${derived.reason}); REA never renames a device-reported file.`,
message: `The device path ${JSON.stringify(input.device_path)} does not name a file that can be written on this host (${derived.reason}); REA never renames a device-reported file.`,
},
]),
);
src/android/JadxAnalysis.ts
+13 -3
View File
@@ -167,16 +167,26 @@ const searchClasses = async (
"JADX class pagination is incomplete",
);
} while (classes.length < total);
if (new Set(classes).size !== classes.length || total !== loaded.class_count)
if (total !== loaded.class_count)
throw new AnalysisOutputError(
request.operation,
"JADX class inventory is duplicated or differs from the loaded class count",
"JADX class inventory count differs from the loaded class count",
);
const occurrences = new Map<string, number>();
for (const name of classes)
occurrences.set(name, (occurrences.get(name) ?? 0) + 1);
const matchingNames = classes.filter((name) =>
name.includes(request.input.query),
);
const duplicateNames = [...occurrences]
.filter(([name, count]) => count > 1 && name.includes(request.input.query))
.map(([name, count]) => ({ name, occurrences: count }));
return androidResultSchemas.search_android_classes.parse({
engine,
query: request.input.query,
total_classes: total,
matches: classes.filter((name) => name.includes(request.input.query)),
matches: matchingNames,
duplicate_names: duplicateNames,
coverage: "complete",
});
};
src/apktool/ApktoolDecodeOutput.test.ts
+131 -3
View File
@@ -26,7 +26,6 @@ const REAL_YML = [
" ids:",
" - 1",
" tag: null",
"version: 2.7.0-dirty",
"versionInfo:",
" versionCode: '7'",
" versionName: 1.2.3",
@@ -44,7 +43,11 @@ describe("parseApktoolYml", () => {
});
it("accepts quoted version values some builds emit", () => {
expect(parseApktoolYml("versionName: '2.0.0'\nversionCode: 45\n")).toEqual({
expect(
parseApktoolYml(
'!!brut.androlib.meta.MetaInfo\nversionInfo:\n versionName: "2.0.0"\n versionCode: 45\n',
),
).toEqual({
versionName: "2.0.0",
versionCode: "45",
minSdkVersion: null,
@@ -52,6 +55,49 @@ describe("parseApktoolYml", () => {
});
});
it("reads only the producer's nested fields and preserves scalar spelling", () => {
expect(
parseApktoolYml(
[
"!!brut.androlib.meta.MetaInfo",
"versionName: decoy",
"versionInfo:",
" versionName: 01.2.3",
" versionCode: 045",
" custom:",
" versionName: unrelated",
"sdkInfo:",
" minSdkVersion: 24",
" targetSdkVersion: '35'",
"",
].join("\n"),
),
).toEqual({
versionName: "01.2.3",
versionCode: "045",
minSdkVersion: "24",
targetSdkVersion: "35",
});
});
it("distinguishes YAML nulls from quoted null strings in flow mappings", () => {
expect(
parseApktoolYml(
[
"!!brut.androlib.meta.MetaInfo",
'versionInfo: { versionName: "null", versionCode: 0045, nested: { versionName: decoy } }',
"sdkInfo: { minSdkVersion: null, targetSdkVersion: ~ }",
"",
].join("\n"),
),
).toEqual({
versionName: "null",
versionCode: "0045",
minSdkVersion: null,
targetSdkVersion: null,
});
});
it("reports null facts for unrelated YAML", () => {
expect(parseApktoolYml("version: 2.7.0\n")).toEqual({
versionName: null,
@@ -66,7 +112,15 @@ describe("parseManifestPackage", () => {
it("reads the package attribute from a decoded manifest", () => {
const manifest =
'<?xml version="1.0" encoding="utf-8" standalone="no"?><manifest xmlns:android="http://schemas.android.com/apk/res/android" android:compileSdkVersion="34" package="com.rea.apktool.probe" platformBuildVersionCode="34">';
expect(parseManifestPackage(manifest)).toBe("com.rea.apktool.probe");
expect(parseManifestPackage(`${manifest}</manifest>`)).toBe(
"com.rea.apktool.probe",
);
});
it("reads package with reordered or single-quoted XML attributes", () => {
expect(
parseManifestPackage("<manifest debug='false' package='org.rea.app' />"),
).toBe("org.rea.app");
});
it("reports null when no package attribute exists", () => {
@@ -111,4 +165,78 @@ describe("parseDecodedStrings", () => {
"a &amp; b &lt;c&gt;",
);
});
it("keeps empty, self-closing, and reordered string attributes", () => {
expect(
parseDecodedStrings(
'<resources><string translatable="false" name="empty"/><string name="paired"></string></resources>',
),
).toEqual({
entries: [
{ name: "empty", value: "" },
{ name: "paired", value: "" },
],
unparsedCount: 0,
});
});
it("ignores comments without mistaking them for nested markup", () => {
expect(
parseDecodedStrings(
'<resources><!-- <string name="fake">bad</string> --><string name="real">before<!-- note -->after</string></resources>',
),
).toEqual({
entries: [{ name: "real", value: "beforeafter" }],
unparsedCount: 0,
});
});
it("does not include a trailing document comment in the last string", () => {
expect(
parseDecodedStrings(
'<resources><string name="a">one</string></resources><!-- </string> -->',
),
).toEqual({
entries: [{ name: "a", value: "one" }],
unparsedCount: 0,
});
});
it("preserves source whitespace and entity spelling", () => {
expect(
parseDecodedStrings(
'<resources><string name="raw"> a &amp; b </string></resources>',
).entries,
).toEqual([{ name: "raw", value: " a &amp; b " }]);
});
it("keeps source slices aligned across CRLF and CR line endings", () => {
for (const newline of ["\r\n", "\r"]) {
expect(
parseDecodedStrings(
`<resources>${newline} <string name="raw">x &amp; y</string>${newline}</resources>`,
),
).toEqual({
entries: [{ name: "raw", value: "x &amp; y" }],
unparsedCount: 0,
});
}
});
it("counts a string with no name as unsupported instead of emitting an empty name", () => {
expect(
parseDecodedStrings("<resources><string>value</string></resources>"),
).toEqual({ entries: [], unparsedCount: 1 });
});
it("reports malformed XML and unsupported string markup as partial", () => {
expect(
parseDecodedStrings("<resources><string name='broken'>no</resources>"),
).toEqual({ entries: [], unparsedCount: 1 });
expect(
parseDecodedStrings(
'<resources><string name="styled">Hello <b>world</b></string></resources>',
),
).toEqual({ entries: [], unparsedCount: 1 });
});
});
src/apktool/ApktoolDecodeOutput.ts
+152 -54
View File
@@ -1,11 +1,17 @@
/**
* Parsers for the decoded workspace `apktool d --no-src` produces.
*
* Apktool's apktool.yml is a small YAML document; the facts REA projects
* are flat enough that line-oriented extraction stays honest without a
* YAML dependency, and unrecognized content is ignored rather than guessed.
*/
import {
DOMParser,
type Document,
type Element,
type Node,
} from "@xmldom/xmldom";
import {
isMap,
isScalar,
parseDocument,
type Document as YamlDocument,
} from "yaml";
/** Project only the Apktool metadata fields REA publishes. */
export interface ApktoolMetadata {
readonly versionName: string | null;
readonly versionCode: string | null;
@@ -13,43 +19,48 @@ export interface ApktoolMetadata {
readonly targetSdkVersion: string | null;
}
/**
* Project version and SDK facts from apktool.yml. Values keep apktool's
* own spelling (version codes can print as `45` or `45.1`).
*/
export const parseApktoolYml = (yml: string): ApktoolMetadata => {
const metadata: {
versionName: string | null;
versionCode: string | null;
minSdkVersion: string | null;
targetSdkVersion: string | null;
} = {
versionName: null,
versionCode: null,
minSdkVersion: null,
targetSdkVersion: null,
};
for (const line of yml.split(/\r?\n/u)) {
const versionName = /^\s*versionName:\s*(?:'([^']*)'|(.*))$/u.exec(line);
if (versionName !== null && metadata.versionName === null)
metadata.versionName = (versionName[1] ?? versionName[2] ?? "").trim();
const versionCode = /^\s*versionCode:\s*(?:'([^']*)'|(.*))$/u.exec(line);
if (versionCode !== null && metadata.versionCode === null)
metadata.versionCode = (versionCode[1] ?? versionCode[2] ?? "").trim();
const minSdk = /minSdkVersion:\s*'?([0-9]+)'?/u.exec(line);
if (minSdk !== null && metadata.minSdkVersion === null)
metadata.minSdkVersion = minSdk[1]!;
const targetSdk = /targetSdkVersion:\s*'?([0-9]+)'?/u.exec(line);
if (targetSdk !== null && metadata.targetSdkVersion === null)
metadata.targetSdkVersion = targetSdk[1]!;
}
return metadata;
const metaInfoTag = {
tag: "tag:yaml.org,2002:brut.androlib.meta.MetaInfo",
collection: "map" as const,
resolve: (map: unknown) => map,
};
/** Read the package attribute from decoded AndroidManifest.xml text. */
/**
* Read values from Apktool's tagged metadata document. YAML's core schema
* distinguishes null from quoted strings; numeric values use their source
* range so version codes retain the spelling Apktool emitted.
*/
export const parseApktoolYml = (yml: string): ApktoolMetadata => {
const document = parseDocument(yml, {
schema: "core",
customTags: [metaInfoTag],
strict: true,
uniqueKeys: true,
});
if (document.errors.length > 0)
throw new TypeError(
`apktool.yml is not valid Apktool metadata: ${document.errors[0]?.message ?? "YAML parse error"}`,
);
if (!isMap(document.contents))
throw new TypeError("apktool.yml root must be an Apktool metadata mapping");
return {
versionName: yamlScalar(document, ["versionInfo", "versionName"], yml),
versionCode: yamlScalar(document, ["versionInfo", "versionCode"], yml),
minSdkVersion: yamlScalar(document, ["sdkInfo", "minSdkVersion"], yml),
targetSdkVersion: yamlScalar(
document,
["sdkInfo", "targetSdkVersion"],
yml,
),
};
};
/** Read the package attribute from the decoded Android manifest root. */
export const parseManifestPackage = (manifest: string): string | null => {
const match = /<manifest[^>]*\spackage="([^"]+)"/u.exec(manifest);
return match === null ? null : match[1]!;
const document = parseXml(manifest, "AndroidManifest.xml");
const root = document.documentElement;
if (root === null || root.nodeName !== "manifest") return null;
return root.getAttribute("package") || null;
};
export interface DecodedStringEntry {
@@ -58,10 +69,10 @@ export interface DecodedStringEntry {
}
/**
* Project `<string name="...">value</string>` entries from one decoded
* strings.xml. Values are preserved exactly as apktool printed them —
* escaped entities stay escaped — and entries containing nested markup
* are counted as unparsed rather than flattened.
* Project direct `<string>` children from decoded resources XML. DOM parsing
* identifies elements and attributes; text values are sliced from the input
* at parser-reported locations so entity spelling and whitespace are kept.
* Markup-bearing string values are omitted and counted as partial evidence.
*/
export const parseDecodedStrings = (
stringsXml: string,
@@ -69,20 +80,107 @@ export const parseDecodedStrings = (
entries: readonly DecodedStringEntry[];
unparsedCount: number;
} => {
let document: Document;
try {
document = parseXml(stringsXml, "strings.xml");
} catch {
return { entries: [], unparsedCount: 1 };
}
const root = document.documentElement;
if (root === null || root.nodeName !== "resources")
return { entries: [], unparsedCount: 1 };
const entries: DecodedStringEntry[] = [];
let unparsedCount = 0;
const pattern = /<string\s+name="([^"]+)"[^>]*>([\s\S]*?)<\/string>/gu;
for (
let match = pattern.exec(stringsXml);
match !== null;
match = pattern.exec(stringsXml)
) {
const value = match[2]!;
if (value.includes("<")) {
const offsets = sourceLineOffsets(stringsXml);
for (const child of childNodes(root)) {
if (child.nodeType !== 1) continue;
const element = child as Element;
if (element.nodeName !== "string") continue;
const name = element.getAttribute("name");
if (name === null || name.length === 0) {
unparsedCount += 1;
continue;
}
entries.push({ name: match[1]!, value: value.trim() });
const content = childNodes(element);
if (content.some((node) => node.nodeType !== 3 && node.nodeType !== 8)) {
unparsedCount += 1;
continue;
}
if (content.every((node) => node.nodeType === 8)) {
entries.push({ name, value: "" });
continue;
}
const value = content
.filter((node) => node.nodeType === 3)
.map((node) => {
const start = sourceOffset(node, offsets);
if (start === undefined) return null;
// A literal '<' cannot appear in well-formed XML text. Ending here
// also excludes comments after the document element from this value.
const end = stringsXml.indexOf("<", start);
if (end < start) return null;
return stringsXml.slice(start, end);
});
if (value.some((part) => part === null)) {
unparsedCount += 1;
continue;
}
entries.push({ name, value: value.join("") });
}
return { entries, unparsedCount };
};
const yamlScalar = (
document: YamlDocument,
path: readonly string[],
source: string,
): string | null => {
const node = document.getIn(path, true);
if (!isScalar(node) || node.value === null) return null;
if (typeof node.value === "string") return node.value;
const range = node.range;
return range === null || range === undefined
? String(node.value)
: source.slice(range[0], range[1]);
};
const parseXml = (xml: string, label: string) => {
const errors: string[] = [];
const document = new DOMParser({
normalizeLineEndings: (source) => source,
onError: (_level, message) => errors.push(message),
}).parseFromString(xml, "application/xml");
if (errors.length > 0 || document.documentElement === null)
throw new TypeError(
`${label} is not well-formed XML${errors[0] === undefined ? "" : `: ${errors[0]}`}`,
);
return document;
};
const childNodes = (node: Node): Node[] => {
const children: Node[] = [];
for (let child = node.firstChild; child !== null; child = child.nextSibling)
children.push(child);
return children;
};
const sourceLineOffsets = (source: string): readonly number[] => {
const offsets = [0];
for (const match of source.matchAll(/\r\n?|\n/gu)) {
const index = match.index;
offsets.push(index + match[0].length);
}
return offsets;
};
const sourceOffset = (
node: Node,
lineOffsets: readonly number[],
): number | undefined => {
const line = node.lineNumber;
const column = node.columnNumber;
if (line === undefined || column === undefined) return undefined;
const lineStart = lineOffsets[line - 1];
return lineStart === undefined ? undefined : lineStart + column - 1;
};
src/apktool/ApktoolProvider.test.ts
+19 -1
View File
@@ -89,10 +89,12 @@ const decodeStub = (
readonly withStrings?: boolean;
readonly withGerman?: boolean;
readonly failDecode?: boolean;
readonly stringsXml?: string;
} = {},
): ((root: string) => string) => {
const withStrings = options.withStrings ?? true;
const withGerman = options.withGerman ?? true;
const stringsXml = options.stringsXml ?? REAL_STRINGS;
return () => `if [ "$1" = "--version" ]; then ${VERSION_BLOCK}; exit 0; fi
if [ "$1" = "d" ]; then
${
@@ -101,7 +103,7 @@ if [ "$1" = "d" ]; then
: `printf '%s\\n' '${REAL_YML.replace(/'/gu, "'\\''")}' > "$5/apktool.yml"
printf '%s\\n' '${REAL_MANIFEST.replace(/'/gu, "'\\''")}' > "$5/AndroidManifest.xml"
mkdir -p "$5/res/values"${withGerman ? ' "$5/res/values-de"' : ""}
${withStrings ? `printf '%s\\n' '${REAL_STRINGS.replace(/'/gu, "'\\''")}' > "$5/res/values/strings.xml"` : ""}
${withStrings ? `printf '%s\\n' '${stringsXml.replace(/'/gu, "'\\''")}' > "$5/res/values/strings.xml"` : ""}
${withGerman ? `printf '%s\\n' '${DE_STRINGS.replace(/'/gu, "'\\''")}' > "$5/res/values-de/strings.xml"` : ""}
printf 'I: Using Apktool on stub\\n'
exit 0`
@@ -173,6 +175,22 @@ it("decodes resources with target identity, metadata, and strings", async () =>
expect(result.coverage).toBe("complete");
});
it("preserves empty resource strings and reports markup omissions as partial", async () => {
const stringsXml =
'<resources><string name="empty"/><string name="raw"> A &amp; B </string><string name="styled">Hello <b>world</b></string></resources>';
const { command, root } = await writeApktoolStub(decodeStub({ stringsXml }));
const apk = await apkFixture(root);
const result = await evidenceResult(providerOver(command), {
operation: "decode_android_resources",
input: { path: apk.path, include_strings: true },
});
expect(result.strings).toEqual([
{ name: "empty", value: "" },
{ name: "raw", value: " A &amp; B " },
]);
expect(result.coverage).toBe("partial");
});
it("projects one locale's strings on request", async () => {
const { command, root } = await writeApktoolStub(decodeStub());
const apk = await apkFixture(root);
src/apktool/ApktoolProvider.ts
+2 -2
View File
@@ -1,5 +1,5 @@
import { createHash, randomUUID } from "node:crypto";
import { createWriteStream, constants as fsConstants } from "node:fs";
import { createWriteStream } from "node:fs";
import {
access,
chmod,
@@ -717,7 +717,7 @@ export class ApktoolProvider {
: []),
...(parsedStrings.unparsedCount > 0
? [
`${String(parsedStrings.unparsedCount)} string entries with nested markup were not projected`,
`String entries omitted due to malformed XML or unsupported markup: ${String(parsedStrings.unparsedCount)}`,
]
: []),
...(parsedStrings.entries.length > MAX_STRINGS
src/application/AnalysisContextQueries.ts
+1 -2
View File
@@ -3,12 +3,11 @@ import type {
AnalysisOperation,
AnalysisOperationPort,
} from "./AnalysisProvider.js";
import { AnalysisOutputError } from "../domain/analysisErrorCore.js";
import type { AnalysisError } from "../domain/analysisErrorBase.js";
import type { JsonValue } from "../domain/jsonValue.js";
import { analysisBookmarkSchema } from "../domain/hopperValues.js";
import { resolveAnalysisDocument } from "./AnalysisDocument.js";
import { err, ok, type Result } from "../domain/result.js";
import { ok, type Result } from "../domain/result.js";
type Facet =
| { readonly state: "available"; readonly value: JsonValue }
src/application/AnalysisProvider.ts
+14
View File
@@ -15,6 +15,10 @@ import type { ProgressReporter } from "./ProgressReporter.js";
import type { ProviderRejectionCode } from "../contracts/providerSelection.js";
import type { ProcessLineageObservation } from "../process/ProcessOwnership.js";
import type { ProviderOperationHealth } from "../domain/providerOperationHealth.js";
import {
relationshipCoverageSchema,
type RelationshipCoverage,
} from "../domain/relationshipCoverage.js";
export interface ExecutionOptions {
readonly signal?: AbortSignal;
@@ -44,6 +48,8 @@ export interface AnalysisExecution {
readonly provider: ProviderIdentity;
readonly analysisProfile?: AnalysisProfileCommitment;
readonly limitations: readonly string[];
/** Provider-owned completeness assessment for caller/callee observations. */
readonly relationshipCoverage?: RelationshipCoverage;
readonly locations: readonly EvidenceLocation[];
readonly subject: EvidenceSubjectTarget | null;
}
@@ -62,6 +68,7 @@ export const createAnalysisExecution = (
readonly rawResult?: unknown;
readonly analysisProfile?: AnalysisProfileCommitment;
readonly limitations?: readonly string[];
readonly relationshipCoverage?: RelationshipCoverage;
readonly locations?: readonly EvidenceLocation[];
readonly subject?: EvidenceSubjectTarget;
} = {},
@@ -76,6 +83,13 @@ export const createAnalysisExecution = (
? {}
: { analysisProfile: structuredClone(options.analysisProfile) }),
limitations: [...(options.limitations ?? [])],
...(options.relationshipCoverage === undefined
? {}
: {
relationshipCoverage: relationshipCoverageSchema.parse(
options.relationshipCoverage,
),
}),
locations: [...(options.locations ?? [])],
subject: options.subject ?? null,
});
src/application/BinaryTargetResolver.ts
+55 -16
View File
@@ -17,7 +17,7 @@ import {
import { AnalysisCancelledError } from "../domain/analysisErrorCore.js";
import type { BinaryTarget } from "../domain/binaryTargetTypes.js";
import { err, ok, type Result } from "../domain/result.js";
import { readMipsElfAbiFlags } from "./MipsElfAbiFlags.js";
import { readMipsElfMetadata } from "./MipsElfMetadata.js";
import {
resolveAppBundleExecutable,
type ResolvedAppBundle,
@@ -26,7 +26,10 @@ import {
validateDosComLength,
type ExecutableFormatHint,
} from "../domain/dosCom.js";
import { admitRawBinary } from "../domain/rawBinary.js";
import {
admitRawBinary,
rawBinarySelectionError,
} from "../domain/rawBinary.js";
import type { BinaryArchitecture } from "../domain/binaryTargetTypes.js";
import { mzWindowsHeaderOffset } from "../domain/dosMz.js";
import {
@@ -93,6 +96,7 @@ export const parseBinaryTarget = async (
new BinaryTargetError(
path,
"raw-binary format requires an executable target kind",
{ constraint: "load_selection_rejected" },
),
);
const admitted = admitRawBinary({
@@ -101,7 +105,11 @@ export const parseBinaryTarget = async (
fileBytes: metadata.size,
});
if (!admitted.ok)
return err(new BinaryTargetError(path, admitted.error));
return err(
new BinaryTargetError(path, admitted.error.reason, {
constraint: admitted.error.constraint,
}),
);
return ok({
path,
sourcePath: canonical,
@@ -114,12 +122,16 @@ export const parseBinaryTarget = async (
entryPointSource: admitted.value.entryPointSource,
});
}
if (architecture !== undefined || entryPoint !== undefined)
const pairing = rawBinarySelectionError({
...(formatHint === undefined ? {} : { format: formatHint }),
architecture,
entryPoint,
});
if (pairing !== undefined)
return err(
new BinaryTargetError(
path,
"architecture and entry_point apply only with format=raw-binary",
),
new BinaryTargetError(path, pairing, {
constraint: "load_selection_rejected",
}),
);
if (formatHint === "dos-com") {
if (targetKind !== undefined && targetKind !== "executable")
@@ -127,10 +139,16 @@ export const parseBinaryTarget = async (
new BinaryTargetError(
path,
"DOS COM format requires an executable target kind",
{ constraint: "load_selection_rejected" },
),
);
const length = validateDosComLength(metadata.size);
if (!length.ok) return err(new BinaryTargetError(path, length.error));
if (!length.ok)
return err(
new BinaryTargetError(path, length.error, {
constraint: "file_format_rejected",
}),
);
return ok({
path,
sourcePath: canonical,
@@ -168,7 +186,12 @@ export const parseBinaryTarget = async (
hostArchitecture,
signal,
);
if (!detected.ok) return err(new BinaryTargetError(path, detected.error));
if (!detected.ok)
return err(
new BinaryTargetError(path, detected.error, {
constraint: "file_format_rejected",
}),
);
return ok({
path,
sourcePath: process.platform === "win32" ? candidate : canonical,
@@ -234,6 +257,7 @@ const detectArtifactFormat = async (
BinaryTarget["format"],
| "analysis-database"
| "mach-o"
| "mach-o-universal"
| "elf"
| "pe"
| "dos-mz"
@@ -247,6 +271,15 @@ const detectArtifactFormat = async (
const magic = Buffer.alloc(8);
const observed = await handle.read(magic, 0, magic.length, 0);
throwIfTargetResolutionCancelled(signal);
if (
observed.bytesRead === 8 &&
magic.equals(Buffer.from([0, 0x61, 0x73, 0x6d, 1, 0, 0, 0]))
)
throw new BinaryTargetError(
path,
"WebAssembly v1 artifacts are not supported by the native opening route",
{ constraint: "wasm_requires_static_inspection" },
);
if (hasZipSignature(magic.subarray(0, observed.bytesRead))) {
// The HarmonyOS App Pack suffix `.app` is a root-input concept; a nested
// ZIP member ending in `.app` keeps the generic zip classification.
@@ -290,6 +323,7 @@ const isArchiveFormat = (
BinaryTarget["format"],
| "analysis-database"
| "mach-o"
| "mach-o-universal"
| "elf"
| "pe"
| "dos-mz"
@@ -421,18 +455,23 @@ const readExecutableMetadata = async (
parsed.value.mips.elfClass !== 32
)
return parsed;
const abiFlags = await readMipsElfAbiFlags(handle, parsed.value.mips, () =>
throwIfTargetResolutionCancelled(signal),
const declarations = await readMipsElfMetadata(
handle,
parsed.value.mips,
() => throwIfTargetResolutionCancelled(signal),
);
if (!abiFlags.ok) return abiFlags;
if (!declarations.ok) return declarations;
return ok({
...parsed.value,
mips: {
...parsed.value.mips,
abiFlags: abiFlags.value.abiFlags,
...(abiFlags.value.limitations.length === 0
abiFlags: declarations.value.abiFlags,
...(declarations.value.sharedLoad === undefined
? {}
: { limitations: abiFlags.value.limitations }),
: { sharedLoad: declarations.value.sharedLoad }),
...(declarations.value.limitations.length === 0
? {}
: { limitations: declarations.value.limitations }),
},
});
};
src/application/CallPathTracing.ts
+26 -3
View File
@@ -10,6 +10,7 @@ import type { JsonValue } from "../domain/jsonValue.js";
import { ok, type Result } from "../domain/result.js";
import type { EnhancedResult } from "./EnhancedToolTypes.js";
import { resolveProcedureAddress } from "./ProcedureAddressResolution.js";
import type { RelationshipCoverage } from "../domain/relationshipCoverage.js";
type AnalysisCall = (
name: AnalysisOperation,
@@ -51,6 +52,7 @@ interface TraceState {
}>;
readonly residual: Set<string>;
readonly limitations: Set<string>;
readonly relationshipCoverage: Map<string, RelationshipCoverage>;
queueIndex: number;
traversalPath: string[];
}
@@ -117,6 +119,7 @@ const createTraceState = (input: CallPathTraceInput): TraceState => ({
failures: [],
residual: new Set(),
limitations: new Set(),
relationshipCoverage: new Map(),
queueIndex: 0,
traversalPath: input.goal === input.start ? [input.start] : [],
});
@@ -147,12 +150,27 @@ const expandAddress = async (
state.residual.add(`Call relationships were unreadable for ${address}.`);
return false;
}
for (const limitation of result.value.limitations) {
for (const limitation of result.value.limitations)
state.limitations.add(limitation);
const coverage = result.value.relationshipCoverage ?? {
status: "unknown",
reason: "Provider did not report relationship coverage.",
};
state.relationshipCoverage.set(address, coverage);
for (const diagnostic of "diagnostics" in coverage &&
Array.isArray(coverage.diagnostics)
? coverage.diagnostics
: [])
state.limitations.add(diagnostic);
if (coverage.status === "partial")
for (const limitation of coverage.limitations)
state.residual.add(
`Call relationships for ${address} are limited: ${limitation}`,
);
else if (coverage.status === "unknown")
state.residual.add(
`Call relationships for ${address} are limited: ${limitation}`,
`Call relationship coverage is unknown for ${address}: ${coverage.reason}`,
);
}
for (const relatedAddress of sortedUniqueAddresses(related.value)) {
recordEdge(input, state, {
address,
@@ -228,6 +246,11 @@ const projectTrace = (input: CallPathTraceInput, state: TraceState) => ({
traversal: {
nodes_visited: state.visited.size,
},
relationship_coverage: Object.fromEntries(
[...state.relationshipCoverage.entries()].sort(([left], [right]) =>
compareAddress(left, right),
),
),
truncated: state.residual.size > 0,
residual_unknowns: [...state.residual],
limitations: [
src/application/CapabilityInventory.test.ts
+225 -17
View File
@@ -1,6 +1,15 @@
import { describe, expect, it } from "vitest";
import { buildCapabilityInventory } from "./CapabilityInventory.js";
import {
buildCapabilityInventory,
compactToolAvailability,
} from "./CapabilityInventory.js";
import { TOOL_CONTRACTS } from "../contracts/toolContracts.js";
import {
expandToolAvailability,
toolAvailabilityInventory,
type ToolAvailability,
} from "../contracts/toolOutputSchemaPrimitives.js";
it("reports module syntax and native resolution availability separately", () => {
const input = { open: false, capabilities: [] };
@@ -55,22 +64,25 @@ const status = (
open: options.open ?? false,
...(options.kind === undefined ? {} : { kind: options.kind }),
...(options.format === undefined ? {} : { format: options.format }),
capabilities: (options.capabilities ?? []).map((capability) => ({
...capability,
availability_code: capability.available
? null
: (capability.availability_code ?? null),
effects: {
mutates_artifact: false,
launches_process: false,
may_show_ui: false,
may_access_network: false,
may_write_filesystem: false,
changes_permissions: false,
requires_root: false,
},
limitations: [],
})),
capabilities:
options.capabilities === undefined || options.capabilities.length === 0
? []
: [
{
provider: { id: "fixture", name: "Fixture", version: "1" },
shared_limitations: [],
groups: options.capabilities.map((capability) => ({
operations: [capability.operation],
effects: [],
limitations: [],
available: capability.available,
reason: capability.reason,
availability_code: capability.available
? null
: (capability.availability_code ?? null),
})),
},
],
});
const entry = (
@@ -86,6 +98,26 @@ const entry = (
};
describe("capability failure reasons", () => {
it("admits physical universal Mach-O targets for native Mach-O operations", () => {
expect(
entry(
"inspect_native_dispatch_metadata",
status({
open: true,
kind: "executable",
format: "mach-o-universal",
capabilities: [
{
operation: "inspect_native_dispatch_metadata",
available: true,
reason: null,
},
],
}),
),
).toMatchObject({ available: false, reason: "provider_missing" });
});
it.each([
{
label: "requires a target",
@@ -171,6 +203,152 @@ describe("capability failure reasons", () => {
});
});
describe("compact tool availability", () => {
it("round-trips complete facts, omitted modes, and per-group client requirements", () => {
const full = buildCapabilityInventory(status(), enabledPolicy);
const alternateRequirements = {
required: ["roots"],
optional: [],
missing_required: ["roots"],
missing_optional: [],
} satisfies ToolAvailability["client_requirements"];
const withOverride = full.map((entry, index) =>
index === 1
? {
...entry,
available: false as const,
reason: "client_capability_missing" as const,
remediation: "Use an MCP client with roots support.",
client_requirements: alternateRequirements,
}
: entry,
);
const compact = compactToolAvailability(withOverride);
const expanded = expandToolAvailability(compact).toSorted((left, right) =>
left.name.localeCompare(right.name),
);
expect(expanded).toEqual(
withOverride.toSorted((left, right) =>
left.name.localeCompare(right.name),
),
);
expect(compact.client_requirements).toEqual(
withOverride[0]?.client_requirements,
);
expect(
compact.groups.find((group) => group.client_requirements !== undefined)
?.client_requirements,
).toEqual(alternateRequirements);
expect(compact.groups.every((group) => group.names.length > 0)).toBe(true);
const names = compact.groups.flatMap((group) => group.names).sort();
expect(names).toEqual(TOOL_CONTRACTS.map(({ name }) => name).sort());
expect(new Set(names).size).toBe(names.length);
expect(expanded.some((entry) => !Object.hasOwn(entry, "modes"))).toBe(true);
});
it("preserves an empty inventory with an explicit empty client baseline", () => {
const compact = compactToolAvailability([]);
expect(compact).toEqual({
client_requirements: {
required: [],
optional: [],
missing_required: [],
missing_optional: [],
},
groups: [],
});
expect(expandToolAvailability(compact)).toEqual([]);
});
it("rejects malformed compact facts and accepts an equivalent explicit baseline", () => {
const compact = compactToolAvailability(
buildCapabilityInventory(status(), enabledPolicy),
);
const firstGroup = compact.groups[0];
expect(firstGroup).toBeDefined();
if (firstGroup === undefined) throw new Error("Expected tool groups");
const availableIndex = compact.groups.findIndex((group) => group.available);
if (availableIndex < 0) throw new Error("Expected an available tool group");
const malformedInventories = [
{
...compact,
groups: compact.groups.map((group, index) =>
index === 0
? { ...group, effects: [...group.effects, "unknown_effect"] }
: group,
),
},
{
...compact,
groups: compact.groups.map((group, index) =>
index === 0
? {
...group,
annotations: [...group.annotations, "unknown_hint"],
}
: group,
),
},
{
...compact,
groups: compact.groups.map((group, index) =>
index === 0 ? { ...group, names: [] } : group,
),
},
{
...compact,
groups: compact.groups.map((group, index) =>
index === 0 ? { ...group, names: [""] } : group,
),
},
{
...compact,
groups: compact.groups.map((group, index) =>
index === availableIndex
? { ...group, reason: "provider_missing" }
: group,
),
},
{
...compact,
groups: compact.groups.map((group, index) =>
index === availableIndex
? { ...group, remediation: "unexpected remediation" }
: group,
),
},
];
for (const malformed of malformedInventories)
expect(toolAvailabilityInventory.safeParse(malformed).success).toBe(
false,
);
const inheritedGroup = compact.groups.find(
(group) => group.client_requirements === undefined,
);
expect(inheritedGroup).toBeDefined();
if (inheritedGroup === undefined)
throw new Error("Expected an inherited client-requirements group");
const explicitBaseline = {
...compact,
groups: compact.groups.map((group) =>
group === inheritedGroup
? { ...group, client_requirements: compact.client_requirements }
: group,
),
};
const parsedOverride =
toolAvailabilityInventory.safeParse(explicitBaseline);
expect(parsedOverride.success).toBe(true);
if (parsedOverride.success)
expect(expandToolAvailability(parsedOverride.data)).toEqual(
expandToolAvailability(compact),
);
});
});
describe("partial navigation availability", () => {
it("allows explicit documents when the provider has no current-document getter", () => {
const capabilities = [
@@ -258,6 +436,36 @@ describe("partial navigation availability", () => {
remediation: "Navigation requires macOS.",
});
});
it("lists shared missing operations once in combined remediation", () => {
const availability = entry(
"get_navigation_context",
status({ open: true, kind: "executable" }),
);
expect(availability).toMatchObject({
reason: "provider_missing",
remediation:
"Configure a provider for one of: current_document, current_address, resolve_containing_procedure.",
modes: [
{
name: "current_selection",
missing_operations: [
"current_document",
"current_address",
"resolve_containing_procedure",
],
},
{
name: "explicit_document",
missing_operations: [
"current_address",
"resolve_containing_procedure",
],
},
],
});
});
});
it("reports host denial before target mismatch but keeps composed workflow prerequisites", () => {
src/application/CapabilityInventory.ts
+69 -6
View File
@@ -5,15 +5,21 @@ import type {
} from "./OptionalObservationProviders.js";
import {
providerCapability,
expandProviderCapabilities,
providerCapabilitySet,
toolAvailabilityInventory,
TOOL_ANNOTATION_NAMES,
TOOL_EFFECT_NAMES,
type ClientFeatureAvailability,
type ProviderCapability,
type ToolAvailabilityInventory,
type ToolAvailability,
type ToolAvailabilityMode,
type ToolUnavailabilityReason,
} from "../contracts/toolOutputSchemaPrimitives.js";
import type { ToolKind } from "../contracts/toolContractTypes.js";
import type { JsonValue } from "../domain/jsonValue.js";
import { isMachOFormat } from "../domain/binaryTargetTypes.js";
import type { ProviderAvailability } from "./AnalysisProvider.js";
import { TOOL_CONTRACTS } from "../contracts/toolContracts.js";
import {
@@ -26,7 +32,9 @@ const statusSchema = z.object({
open: z.boolean(),
kind: z.enum(["executable", "database", "archive", "artifact"]).optional(),
format: z.string().optional(),
capabilities: z.array(providerCapability),
capabilities: z
.array(providerCapabilitySet)
.transform(expandProviderCapabilities),
});
type ToolAvailabilityReason = "available" | ToolUnavailabilityReason;
@@ -214,6 +222,60 @@ export const buildCapabilityInventory = (
}).sort((left, right) => compareUnicodeCodePoints(left.name, right.name));
};
type ClientRequirements = ToolAvailability["client_requirements"];
type MutableAvailabilityGroup = Omit<
ToolAvailabilityInventory["groups"][number],
"names"
> & { names: string[] };
const EMPTY_CLIENT_REQUIREMENTS: ClientRequirements = {
required: [],
optional: [],
missing_required: [],
missing_optional: [],
};
/** Compact repeated metadata while retaining every operation's current availability inline. */
export const compactToolAvailability = (
availability: readonly ToolAvailability[],
): ToolAvailabilityInventory => {
const clientRequirements =
availability[0]?.client_requirements ?? EMPTY_CLIENT_REQUIREMENTS;
const groups = new Map<string, MutableAvailabilityGroup>();
for (const entry of availability) {
const effects = TOOL_EFFECT_NAMES.filter((name) => entry.effects[name]);
const annotations = TOOL_ANNOTATION_NAMES.filter(
(name) => entry.annotations[name],
);
const groupFacts = {
surface: entry.surface,
effects,
annotations,
available: entry.available,
reason: entry.reason,
remediation: entry.remediation,
...(entry.default_mode_available === undefined
? {}
: { default_mode_available: entry.default_mode_available }),
...(entry.modes === undefined ? {} : { modes: [...entry.modes] }),
...(JSON.stringify(entry.client_requirements) ===
JSON.stringify(clientRequirements)
? {}
: { client_requirements: entry.client_requirements }),
};
const key = JSON.stringify(groupFacts);
const existing = groups.get(key);
if (existing) existing.names.push(entry.name);
else groups.set(key, { ...groupFacts, names: [entry.name] });
}
return toolAvailabilityInventory.parse({
client_requirements: clientRequirements,
groups: [...groups.values()],
});
};
const availabilityFor = (context: AvailabilityContext): Availability => {
// These built-in workflows accept a caller-selected endpoint; they do not load the optional passive provider.
if (
@@ -488,7 +550,7 @@ const targetAvailability = ({
if (
targetOpen &&
MACHO_TARGET_OPERATIONS.has(name) &&
(targetKind !== "executable" || targetFormat !== "mach-o")
(targetKind !== "executable" || !isMachOFormat(targetFormat))
)
return {
reason: "target_unsupported",
@@ -626,14 +688,15 @@ const navigationContextAvailability = (
};
if (availableMode !== undefined)
return { ...facts, reason: "available", remediation: null };
const missingOperations = [
...new Set(modes.flatMap((mode) => mode.missing_operations)),
];
return {
...facts,
reason: failure?.reason ?? "provider_missing",
remediation:
specificFailureRemediation ??
`Configure a provider for one of: ${modes
.flatMap((mode) => mode.missing_operations)
.join(", ")}.`,
`Configure a provider for one of: ${missingOperations.join(", ")}.`,
};
};
src/application/ClientConfigurationDocument.ts
+22 -2
View File
@@ -5,6 +5,13 @@ import {
stringify as stringifyYaml,
} from "yaml";
import { isDeepStrictEqual } from "node:util";
import {
parseDshCordisDocument,
serializeDshDocument,
editDshDocument,
dshClientRegistrationEntry,
} from "./DshCordisPatch.js";
import {
applyEdits,
findNodeAtLocation,
@@ -104,11 +111,11 @@ const parseDocument = (
// comments, trailing commas, a BOM, or an empty existing file must be
// rejected instead of registering into a document Pi cannot load.
if (format === "pi") return objectSchema.parse(JSON.parse(text));
// Hermes stores MCP servers in `mcp_servers` inside a YAML document.
if (format === "hermes") return objectSchema.parse(parseYaml(text));
// An empty file, which some clients create before any server is added,
// holds no settings to preserve.
if (isEmptyClientConfigurationText(text)) return {};
// Hermes stores MCP servers in `mcp_servers` inside a YAML document.
if (format === "hermes") return objectSchema.parse(parseYaml(text));
const errors: ParseError[] = [];
// Accept a UTF-8 BOM without shifting diagnostics or editing the original text.
const jsonText = text.startsWith("\uFEFF") ? ` ${text.slice(1)}` : text;
@@ -131,6 +138,8 @@ export const parseClientConfiguration = (
): ClientConfigurationDocument => {
if (format === undefined || format === "unsupported")
throw new TypeError("client does not have a supported MCP config format");
// DeepSeek Harness stores a Cordis patch list, not a server table.
if (format === "dsh") return parseDshCordisDocument(text);
const document = parseDocument(text, format);
const serversKey = clientConfigurationServersKey(format);
const value = document[serversKey];
@@ -184,6 +193,14 @@ export const withClientServers = (
servers: Record<string, unknown>,
removeLegacy?: string,
): Record<string, unknown> => {
// The Harness patch list upserts or removes by entry id, not by table.
if (parsed.dialect === "dsh")
return Object.hasOwn(servers, PRODUCT_IDENTITY.mcpServerKey)
? editDshDocument(
parsed,
objectSchema.parse(servers[PRODUCT_IDENTITY.mcpServerKey]),
)
: editDshDocument(parsed);
const [key = "", nested] = parsed.serversPath;
if (nested === undefined) return { ...parsed.document, [key]: servers };
const outer = { ...objectSchema.parse(parsed.document[key]) };
@@ -1280,6 +1297,7 @@ export const serializeClientConfiguration = (
return serializeGrokConfiguration(document, originalText);
if (format === "hermes")
return serializeHermesConfiguration(document, originalText);
if (format === "dsh") return serializeDshDocument(document);
// An empty original holds no comments or settings to preserve. Writing a
// fresh document also keeps a lone BOM from shifting past the closing brace.
if (
@@ -1398,6 +1416,8 @@ export const clientRegistrationEntry = (
args,
...(Object.keys(environment).length === 0 ? {} : { env: environment }),
};
case "dsh":
return dshClientRegistrationEntry(command, environment);
case "hermes":
// Hermes reads stdio servers from `mcp_servers.<name>` and connects them
// unless `enabled` is false, so the entry states it explicitly.
src/application/ClientConfigurationFiles.ts
+16 -3
View File
@@ -10,6 +10,7 @@ import {
} from "./ClientConfigurationDocument.js";
import type { SetupClient } from "./SupportedClients.js";
import { clientRegistrationEnvironment } from "./ClientRegistrationEnvironment.js";
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
/** A named source failure that setup can explain without losing the bad path. */
export interface ClientConfigurationFileError {
@@ -22,9 +23,15 @@ export interface ClientConfigurationFileError {
/** Preserve named read failures and distinguish a missing source from an empty file. */
export const readConfigurationText = async (
path: string,
resources?: ArtifactResourceScope,
): Promise<Result<string | undefined, ClientConfigurationFileError>> => {
try {
return ok(await readRegularFileText(path));
return ok(
await readRegularFileText(
path,
resources === undefined ? {} : { resources },
),
);
} catch (cause: unknown) {
if (cause instanceof Error && "code" in cause && cause.code === "ENOENT")
return ok(undefined);
@@ -43,10 +50,11 @@ export const readConfigurationText = async (
export const readConfigurationFile = async (
path: string,
format: SetupClient["format"],
resources?: ArtifactResourceScope,
): Promise<
Result<ClientConfigurationDocument | undefined, ClientConfigurationFileError>
> => {
const source = await readConfigurationText(path);
const source = await readConfigurationText(path, resources);
if (!source.ok) return source;
const content = source.value;
if (content === undefined) return ok(undefined);
@@ -64,12 +72,17 @@ export const readConfigurationFile = async (
/** Read every applicable user file; malformed lower-priority files also stop setup. */
export const readClientConfigurationFiles = async (
client: SetupClient,
resources?: ArtifactResourceScope,
): Promise<
Result<readonly ClientConfigurationDocument[], ClientConfigurationFileError>
> => {
const documents: ClientConfigurationDocument[] = [];
for (const path of client.configPaths ?? [client.configPath]) {
const document = await readConfigurationFile(path, client.format);
const document = await readConfigurationFile(
path,
client.format,
resources,
);
if (!document.ok) return document;
if (document.value !== undefined) {
const environment = clientRegistrationEnvironment(document.value);
src/application/ClientRegistrationEnvironment.ts
+3
View File
@@ -9,12 +9,15 @@ import {
} from "./ClientConfigurationDocument.js";
import { PRODUCT_IDENTITY } from "../identity.js";
import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js";
import { dshDocument } from "./DshCordisPatch.js";
/** Read explicit REA overrides without dropping peers when one value is invalid. */
export const clientRegistrationEnvironment = (
parsed: ClientConfigurationDocument | undefined,
overrides: Readonly<Record<string, string>> = {},
): Result<Readonly<Record<string, string>>, ConfigurationError> => {
if (parsed?.dialect === "dsh")
return dshDocument(parsed.document).environment(overrides);
const key =
parsed?.dialect === "opencode" || parsed?.dialect === "opencode_v2"
? "environment"
src/application/ClientRegistrationStatus.ts
+27 -2
View File
@@ -26,6 +26,8 @@ import {
} from "./SupportedClients.js";
import type { SetupClient } from "./SupportedClients.js";
import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js";
import { dshDocument } from "./DshCordisPatch.js";
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
interface ClientRegistrationStatusBase {
readonly client: string;
@@ -65,6 +67,7 @@ const registrationSchema = z
startup_timeout_sec: z.number().positive().optional(),
type: z.string().optional(),
transport: z.string().optional(),
serverName: z.string().optional(),
tools: z.array(z.string()).optional(),
disabled: z.boolean().optional(),
enabled: z.boolean().optional(),
@@ -78,6 +81,7 @@ export const readClientRegistrationStatuses = async (
options: {
readonly platform?: NodeJS.Platform;
readonly environment?: NodeJS.ProcessEnv;
readonly resources?: ArtifactResourceScope;
} = {},
): Promise<readonly ClientRegistrationStatus[]> => {
const statuses: ClientRegistrationStatus[] = [];
@@ -110,7 +114,10 @@ export const readClientRegistrationStatuses = async (
continue;
}
try {
const files = await readClientConfigurationFiles(client);
const files = await readClientConfigurationFiles(
client,
options.resources,
);
if (!files.ok) {
statuses.push({
client: client.name,
@@ -128,7 +135,12 @@ export const readClientRegistrationStatuses = async (
);
continue;
}
const policy = await readClientPolicyBlock(client, parsed);
const policy = await readClientPolicyBlock(
client,
parsed,
undefined,
options.resources,
);
if (!policy.ok) {
statuses.push({
client: client.name,
@@ -168,6 +180,7 @@ export const readClientRegistrationStatuses = async (
parsed,
PRODUCT_IDENTITY.mcpServerKey,
) &&
(parsed.dialect !== "dsh" || dshDocument(parsed.document).ready) &&
policyBlock === undefined
? "aligned"
: "stale",
@@ -211,6 +224,13 @@ const registrationAligned = (
)
return false;
if (client.format === "vscode" && registration.type !== "stdio") return false;
// A Harness registration only connects as the rea stdio server; another
// server name or transport leaves the session without REA tools.
if (
client.format === "dsh" &&
(registration.serverName !== "rea" || registration.transport !== "stdio")
)
return false;
// OMP and Pi infer stdio for a command entry without an explicit type.
if (
(client.format === "omp" || client.format === "pi") &&
@@ -254,6 +274,11 @@ const parseRegistration = (
value: unknown,
client: SetupClient,
): z.output<typeof registrationSchema> => {
if (client.format === "dsh")
return registrationSchema
.pick({ command: true, args: true, serverName: true, transport: true })
.strip()
.parse(value);
if (client.format === "opencode") {
const entry = z
.object({
src/application/DirectAnalysis.ts
+30 -18
View File
@@ -15,14 +15,14 @@ import {
isManagedToolName,
type ManagedToolName,
} from "../contracts/managed/managedToolContracts.js";
import { EvidenceIntegrityError } from "../domain/evidenceErrors.js";
import {
EvidenceFileError,
EvidenceIntegrityError,
} from "../domain/evidenceErrors.js";
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
import type { AnalysisError } from "../domain/analysisErrorBase.js";
import { access } from "node:fs/promises";
import {
readAnalysisSnapshot,
writeAnalysisSnapshot,
} from "./binary/AnalysisSnapshotFiles.js";
import { readAnalysisSnapshot } from "./binary/AnalysisSnapshotFiles.js";
import { parseBinaryTarget } from "./BinaryTargetResolver.js";
import type { BinaryTarget } from "../domain/binaryTargetTypes.js";
import {
@@ -128,7 +128,7 @@ export const runProviderAnalysis = async (
]
): Promise<JsonValue> =>
isManagedToolName(tool)
? runManagedProviderAnalysis(dependencies, path, tool, signal)
? runManagedProviderAnalysis(dependencies, path, tool, arguments_, signal)
: withProcessCancellation(signal, (operationSignal) =>
runAnalysis(dependencies, path, tool, arguments_, {
logger,
@@ -143,6 +143,7 @@ export const runManagedProviderExecution = async (
path: string,
tool: ManagedToolName,
signal?: AbortSignal,
parameters: Readonly<Record<string, JsonValue>> = {},
): Promise<Result<AnalysisExecution, AnalysisError>> =>
withProcessCancellation(signal, async (operationSignal) => {
const session = await dependencies.createManagedBinarySession();
@@ -151,7 +152,9 @@ export const runManagedProviderExecution = async (
async () => {
const opened = await session.open(path, { signal: operationSignal });
if (!opened.ok) return opened;
return await session.execute(tool, {}, { signal: operationSignal });
return await session.execute(tool, parameters, {
signal: operationSignal,
});
},
managedAnalysisCleanupFailure,
);
@@ -161,6 +164,7 @@ const runManagedProviderAnalysis = async (
dependencies: DirectAnalysisDependencies,
path: string,
tool: ManagedToolName,
arguments_: Readonly<Record<string, JsonValue>>,
signal?: AbortSignal,
): Promise<JsonValue> => {
const execution = await runManagedProviderExecution(
@@ -168,15 +172,19 @@ const runManagedProviderAnalysis = async (
path,
tool,
signal,
arguments_,
);
if (!execution.ok) return cliError(execution.error);
const value = execution.value;
return createEvidence(value.subject ?? undefined, value.provider, {
operation: tool,
parameters: {},
parameters: arguments_,
result: value.result,
rawResult: value.rawResult,
limitations: value.limitations,
...(value.relationshipCoverage === undefined
? {}
: { relationshipCoverage: value.relationshipCoverage }),
locations: value.locations,
});
};
@@ -331,13 +339,19 @@ const runAnalysis = async (
const snapshot = session.exportAnalysisSnapshot();
if (!snapshot.ok)
return cliError(snapshot.error.retainPartialObservation(evidence));
const written = await writeAnalysisSnapshot(
const written = await session.writeAnalysisSnapshot(
snapshot.value,
snapshotPath,
true,
);
if (!written.ok)
return cliError(written.error.retainPartialObservation(evidence));
if (!written.ok) {
const recovered =
written.error instanceof EvidenceFileError &&
written.error.published !== undefined &&
(await session.closeWriteResources()).ok;
if (!recovered)
return cliError(written.error.retainPartialObservation(evidence));
}
}
return output;
},
@@ -436,6 +450,9 @@ const executeAnalysisTool = async (input: {
: { analysisProfile: result.value.analysisProfile }),
rawResult: result.value.rawResult,
limitations: result.value.limitations,
...(result.value.relationshipCoverage === undefined
? {}
: { relationshipCoverage: result.value.relationshipCoverage }),
locations: result.value.locations,
evidenceLinks:
tool === "inspect_artifact"
@@ -543,13 +560,8 @@ const analysisProfileForRoute = (
const providerIdentityForRoute = (
route: SessionProviderRoute,
operation: string,
): ProviderIdentity => {
const selected =
route.capabilities?.get(operation)?.provider ?? route.identity;
return route.profile?.provider.id === selected.id
? route.profile.provider
: selected;
};
): ProviderIdentity =>
route.capabilities?.get(operation)?.provider ?? route.identity;
const analysisProfileForEvidence = (
session: BinarySession,
src/application/Doctor.ts
+79 -2
View File
@@ -1,5 +1,6 @@
import { constants } from "node:fs";
import { access, readdir, stat } from "node:fs/promises";
import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
import { homeDirectoryFromEnvironment } from "../config/homeDirectory.js";
import { join } from "node:path";
@@ -100,6 +101,7 @@ export interface DoctorHost {
installedSkillIdentity?(
clientIds: readonly string[],
): Promise<InstalledSkillIdentity | undefined>;
close?(): Promise<string | undefined>;
clientRegistrations?(): Promise<readonly ClientRegistrationStatus[]>;
ilspyCmdVersion?(path: string): Promise<string | undefined>;
}
@@ -155,7 +157,41 @@ interface DoctorIdentity {
*/
export const runDoctor = async (
target?: string,
host: DoctorHost = systemDoctorHost(),
selectedHost?: DoctorHost,
requestedScope?: DoctorScope,
): Promise<DoctorReport> =>
selectedHost === undefined
? runDoctorOwned(target, systemDoctorHost(), requestedScope)
: inspectDoctor(target, selectedHost, requestedScope);
/** Inspect with a host whose lifetime belongs to this operation. */
export const runDoctorOwned = async (
target: string | undefined,
host: DoctorHost,
requestedScope?: DoctorScope,
): Promise<DoctorReport> => {
let report: DoctorReport;
try {
report = await inspectDoctor(target, host, requestedScope);
} catch (cause: unknown) {
const cleanup = await closeDoctorHost(host);
if (cleanup !== undefined)
throw new AggregateError(
[cause, new Error(cleanup)],
"Doctor failed and its resource cleanup remains incomplete.",
{ cause },
);
throw cause;
}
const cleanup = await closeDoctorHost(host);
return cleanup === undefined
? report
: doctorReportWithCleanup(report, cleanup);
};
const inspectDoctor = async (
target: string | undefined,
host: DoctorHost,
requestedScope?: DoctorScope,
): Promise<DoctorReport> => {
const scope = normalizeDoctorScope(requestedScope, target);
@@ -197,6 +233,36 @@ export const runDoctor = async (
};
};
const closeDoctorHost = async (
host: DoctorHost,
): Promise<string | undefined> => {
try {
return await host.close?.();
} catch (cause: unknown) {
return `Doctor host cleanup failed: ${cause instanceof Error ? cause.message : String(cause)}`;
}
};
const doctorReportWithCleanup = (
report: DoctorReport,
detail: string,
): DoctorReport => {
const check: DoctorCheck = {
name: "doctor-resource-cleanup",
ok: false,
classification: "config_drift",
detail,
remediation: "Resolve the reported cleanup failure, then rerun rea doctor.",
};
return {
...report,
healthy: false,
environment_healthy: false,
checks: [...report.checks, check],
scope_checks: [...report.scope_checks, check],
};
};
const collectDoctorIdentity = async (
host: DoctorHost,
scope: DoctorScopeReport,
@@ -307,6 +373,7 @@ export interface SystemDoctorHostOptions {
export const systemDoctorHost = (
options: SystemDoctorHostOptions = {},
): DoctorHost => {
const resources = new ArtifactResourceScope();
const platform = options.platform ?? process.platform;
const architecture = options.architecture ?? process.arch;
const environment = options.environment ?? process.env;
@@ -354,7 +421,7 @@ export const systemDoctorHost = (
name: "target",
ok: false,
classification:
targetError?.constraint === "directory_requires_file"
targetError?.constraint !== undefined
? "unsupported_target"
: "config_drift",
detail: path,
@@ -446,11 +513,21 @@ export const systemDoctorHost = (
clientIds,
environment,
platform,
resources,
),
close: async () => {
try {
await resources.close();
return undefined;
} catch (cause: unknown) {
return `Doctor resource cleanup failed: ${cause instanceof Error ? cause.message : String(cause)}`;
}
},
clientRegistrations: () =>
readClientRegistrationStatuses(homeDirectory, undefined, {
platform,
environment,
resources,
}),
};
};
src/application/DshCordisPatch.ts
+375
View File
@@ -0,0 +1,375 @@
import { isMap, isSeq, type Node, type YAMLMap } from "yaml";
import { parseEnvironment } from "../config/environment.js";
import { ConfigurationError } from "../domain/configurationErrors.js";
import { err, ok, type Result } from "../domain/result.js";
import type { ClientConfigurationDocument } from "./ClientConfigurationDocument.js";
import { CordisExpression, DshCordisYaml } from "./DshCordisYaml.js";
export const DSH_ENTRY_ID = "rea-mcp";
export const DSH_CLIENT_PACKAGE = "@deepseek-ai/dsh-mcp-client";
const LAUNCHER_KEYS = new Set(["serverName", "transport", "command", "args"]);
const isRecord = (value: unknown): value is Record<string, unknown> =>
typeof value === "object" &&
value !== null &&
!Array.isArray(value) &&
!(value instanceof CordisExpression);
interface PatchOperation {
readonly source: unknown;
readonly map: YAMLMap<unknown, unknown>;
readonly ownedRows: readonly number[];
readonly direct: boolean;
}
/** One interpretation of identity, ordered patch applicability and source nodes. */
export class DshCordisDocument {
readonly #yaml: DshCordisYaml;
readonly #operations: readonly PatchOperation[];
readonly #fields = new Map<string, unknown>();
#entryNode: YAMLMap<unknown, unknown> | undefined;
readonly #insertions: number;
#unknownSelector = false;
readonly config: Record<string, unknown> | undefined;
constructor(text: string) {
this.#yaml = new DshCordisYaml(text);
const root = this.#yaml.document.contents;
if (!isSeq(root))
throw new ConfigurationError("Cordis patches must be a list");
const operations: PatchOperation[] = [];
let insertions = 0;
for (const source of root.items) {
const map = this.#map(source);
if (map === undefined) continue;
const inserted = this.#yaml.resolve(this.#yaml.field(map, "insert"));
const targetId = this.#id(map);
const ownedRows = isSeq(inserted)
? inserted.items.flatMap((entry, index) =>
this.#id(entry) === DSH_ENTRY_ID ? [index] : [],
)
: [];
const direct = !isSeq(inserted) && targetId === DSH_ENTRY_ID;
if (ownedRows.length > 0 && targetId)
throw new ConfigurationError(
"Cannot establish the target group for a Cordis REA insertion from the home patch alone",
);
operations.push({ source, map, ownedRows, direct });
if (isSeq(inserted)) {
for (const index of ownedRows) {
const entry = this.#map(inserted.items[index]);
if (entry === undefined) continue;
insertions += 1;
this.#entryNode = entry;
this.#fields.clear();
this.#unknownSelector = false;
this.#readFields(entry);
}
} else if (direct) {
if (insertions === 0) continue;
// Cordis uses name as a selector, not a rename operation.
const selector = this.#yaml.value(this.#yaml.field(map, "name"));
if (
selector instanceof CordisExpression ||
(selector && this.#field("name") instanceof CordisExpression)
) {
this.#unknownSelector = true;
continue;
}
if (selector && selector !== this.#field("name")) continue;
this.#readFields(map, new Set(["id", "name"]));
}
}
this.#operations = operations;
this.#insertions = insertions;
const config = this.#field("config");
// Ownership survives malformed config so uninstall can remove the source row.
this.config = insertions === 0 ? undefined : isRecord(config) ? config : {};
}
#map(node: unknown): YAMLMap<unknown, unknown> | undefined {
const map = this.#yaml.resolve(node);
return isMap(map) ? map : undefined;
}
#id(node: unknown): unknown {
const id = this.#yaml.value(this.#yaml.field(this.#map(node), "id"));
if (id instanceof CordisExpression)
throw new ConfigurationError(
"Expression-backed Cordis IDs have unknown ownership; use literal IDs before changing registrations",
);
return id;
}
#field(key: string): unknown {
return this.#yaml.value(this.#fields.get(key));
}
#readFields(
map: YAMLMap<unknown, unknown>,
omitted: ReadonlySet<string> = new Set(),
): void {
for (const pair of map.items) {
const key = this.#yaml.value(pair.key);
if (typeof key === "string" && !omitted.has(key))
this.#fields.set(key, pair.value);
}
}
get carriesEntry(): boolean {
return this.#operations.some((op) => op.direct || op.ownedRows.length > 0);
}
get ready(): boolean {
const disabled = this.#field("disabled");
return (
this.#insertions === 1 &&
!this.#unknownSelector &&
this.#field("name") === DSH_CLIENT_PACKAGE &&
(disabled === undefined || disabled === false)
);
}
/** Validate literals without treating expression source as an environment value. */
environment(
overrides: Readonly<Record<string, string>> = {},
): Result<Readonly<Record<string, string>>, ConfigurationError> {
if (this.#unknownSelector)
return err(
new ConfigurationError(
"Expression-backed Cordis name selectors have unknown applicability; use literal selectors before setup",
{
settings: [
{
setting: "name",
constraint: "Use literal selectors before setup",
},
],
},
),
);
const config = this.#field("config");
if (config instanceof CordisExpression)
return err(
new ConfigurationError(
"Expression-backed Cordis config cannot be safely edited",
{
settings: [
{
setting: "config",
constraint:
"Use a literal mapping to preserve caller fields during setup",
},
],
},
),
);
if (config !== undefined && !isRecord(config))
return err(
new ConfigurationError("Cordis REA config must be a mapping", {
settings: [
{
setting: "config",
constraint: "Use a literal mapping before setup",
},
],
}),
);
const raw = this.config?.env;
if (raw !== undefined && !isRecord(raw))
return err(
new ConfigurationError("Cordis server environment must be a mapping"),
);
const literals: Record<string, string> = {};
for (const [key, value] of Object.entries(raw ?? {})) {
if (value instanceof CordisExpression) continue;
if (typeof value !== "string")
return err(
new ConfigurationError("Invalid Cordis server environment", {
settings: [
{
setting: `env.${key}`,
constraint:
"Environment values must be strings or Cordis expressions",
},
],
}),
);
literals[key] = value;
}
const environment = { ...literals, ...overrides };
const validated = parseEnvironment(environment);
return validated.ok ? ok(environment) : validated;
}
desired(
command: readonly string[],
overrides: Readonly<Record<string, string>>,
): Result<Record<string, unknown>, ConfigurationError> {
const environment = this.environment(overrides);
if (!environment.ok) return environment;
const env = {
...(isRecord(this.config?.env) ? this.config.env : {}),
...overrides,
};
const desired = { ...this.config, ...dshClientRegistrationEntry(command) };
delete desired.env;
if (Object.keys(env).length > 0) desired.env = env;
return ok(desired);
}
/** Plan every replacement from source snapshots before changing an output tree. */
serialize(desired?: Record<string, unknown>): string {
const root = this.#yaml.document.contents;
if (!isSeq(root))
throw new ConfigurationError("Cordis patches must be a list");
const replacements = new Map<unknown, unknown>();
const invalidated = new Set(
this.#operations.flatMap((op) =>
op.direct || op.ownedRows.length > 0
? this.#yaml.anchors(op.source)
: [],
),
);
let carried = false;
for (const op of this.#operations) {
if (op.direct) {
replacements.set(op.source, undefined);
continue;
}
if (op.ownedRows.length === 0) continue;
const copy = this.#yaml.detach(op.map, invalidated);
if (!isMap(copy))
throw new ConfigurationError("Cordis operation is not a mapping");
const inserted = this.#yaml.detach(
this.#yaml.resolve(this.#yaml.field(op.map, "insert")),
invalidated,
);
if (!isSeq(inserted))
throw new ConfigurationError("Cordis insertion is not a list");
copy.set("insert", inserted);
const owned = new Set(op.ownedRows);
inserted.items = inserted.items.flatMap((entry, index) => {
if (!owned.has(index)) return [entry];
if (carried || desired === undefined) return [];
carried = true;
return [this.#entry(desired, invalidated)] as typeof inserted.items;
});
replacements.set(
op.source,
inserted.items.length === 0 ? undefined : copy,
);
}
const output = this.#yaml.document.clone();
const outputRoot = output.contents;
if (!isSeq(outputRoot))
throw new ConfigurationError("Cordis patches must be a list");
outputRoot.items = root.items.flatMap((source) => {
const copy = replacements.has(source)
? replacements.get(source)
: this.#yaml.copy(source, invalidated);
return copy === undefined ? [] : [copy];
}) as typeof outputRoot.items;
if (!carried && desired !== undefined)
outputRoot.items.push(
output.createNode({
insert: [
{ id: DSH_ENTRY_ID, name: DSH_CLIENT_PACKAGE, config: desired },
],
}) as never,
);
return output.toString({ lineWidth: 0 });
}
#entry(
desired: Record<string, unknown>,
invalidated: ReadonlySet<Node>,
): YAMLMap<unknown, unknown> {
const entry = this.#yaml.detach(this.#entryNode, invalidated);
if (!isMap(entry))
throw new ConfigurationError("Cordis entry is not a mapping");
this.#yaml.setValue(entry, "id", DSH_ENTRY_ID);
this.#yaml.setValue(entry, "name", DSH_CLIENT_PACKAGE);
entry.delete("disabled");
for (const [key, value] of this.#fields) {
if (["id", "name", "config", "disabled"].includes(key)) continue;
entry.set(key, this.#yaml.detach(value, invalidated));
}
const sourceConfig = this.#map(this.#fields.get("config"));
const config =
sourceConfig === undefined
? this.#yaml.document.createNode({})
: this.#yaml.detach(sourceConfig, invalidated);
if (!isMap(config))
throw new ConfigurationError(
"Cordis registration config is not a mapping",
);
// Caller nodes keep their tags and spelling; only literal launcher values change.
for (const key of LAUNCHER_KEYS)
this.#yaml.setValue(config, key, desired[key]);
const env = this.#map(this.#yaml.field(sourceConfig, "env"));
const desiredEnv = desired.env;
if (isRecord(desiredEnv)) {
const nextEnv =
env === undefined
? this.#yaml.document.createNode({})
: this.#yaml.detach(env, invalidated);
if (!isMap(nextEnv))
throw new ConfigurationError("Cordis environment is not a mapping");
for (const [key, value] of Object.entries(desiredEnv)) {
this.#yaml.setValue(nextEnv, key, value);
}
config.set("env", nextEnv);
} else config.delete("env");
entry.set("config", config);
return entry;
}
}
class DshCordisEdit {
constructor(
readonly source: DshCordisDocument,
readonly desired?: Record<string, unknown>,
) {}
}
export const dshDocument = (
document: Record<string, unknown>,
): DshCordisDocument => {
if (!(document.cordis instanceof DshCordisDocument))
throw new ConfigurationError("Missing Cordis document owner");
return document.cordis;
};
export const parseDshCordisDocument = (
text: string,
): ClientConfigurationDocument => {
const source = new DshCordisDocument(text);
return {
document: { cordis: source },
servers: source.config === undefined ? {} : { rea: source.config },
serversPath: [],
dialect: "dsh",
legacyServers: {},
};
};
export const editDshDocument = (
parsed: ClientConfigurationDocument,
desired?: Record<string, unknown>,
): Record<string, unknown> => ({
cordis: new DshCordisEdit(dshDocument(parsed.document), desired),
});
export const serializeDshDocument = (
document: Record<string, unknown>,
): string => {
if (!(document.cordis instanceof DshCordisEdit))
throw new ConfigurationError("Missing Cordis edit plan");
return document.cordis.source.serialize(document.cordis.desired);
};
export const dshClientRegistrationEntry = (
command: readonly string[],
environment: Readonly<Record<string, string>> = {},
): Record<string, unknown> => {
const [executable = "rea", ...args] = command;
return {
serverName: "rea",
transport: "stdio",
command: executable,
args,
...(Object.keys(environment).length === 0
? {}
: { env: { ...environment } }),
};
};
src/application/DshCordisYaml.ts
+175
View File
@@ -0,0 +1,175 @@
import { isDeepStrictEqual } from "node:util";
import {
isAlias,
isMap,
isNode,
isScalar,
isSeq,
parseDocument,
visit,
type Document,
type Node,
type YAMLMap,
} from "yaml";
import { ConfigurationError } from "../domain/configurationErrors.js";
/** Expression source is never a literal value and is never evaluated by REA. */
export class CordisExpression {
readonly kind = "cordis-expression";
constructor(readonly source: string) {}
}
/** Immutable source graph; all alias resolution happens against this document. */
export class DshCordisYaml {
readonly document: Document;
readonly #values = new Map<Node, unknown>();
readonly #visiting = new Set<Node>();
constructor(text: string) {
this.document = parseDocument(text, {
customTags: [
{ tag: "tag:yaml.org,2002:js", resolve: (source: string) => source },
],
logLevel: "silent",
});
if (this.document.errors.length > 0 || this.document.warnings.length > 0)
throw new ConfigurationError("Invalid or unsupported Cordis YAML", {
cause: this.document.errors[0] ?? this.document.warnings[0],
});
if (this.document.contents === null)
this.document.contents = this.document.createNode([]);
if (!isSeq(this.document.contents))
throw new ConfigurationError(
"Cordis patches must be a YAML operation list",
);
this.value(this.document.contents);
try {
// Retain YAML's native alias-expansion admission before copying edits.
// This value is discarded; source nodes own every Cordis interpretation.
this.document.toJS();
} catch (cause) {
throw new ConfigurationError("Unsafe Cordis YAML alias expansion", {
cause,
});
}
}
resolve(node: unknown): unknown {
if (!isAlias(node)) return node;
const resolved = node.resolve(this.document);
if (resolved === undefined)
throw new ConfigurationError("Cordis YAML contains an unresolved alias");
return resolved;
}
value(node: unknown): unknown {
const resolved = this.resolve(node);
if (!isNode(resolved)) return resolved;
if (this.#values.has(resolved)) return this.#values.get(resolved);
if (this.#visiting.has(resolved))
throw new ConfigurationError(
"Cyclic Cordis YAML cannot be safely edited",
);
this.#visiting.add(resolved);
let value: unknown;
if (isScalar(resolved))
value =
resolved.tag === "tag:yaml.org,2002:js"
? new CordisExpression(String(resolved.value))
: resolved.value;
else if (isSeq(resolved))
value = resolved.items.map((item) => this.value(item));
else if (isMap(resolved)) {
const entries: [string, unknown][] = [];
for (const pair of resolved.items) {
entries.push([this.#key(pair.key), this.value(pair.value)]);
}
value = Object.fromEntries(entries);
}
this.#visiting.delete(resolved);
this.#values.set(resolved, value);
return value;
}
#key(node: unknown): string {
const key = this.value(node);
if (typeof key === "object" && key !== null)
throw new ConfigurationError(
"Cordis YAML mapping keys must be literal scalars",
);
// Match YAML's JS object projection while retaining the original key node.
return key === null ? "" : String(key);
}
field(map: YAMLMap | undefined, key: string): unknown {
return map?.items.findLast((pair) => this.#key(pair.key) === key)?.value;
}
/** Preserve unchanged nodes and carry their comments when replacing values. */
setValue(map: YAMLMap, key: string, value: unknown): void {
const original = this.field(map, key);
if (isDeepStrictEqual(this.value(original), value)) return;
const replacement = this.document.createNode(value);
if (isNode(original) && isNode(replacement)) {
if (original.comment !== undefined)
replacement.comment = original.comment;
if (original.commentBefore !== undefined)
replacement.commentBefore = original.commentBefore;
}
map.set(key, replacement);
}
/** Copy from the source before edits; materialized nodes own no anchors. */
copy(node: unknown, invalidated?: ReadonlySet<Node>): unknown {
if (isAlias(node)) {
const target = this.resolve(node);
return invalidated === undefined ||
(isNode(target) && invalidated.has(target))
? this.copy(target)
: node.clone();
}
if (!isNode(node)) return node;
const copied = node.clone();
if (
(invalidated === undefined || invalidated.has(node)) &&
(isScalar(copied) || isMap(copied) || isSeq(copied))
)
delete copied.anchor;
if (isSeq(node) && isSeq(copied))
copied.items = node.items.map((item) =>
this.copy(item, invalidated),
) as typeof copied.items;
if (isMap(node) && isMap(copied))
copied.items.forEach((pair, index) => {
pair.key = this.copy(node.items[index]!.key, invalidated);
pair.value = this.copy(node.items[index]!.value, invalidated);
});
return copied;
}
/** Replacing an anchor must not change untouched consumers of its old value. */
detach(node: unknown, invalidated: ReadonlySet<Node>): unknown {
const copied = this.copy(node, invalidated);
if (isNode(copied))
visit(copied, {
Pair: (_key, pair) => {
if (isAlias(pair.key)) pair.key = this.copy(this.resolve(pair.key));
},
Value: (_key, value) => {
delete value.anchor;
},
});
return copied;
}
anchors(node: unknown): readonly Node[] {
if (!isNode(node)) return [];
const anchors: Node[] = [];
visit(node, {
Value: (_key, value) => {
if (value.anchor) anchors.push(value);
},
});
return anchors;
}
}
src/application/EnhancedTools.ts
+50 -8
View File
@@ -33,6 +33,7 @@ import { inspectNativeDispatch } from "./native/NativeDispatchMetadataInspection
import { traceNativeValues } from "./native/NativeValueTrace.js";
import { traceNativeUiAction } from "./native/NativeUiActionTrace.js";
import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js";
import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js";
import {
invalidEnhancedInput,
@@ -343,15 +344,26 @@ export class EnhancedTools {
const relation = input.direction === "forward" ? "callees" : "callers";
const tool =
input.direction === "forward" ? "procedure_callees" : "procedure_callers";
const resolved = await resolveProcedureAddress(
this.#call.bind(this),
input.address,
const limitations = new Set<string>();
const resolved = await this.#observe(
"procedure_address",
{ procedure: input.address },
signal,
);
if (!resolved.ok) return resolved;
const discovered = new Set([resolved.value]);
for (const limitation of resolved.value.limitations)
limitations.add(limitation);
const startAddress = resolved.value.result;
if (typeof startAddress !== "string" || startAddress.length === 0)
return err(
new AnalysisOutputError(
"procedure_address",
"provider returned no canonical procedure address",
),
);
const discovered = new Set([startAddress]);
const queue: Array<{ address: string; depth: number }> = [
{ address: resolved.value, depth: 0 },
{ address: startAddress, depth: 0 },
];
let queueIndex = 0;
const graph: Record<string, JsonValue[]> = {};
@@ -385,13 +397,40 @@ export class EnhancedTools {
});
continue;
}
for (const limitation of result.value.limitations)
limitations.add(limitation);
const reportedCoverage = result.value.relationshipCoverage;
const coverage = reportedCoverage ?? {
status: "unknown" as const,
reason: "Provider did not report relationship coverage.",
};
const relationshipLimitations =
reportedCoverage === undefined
? [
...result.value.limitations,
"Provider did not report relationship coverage.",
]
: [
...(reportedCoverage.status === "partial"
? reportedCoverage.limitations
: reportedCoverage.status === "unknown"
? [reportedCoverage.reason]
: []),
...("diagnostics" in reportedCoverage
? reportedCoverage.diagnostics
: []),
];
const nodeLimitations = [...new Set(relationshipLimitations)].sort(
compareUnicodeCodePoints,
);
graph[level].push({
address: current.address,
status: "ok",
calls: [...related.value],
...(result.value.limitations.length === 0
relationship_coverage: coverage,
...(nodeLimitations.length === 0
? {}
: { limitations: [...result.value.limitations] }),
: { limitations: nodeLimitations }),
});
for (const address of related.value) {
if (!discovered.has(address)) {
@@ -400,7 +439,10 @@ export class EnhancedTools {
}
}
}
return ok(graph);
return ok({
graph,
limitations: [...limitations].sort(compareUnicodeCodePoints),
});
}
async #analyzeSwiftTypes(
src/application/EvidenceBundleCommands.ts
+30 -3
View File
@@ -1,10 +1,14 @@
import { EvidenceIntegrityError } from "../domain/evidenceErrors.js";
import {
EvidenceFileError,
EvidenceIntegrityError,
} from "../domain/evidenceErrors.js";
import type { AnalysisError } from "../domain/analysisErrorBase.js";
import type { EvidenceBundle } from "../domain/evidenceBundle.js";
import type { JsonValue } from "../domain/jsonValue.js";
import { jsonValueSchema } from "../domain/jsonValue.js";
import { err, ok, type Result } from "../domain/result.js";
import { compareBundles } from "../domain/bundleComparison.js";
import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
import {
readEvidenceBundle,
writeEvidenceBundle,
@@ -31,10 +35,33 @@ export const exportEvidenceBundleCommand = async (
): Promise<Result<JsonValue, AnalysisError>> => {
const loaded = await readEvidenceBundle(sourcePath);
if (!loaded.ok) return loaded;
return projectWrite(
const resources = new ArtifactResourceScope();
const written = await writeEvidenceBundle(
loaded.value,
await writeEvidenceBundle(loaded.value, outputPath, overwrite),
outputPath,
overwrite,
resources,
);
if (written.ok) return projectWrite(loaded.value, written);
let cleanupSucceeded = false;
try {
await resources.close();
cleanupSucceeded = true;
} catch {
// The original error cause retains the same staged-output owner for diagnostics.
}
if (
cleanupSucceeded &&
written.error instanceof EvidenceFileError &&
written.error.published
) {
return projectWrite(loaded.value, {
ok: true,
value: written.error.published,
});
}
return written;
};
/** Compare two validated canonical Evidence bundles without session state. */
src/application/EvidenceBundleFiles.ts
+3
View File
@@ -21,6 +21,7 @@ import {
} from "../domain/jsonSerialization.js";
import { readJsonFile, writeTextParts } from "./JsonFiles.js";
import { RegularFileCleanupFailure } from "./RegularFileRead.js";
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
type EvidenceReadFailure =
| EvidenceFileError
@@ -105,6 +106,7 @@ export const writeEvidenceBundle = async (
bundle: EvidenceBundle,
path: string,
overwrite: boolean,
resources: ArtifactResourceScope,
signal?: AbortSignal,
): Promise<
Result<
@@ -128,6 +130,7 @@ export const writeEvidenceBundle = async (
bufferedJsonParts(canonicalJsonParts(checked)),
path,
overwrite,
resources,
signal === undefined
? undefined
: { signal, operation: "export_evidence_bundle" },
src/application/FunctionAnalysisEvidence.ts
+3
View File
@@ -46,6 +46,9 @@ export const executeFunctionAnalysisEvidence = async (
rawResult: observation.rawResult,
limitations: observation.limitations,
locations: observation.locations,
...(observation.relationshipCoverage === undefined
? {}
: { relationshipCoverage: observation.relationshipCoverage }),
...(observation.analysisProfile === undefined
? {}
: { analysisProfile: observation.analysisProfile }),
src/application/GeminiClientSettings.ts
+10 -4
View File
@@ -4,6 +4,7 @@ import { stripComments } from "jsonc-parser";
import { z } from "zod";
import { err, ok, type Result } from "../domain/result.js";
import { PRODUCT_IDENTITY } from "../identity.js";
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
import { npxRegistrationCommand } from "./ClientRegistrationIdentity.js";
import {
clientConfigurationValuesEqual,
@@ -61,6 +62,7 @@ const normalizedPath = async (
const workspaceTrusted = async (
settings: NonNullable<SetupClient["geminiSettings"]>,
sources: readonly SettingsSource[],
resources?: ArtifactResourceScope,
): Promise<Result<boolean, ClientConfigurationFileError>> => {
let enabled = true;
for (const { path, document } of sources) {
@@ -70,7 +72,10 @@ const workspaceTrusted = async (
}
if (settings.trustOverride !== undefined) return ok(settings.trustOverride);
if (!enabled) return ok(true);
const text = await readConfigurationText(settings.trustedFoldersPath);
const text = await readConfigurationText(
settings.trustedFoldersPath,
resources,
);
if (!text.ok) return text;
if (text.value === undefined) return ok(false);
let rules: z.output<typeof trustRulesSchema>;
@@ -110,6 +115,7 @@ export const readClientPolicyBlock = async (
client: SetupClient,
userDocument?: ClientConfigurationDocument,
desiredRegistration?: unknown,
resources?: ArtifactResourceScope,
): Promise<Result<string | undefined, ClientConfigurationFileError>> => {
if (client.name !== "gemini_cli" || client.geminiSettings === undefined)
return ok(undefined);
@@ -124,7 +130,7 @@ export const readClientPolicyBlock = async (
const parsed =
path === client.configPath && userDocument !== undefined
? ok(userDocument)
: await readConfigurationFile(path, "json");
: await readConfigurationFile(path, "json", resources);
if (!parsed.ok) return parsed;
if (parsed.value !== undefined)
sources.push({ path, document: parsed.value });
@@ -143,10 +149,10 @@ export const readClientPolicyBlock = async (
".gemini",
"settings.json",
);
const document = await readConfigurationFile(path, "json");
const document = await readConfigurationFile(path, "json", resources);
if (!document.ok) return document;
if (document.value !== undefined) {
const trusted = await workspaceTrusted(settings, sources);
const trusted = await workspaceTrusted(settings, sources, resources);
if (!trusted.ok) return trusted;
if (trusted.value) {
const source = { path, document: document.value };
src/application/InvestigationProviders.ts
+3 -1
View File
@@ -18,7 +18,9 @@ export const workflowAnalysisProfile = (
): AnalysisProfileCommitment =>
createAnalysisProfile(REA_WORKFLOW_PROVIDER, {
upstream_analysis_profile: upstream,
...(operation === "binary_overview" || operation === "batch_decompile"
...(operation === "binary_overview" ||
operation === "batch_decompile" ||
operation === "get_call_graph"
? { workflow_revision: 2 }
: {}),
});
src/application/JsonFiles.ts
+123 -16
View File
@@ -5,12 +5,14 @@ import {
open,
realpath,
rename,
rm,
type FileHandle,
} from "node:fs/promises";
import { basename, dirname, resolve } from "node:path";
import { EvidenceFileError } from "../domain/evidenceErrors.js";
import {
EvidenceFileError,
type PublishedEvidenceFile,
} from "../domain/evidenceErrors.js";
import {
AnalysisCancelledError,
AnalysisResourceConstraintError,
@@ -22,6 +24,10 @@ import {
type JsonObjectMemberReader,
} from "./JsonInputFile.js";
import { NonRegularFileReadError } from "../filesystem/RegularFile.js";
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js";
import { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js";
import { OwnedStagedOutput } from "../filesystem/OwnedStagedOutput.js";
/** Request control and its owning operation for an interruptible atomic write. */
export interface TextWriteCancellation {
@@ -119,28 +125,57 @@ export const writeTextFile = async (
encoded: string,
path: string,
overwrite: boolean,
resources: ArtifactResourceScope,
): Promise<TextWriteResult<EvidenceFileError>> =>
writeTextParts([encoded], path, overwrite);
writeTextParts([encoded], path, overwrite, resources);
/** Publish streamed text atomically unless cancellation wins before publication. */
export function writeTextParts(
parts: Iterable<string>,
path: string,
overwrite: boolean,
resources: ArtifactResourceScope,
): Promise<TextWriteResult<EvidenceFileError>>;
/** Pass request control to stop streamed generation before atomic publication. */
export function writeTextParts(
parts: Iterable<string>,
path: string,
overwrite: boolean,
resources: ArtifactResourceScope,
cancellation: TextWriteCancellation | undefined,
): Promise<TextWriteResult>;
export async function writeTextParts(
export function writeTextParts(
parts: Iterable<string>,
path: string,
overwrite: boolean,
resources: ArtifactResourceScope,
cancellation?: TextWriteCancellation,
): Promise<TextWriteResult> {
return resources
.run(() =>
writeTextPartsOwned(parts, path, overwrite, resources, cancellation),
)
.catch((cause: unknown) => {
const cleanup =
cause instanceof ArtifactReaderFailure && cause.cleanup !== undefined
? cause.cleanup
: undefined;
return err(
new EvidenceFileError("write", "io", {
cause,
path: resolve(path),
...(cleanup === undefined ? {} : { cleanup }),
}),
);
});
}
const writeTextPartsOwned = async (
parts: Iterable<string>,
path: string,
overwrite: boolean,
resources: ArtifactResourceScope,
cancellation?: TextWriteCancellation,
): Promise<TextWriteResult> => {
const requestedPath = resolve(path);
try {
assertWriteActive(cancellation);
@@ -165,10 +200,40 @@ export async function writeTextParts(
destination,
parts,
overwrite,
resources,
cancellation,
);
return ok({ path: requestedPath, bytes });
} catch (cause: unknown) {
if (cause instanceof StagedOutputCleanupFailure) {
const cleanup = {
reason: failureMessage(cause.cleanupCause),
resources: [cause.owner.directory],
};
if (cause.outcome.ok)
return err(
new EvidenceFileError("write", "io", {
cause,
path: requestedPath,
cleanup,
published: { ...cause.outcome.value, path: requestedPath },
}),
);
if (cause.outcome.error instanceof AnalysisCancelledError)
return err(
new AnalysisCancelledError(cause.outcome.error.operation, {
cause,
cleanup,
}),
);
return err(
new EvidenceFileError("write", missingOrIo(cause.outcome.error), {
cause,
path: requestedPath,
cleanup,
}),
);
}
if (
cancellation?.signal.aborted === true &&
cause instanceof AnalysisCancelledError &&
@@ -189,7 +254,7 @@ export async function writeTextParts(
}),
);
}
}
};
/** A missing path or parent is a selection error, not a permission failure. */
const missingOrIo = (cause: unknown): "missing" | "io" => {
@@ -201,33 +266,75 @@ const publishFile = async (
destination: string,
parts: Iterable<string>,
overwrite: boolean,
resources: ArtifactResourceScope,
cancellation?: TextWriteCancellation,
): Promise<number> => {
const stagingDirectory = await mkdtemp(
resolve(dirname(destination), ".rea-write-"),
);
const owner = await OwnedStagedOutput.create(stagingDirectory);
let outcome: Result<PublishedEvidenceFile, unknown>;
const staged = resolve(stagingDirectory, "content");
try {
const staged = resolve(stagingDirectory, "content");
const file = await open(staged, "wx", 0o600);
let bytes = 0;
await owner.setFile(new OwnedFileHandle(file));
let written: Result<number, unknown>;
try {
bytes = await writeStagedParts(file, parts, cancellation);
const bytes = await writeStagedParts(file, parts, cancellation);
assertWriteActive(cancellation);
await file.sync();
} finally {
await file.close();
written = ok(bytes);
} catch (cause: unknown) {
written = err(cause);
}
// Once submitted, the atomic publication can win a later cancellation race.
try {
await owner.closeFile();
} catch (cause: unknown) {
if (written.ok) written = err(cause);
}
if (!written.ok) throw written.error;
// Once submitted, atomic publication can win a later cancellation race.
assertWriteActive(cancellation);
if (overwrite) await rename(staged, destination);
await owner.assertReadyForPublication(staged);
assertWriteActive(cancellation);
if (overwrite) {
await rename(staged, destination);
owner.recordPublishedRename();
}
// A hard link publishes complete bytes atomically and cannot replace an existing name.
else await link(staged, destination);
return bytes;
} finally {
await rm(stagingDirectory, { recursive: true, force: true });
outcome = ok({ path: destination, bytes: written.value });
} catch (cause: unknown) {
outcome = err(cause);
}
// One cleanup phase owns both the open descriptor (if any) and its staging directory.
const released = await resources.release({
kind: "staged-output",
resource: owner.directory,
output: owner,
});
if (released.kind === "failed")
throw new StagedOutputCleanupFailure(outcome, released.cause, owner);
if (!outcome.ok) throw outcome.error;
return outcome.value.bytes;
};
class StagedOutputCleanupFailure extends Error {
constructor(
readonly outcome: Result<PublishedEvidenceFile, unknown>,
readonly cleanupCause: unknown,
readonly owner: OwnedStagedOutput,
) {
super("Staged output cleanup failed", {
cause: outcome.ok ? cleanupCause : outcome.error,
});
}
}
const failureMessage = (cause: unknown): string =>
cause instanceof Error ? cause.message || cause.name : String(cause);
const writeStagedParts = async (
file: FileHandle,
parts: Iterable<string>,
src/application/MipsElfAbiFlags.ts → src/application/MipsElfMetadata.ts
+55 -9
View File
@@ -4,24 +4,29 @@ import type {
MipsElfMetadata,
} from "../domain/binaryTargetTypes.js";
import { err, ok, type Result } from "../domain/result.js";
import {
readMipsElfSharedLoad,
type MipsElfRelocationSection,
} from "./MipsElfSharedLoad.js";
/** Observed ABI declarations and coverage of their source tables. */
export interface MipsAbiFlagsInspection {
/** Observed ABI and load declarations, retaining unavailable source coverage. */
export interface MipsElfInspection {
readonly abiFlags: MipsAbiFlags | null;
readonly limitations: readonly string[];
readonly sharedLoad?: MipsElfMetadata["sharedLoad"];
}
/**
* Read ELF32 MIPS ABI declarations from the caller's already-open file.
* Read ELF32 MIPS ABI and shared-load declarations from the caller's already-open file.
* Inspect both typed tables (not section names), including records beyond the
* initial header probe. A missing record is unknown, not a default FP ABI.
* The caller supplies its cancellation check so its tagged error is preserved.
*/
export const readMipsElfAbiFlags = async (
export const readMipsElfMetadata = async (
handle: FileHandle,
metadata: MipsElfMetadata,
checkCancelled: () => void,
): Promise<Result<MipsAbiFlagsInspection, string>> => {
): Promise<Result<MipsElfInspection, string>> => {
checkCancelled();
if (metadata.elfClass !== 32)
return err("MIPS ABI record inspection currently supports ELF32 only");
@@ -107,6 +112,9 @@ export const readMipsElfAbiFlags = async (
if (u16(bytes, 44) === 0xffff || u16(bytes, 48) >= 0xff00)
return err("extended or reserved MIPS ELF table numbering is unsupported");
let record: Buffer | undefined;
let sharedLoad: MipsElfMetadata["sharedLoad"];
let programHeaders: Buffer | undefined;
const relocationSections: MipsElfRelocationSection[] = [];
const limitations: string[] = [];
for (const table of tables) {
if (table.count === 0) {
@@ -135,12 +143,20 @@ export const readMipsElfAbiFlags = async (
table.label,
);
if (!contents.ok) return contents;
if (!table.optional && metadata.type === 3) programHeaders = contents.value;
let found = false;
for (let i = 0; i < count; i += 1) {
checkCancelled();
const position = i * table.entrySize;
if (u32(contents.value, position + table.typeOffset) !== table.recordType)
continue;
const type = u32(contents.value, position + table.typeOffset);
if (metadata.type === 3 && table.optional && (type === 4 || type === 9))
relocationSections.push({
type,
offset: u32(contents.value, position + 16),
size: u32(contents.value, position + 20),
entrySize: u32(contents.value, position + 36),
});
if (type !== table.recordType) continue;
if (found) return err(`duplicate MIPS ABI records in ${table.label}`);
found = true;
const offset = u32(contents.value, position + table.dataOffset);
@@ -154,7 +170,37 @@ export const readMipsElfAbiFlags = async (
record = candidate.value;
}
}
if (record === undefined) return ok({ abiFlags: null, limitations });
if (metadata.type === 3 && programHeaders !== undefined) {
if (limitations.length > 0)
sharedLoad = {
status: "unavailable",
reason: `MIPS shared-image relocation declaration sources are incomplete. ${limitations.join(" ")}`,
};
else {
const shared = await readMipsElfSharedLoad(
programHeaders,
little,
fileSize,
read,
checkCancelled,
relocationSections,
);
sharedLoad = shared.ok
? shared.value
: { status: "unavailable", reason: shared.error };
}
}
const shared =
metadata.type === 3
? {
sharedLoad: sharedLoad ?? {
status: "unavailable" as const,
reason: "MIPS shared image has no inspected program header table",
},
}
: {};
if (record === undefined)
return ok({ abiFlags: null, limitations, ...shared });
// Preserve declarations verbatim, including unknown versions/values. Provider
// admission, not parsing, decides which interpretations have been verified.
const abiFlags: MipsAbiFlags = {
@@ -170,5 +216,5 @@ export const readMipsElfAbiFlags = async (
flags1: u32(record, 16),
flags2: u32(record, 20),
};
return ok({ abiFlags, limitations });
return ok({ abiFlags, limitations, ...shared });
};
src/application/MipsElfSharedLoad.ts
+220
View File
@@ -0,0 +1,220 @@
import type { MipsElfMetadata } from "../domain/binaryTargetTypes.js";
import { err, ok, type Result } from "../domain/result.js";
type ReadRange = (
offset: number,
length: number,
label: string,
) => Promise<Result<Buffer, string>>;
export interface MipsElfRelocationSection {
readonly type: number;
readonly offset: number;
readonly size: number;
readonly entrySize: number;
}
type SharedLoadFacts = Extract<
NonNullable<MipsElfMetadata["sharedLoad"]>,
{ status: "complete" }
>;
/** Source ELF facts only; the provider owns interpretation and admission. */
export const readMipsElfSharedLoad = async (
programHeaders: Buffer,
little: boolean,
fileSize: number,
read: ReadRange,
checkCancelled: () => void,
sections: readonly MipsElfRelocationSection[],
): Promise<Result<SharedLoadFacts, string>> => {
const u32 = (bytes: Buffer, offset: number): number =>
little ? bytes.readUInt32LE(offset) : bytes.readUInt32BE(offset);
const loads: {
offset: number;
address: number;
size: number;
memorySize: number;
}[] = [];
let dynamic: { offset: number; address: number; size: number } | undefined;
for (let position = 0; position < programHeaders.length; position += 32) {
checkCancelled();
const type = u32(programHeaders, position);
const offset = u32(programHeaders, position + 4);
const size = u32(programHeaders, position + 16);
if (type === 1) {
const address = u32(programHeaders, position + 8);
const memorySize = u32(programHeaders, position + 20);
if (
offset > fileSize - size ||
size > memorySize ||
address + memorySize > 0x100000000
)
return err("invalid or incomplete MIPS PT_LOAD range");
loads.push({ offset, address, size, memorySize });
}
if (type === 2) {
if (dynamic !== undefined) return err("duplicate MIPS PT_DYNAMIC tables");
dynamic = { offset, address: u32(programHeaders, position + 8), size };
}
}
if (dynamic === undefined)
return err("MIPS shared image has no PT_DYNAMIC table");
if (dynamic.size === 0 || dynamic.size % 8 !== 0)
return err("invalid MIPS ELF32 dynamic table size");
if (dynamic.offset > fileSize - dynamic.size)
return err("MIPS PT_DYNAMIC range extends beyond the file");
const ranges = loads
.filter((load) => load.memorySize > 0)
.sort((left, right) => left.address - right.address);
for (let i = 1; i < ranges.length; i += 1) {
checkCancelled();
const previous = ranges[i - 1]!;
if (ranges[i]!.address < previous.address + previous.memorySize)
return err(
"overlapping MIPS PT_LOAD memory ranges require separate mapping verification",
);
}
// One normalized index prevents relocation count * program-header count work.
const mapped = (address: number, length: number, fileBacked: boolean) => {
let low = 0;
let high = ranges.length;
while (low < high) {
const middle = Math.floor((low + high) / 2);
if (ranges[middle]!.address <= address) low = middle + 1;
else high = middle;
}
const range = ranges[low - 1];
return range !== undefined &&
address - range.address <=
(fileBacked ? range.size : range.memorySize) - length
? range
: undefined;
};
const dynamicMapping = mapped(dynamic.address, dynamic.size, true);
if (
dynamicMapping === undefined ||
dynamicMapping.offset + dynamic.address - dynamicMapping.address !==
dynamic.offset
)
return err(
"MIPS PT_DYNAMIC has no unique matching file-backed PT_LOAD range",
);
// Bound temporary storage, not admitted artifact size. ELF32 records are
// exact eight-byte structures; every declared record is examined or refused.
const chunkBytes = 64 * 1024;
const relevant = new Set([2, 7, 8, 9, 17, 18, 19, 20, 23]);
const tags = new Map<number, number>();
let terminated = false;
for (
let offset = 0;
offset < dynamic.size && !terminated;
offset += chunkBytes
) {
const chunk = await read(
dynamic.offset + offset,
Math.min(chunkBytes, dynamic.size - offset),
"dynamic table",
);
if (!chunk.ok) return chunk;
for (let position = 0; position < chunk.value.length; position += 8) {
checkCancelled();
const tag = u32(chunk.value, position);
if (tag === 0) {
terminated = true;
break;
}
if (!relevant.has(tag)) continue;
if (tags.has(tag)) return err(`duplicate MIPS dynamic tag ${tag}`);
tags.set(tag, u32(chunk.value, position + 4));
}
}
if (!terminated) return err("MIPS dynamic table has no DT_NULL terminator");
if ([2, 7, 8, 9, 23].some((tag) => (tags.get(tag) ?? 0) !== 0))
return err(
"MIPS shared RELA or PLT relocation declarations require separate verification",
);
const address = tags.get(17);
const size = tags.get(18);
const stride = tags.get(19);
const checkSections = (
offset: number,
length: number,
): string | undefined => {
let observed = 0;
for (const section of sections) {
checkCancelled();
if (section.size === 0) continue;
if (
section.type !== 9 ||
section.entrySize !== 8 ||
section.offset !== offset ||
section.size !== length
)
return "MIPS section relocation declarations outside the dynamic REL table require separate verification";
observed += 1;
}
return observed > 1
? "duplicate MIPS section relocation declarations require separate verification"
: undefined;
};
const emptyRelocations = (): Result<SharedLoadFacts, string> => {
const reason = checkSections(0, 0);
return reason === undefined
? ok({
status: "complete",
relocationCount: 0,
relocationTypes: [],
symbolRelativeCount: 0,
})
: err(reason);
};
if (address === undefined && size === undefined && stride === undefined)
return emptyRelocations();
if (
address === undefined ||
size === undefined ||
stride !== 8 ||
size % 8 !== 0
)
return err(
"incomplete or invalid MIPS DT_REL/DT_RELSZ/DT_RELENT declaration",
);
if (size === 0) return emptyRelocations();
const mapping = mapped(address, size, true);
if (mapping === undefined)
return err(
"MIPS DT_REL range has no unique complete file-backed PT_LOAD mapping",
);
const fileOffset = mapping.offset + address - mapping.address;
const sectionReason = checkSections(fileOffset, size);
if (sectionReason !== undefined) return err(sectionReason);
let symbolRelativeCount = 0;
const types = new Set<number>();
for (let offset = 0; offset < size; offset += chunkBytes) {
const chunk = await read(
fileOffset + offset,
Math.min(chunkBytes, size - offset),
"dynamic REL records",
);
if (!chunk.ok) return chunk;
for (let position = 0; position < chunk.value.length; position += 8) {
checkCancelled();
const info = u32(chunk.value, position + 4);
const type = info & 0xff;
const relocationAddress = u32(chunk.value, position);
if (type === 3 && mapped(relocationAddress, 4, false) === undefined)
return err(
"MIPS REL32 destination has no unique complete PT_LOAD memory range",
);
types.add(type);
if (type === 3 && info >>> 8 !== 0) symbolRelativeCount += 1;
}
}
return ok({
status: "complete",
relocationCount: size / 8,
relocationTypes: [...types].sort((left, right) => left - right),
symbolRelativeCount,
});
};
src/application/ReconstructionObligationLedger.closure.test.ts
+80 -37
View File
@@ -9,6 +9,8 @@ import {
request,
reviewedObligation,
} from "./ReconstructionObligationLedger.fixture.js";
import { deriveReconstructionObligationCandidates } from "./ReconstructionObligationCandidates.js";
import { evaluateReconstructionObligationLedger } from "./ReconstructionObligationLedgerEvaluation.js";
describe("reconstruction obligation ledger", () => {
it("emits the complete deterministic ledger and closure identity", () => {
@@ -86,6 +88,34 @@ describe("reconstruction obligation ledger", () => {
});
});
describe("reconstruction obligation ledger Evidence provenance", () => {
it("links observed-case Evidence separately from authority references", () => {
const capture = processEvidence();
const caseEvidence = proofEvidence("distinct-original-case");
const bundle = request([capture, caseEvidence]).evidence_bundle;
const generated = deriveReconstructionObligationCandidates(bundle, []);
const candidates = generated.candidates.map((candidate) => ({
...candidate,
observed_cases: candidate.observed_cases.map((observedCase) => ({
...observedCase,
evidence_id: caseEvidence.evidence_id,
})),
}));
const result = evaluateReconstructionObligationLedger({
candidates,
bundle,
manifest: { bindings: [], contradictions: [] },
generationLimitations: [],
});
expect(result.obligations[0]?.authority_references[0]?.evidence_id).toBe(
capture.evidence_id,
);
expect(result.evidence_links).toContain(caseEvidence.evidence_id);
});
});
describe("reconstruction obligation ledger case closure", () => {
it("keeps unobserved original cases open", () => {
const capture = processEvidence();
@@ -166,45 +196,44 @@ describe("reconstruction obligation ledger case closure", () => {
verifier_ids: ["verifier.packaged"],
claim_ids: ["claim.process"],
});
const result = build(
request([capture, proof], {
manifest: {
bindings: [
{
obligation_id: obligation.obligation_id,
owner: {
module_path: "src/processOwner.ts",
symbol: "runProcess",
owner_sha256: "2".repeat(64),
},
parser_type: null,
original_cases: originalCases(obligation, capture.evidence_id),
fixtures: obligation.required_case_kinds.map((caseKind) => ({
fixture_id: `packaged.${caseKind}`,
case_kind: caseKind,
authority: "packaged-process",
evidence_ids: [proof.evidence_id],
})),
verifier: {
verifier_id: "verifier.packaged",
claim_id: "claim.process",
command: "npm run verify:package -- process",
authority: "packaged-process",
status: "pass",
result_evidence_id: proof.evidence_id,
enumerated_obligation_ids: [obligation.obligation_id],
nondeterminism: {
mode: "partial-order",
specification:
"Startup precedes teardown; worker completion is unordered.",
},
const ledgerRequest = request([capture, proof], {
manifest: {
bindings: [
{
obligation_id: obligation.obligation_id,
owner: {
module_path: "src/processOwner.ts",
symbol: "runProcess",
owner_sha256: "2".repeat(64),
},
parser_type: null,
original_cases: originalCases(obligation, capture.evidence_id),
fixtures: obligation.required_case_kinds.map((caseKind) => ({
fixture_id: `packaged.${caseKind}`,
case_kind: caseKind,
authority: "packaged-process",
evidence_ids: [proof.evidence_id],
})),
verifier: {
verifier_id: "verifier.packaged",
claim_id: "claim.process",
command: "npm run verify:package -- process",
authority: "packaged-process",
status: "pass",
result_evidence_id: proof.evidence_id,
enumerated_obligation_ids: [obligation.obligation_id],
nondeterminism: {
mode: "partial-order",
specification:
"Startup precedes teardown; worker completion is unordered.",
},
},
],
contradictions: [],
},
}),
);
},
],
contradictions: [],
},
});
const result = build(ledgerRequest);
expect(result.obligations[0]).toMatchObject({
status: "verified",
@@ -212,5 +241,19 @@ describe("reconstruction obligation ledger case closure", () => {
});
expect(result.status).toBe("ready");
expect(result.summary.required_open).toBe(0);
const generated = deriveReconstructionObligationCandidates(
ledgerRequest.evidence_bundle,
ledgerRequest.reviewed_obligations,
);
const limited = evaluateReconstructionObligationLedger({
candidates: generated.candidates,
bundle: ledgerRequest.evidence_bundle,
manifest: ledgerRequest.manifest,
generationLimitations: ["A supplementary source was unavailable."],
});
expect(limited.obligations[0]?.status).toBe("verified");
expect(limited.status).toBe("unknown");
expect(limited.coverage.status).toBe("partial");
});
});
src/application/ReconstructionObligationLedgerDiagnostics.ts
-5
View File
@@ -57,11 +57,6 @@ export const uniqueObligationDiagnostics = (
),
);
export const hasObligationDiagnostic = (
diagnostics: readonly Diagnostic[],
code: Diagnostic["code"],
): boolean => diagnostics.some((diagnostic) => diagnostic.code === code);
export const contradictionDiagnostics = (
obligationId: string,
contradictions: ReadonlyMap<string, readonly string[]>,
src/application/ReconstructionObligationLedgerEvaluation.ts
+28 -49
View File
@@ -9,6 +9,17 @@ import {
type ReconstructionObligationLedger,
type ReconstructionObligationManifest,
} from "../domain/reconstructionObligationLedgerSchemas.js";
import { digestCanonicalValue } from "../domain/canonicalDigest.js";
import {
reconstructionObligationCoverage,
reconstructionObligationDependencyGraph,
reconstructionObligationLedgerClosureDigest,
reconstructionObligationLedgerStatus,
hasReconstructionObligationDiagnostic as hasDiagnostic,
reconstructionObligationOwnershipGraph,
reconstructionObligationReports,
summarizeReconstructionObligations,
} from "../domain/reconstructionObligationLedgerIntegrity.js";
import type { ReconstructionObligationCandidate } from "./ReconstructionObligationCandidates.js";
import {
evidenceAuthoritySupportsProof,
@@ -20,22 +31,13 @@ import {
contradictionDiagnostics,
groupManifestBindings,
groupManifestContradictions,
hasObligationDiagnostic as hasDiagnostic,
uniqueObligationDiagnostics as uniqueDiagnostics,
} from "./ReconstructionObligationLedgerDiagnostics.js";
import {
originalCaseDiagnostics,
originalCasesFor,
} from "./ReconstructionObligationOriginalCases.js";
import {
digestObligationLedgerValue,
obligationLedgerCoverage,
obligationEdgeOrder,
obligationLedgerEvidenceLinks,
obligationLedgerStatus,
obligationReports,
summarizeObligations,
} from "./ReconstructionObligationLedgerProjection.js";
import { obligationLedgerEvidenceLinks } from "./ReconstructionObligationLedgerProjection.js";
import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js";
type Diagnostic = ReconstructionObligation["diagnostics"][number];
@@ -70,64 +72,41 @@ export const evaluateReconstructionObligationLedger = ({
candidates.map((candidate) => evaluateObligation(candidate, context)),
context,
);
const ownershipGraph = obligations
.flatMap((obligation) =>
obligation.binding === null
? []
: [
{
obligation_id: obligation.obligation_id,
module_path: obligation.binding.owner.module_path,
symbol: obligation.binding.owner.symbol,
},
],
)
.sort(obligationEdgeOrder);
const dependencyGraph = obligations
.flatMap((obligation) =>
obligation.dependency_obligation_ids.map((dependencyId) => ({
obligation_id: obligation.obligation_id,
depends_on_obligation_id: dependencyId,
})),
)
.sort(obligationEdgeOrder);
const ownershipGraph = reconstructionObligationOwnershipGraph(obligations);
const dependencyGraph = reconstructionObligationDependencyGraph(obligations);
const evidenceLinks = obligationLedgerEvidenceLinks(obligations, manifest);
const limitations = ledgerLimitations(
generationLimitations,
allCandidates.length,
orphanManifestIds,
);
const coverage = obligationLedgerCoverage(limitations.length > 0);
const coverage = reconstructionObligationCoverage(limitations);
const semantic = {
coverage,
summary: summarizeObligations(obligations),
reports: obligationReports(obligations),
summary: summarizeReconstructionObligations(obligations),
reports: reconstructionObligationReports(obligations),
ownership_graph: ownershipGraph,
dependency_graph: dependencyGraph,
obligations,
evidence_links: evidenceLinks,
limitations,
};
const evaluatedStatus = obligationLedgerStatus(obligations);
const status =
evaluatedStatus === "failed"
? evaluatedStatus
: limitations.length > 0
? "unknown"
: evaluatedStatus;
const closureDigest = digestObligationLedgerValue({
schema: "ReconstructionObligationClosure",
const status = reconstructionObligationLedgerStatus(obligations, limitations);
const closureDigest = reconstructionObligationLedgerClosureDigest({
status,
...semantic,
});
return reconstructionObligationLedgerSchema.parse({
schema: "ReconstructionObligationLedger",
ledger_id: `rol_${digestObligationLedgerValue({
schema: "ReconstructionObligationLedger",
evidence_ids: bundle.records.map(({ evidence_id: id }) => id),
obligation_ids: obligations.map(({ obligation_id: id }) => id),
closure_digest: closureDigest,
})}`,
ledger_id: `rol_${digestCanonicalValue(
{
schema: "ReconstructionObligationLedger",
evidence_ids: bundle.records.map(({ evidence_id: id }) => id),
obligation_ids: obligations.map(({ obligation_id: id }) => id),
closure_digest: closureDigest,
},
"Reconstruction obligation ledger",
)}`,
closure_digest: closureDigest,
status,
...semantic,
src/application/ReconstructionObligationLedgerProjection.ts
+4 -138
View File
@@ -1,83 +1,9 @@
import { createHash } from "node:crypto";
import canonicalize from "canonicalize";
import { canonicalJson } from "../domain/comparisonSemantics.js";
import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js";
import type {
ReconstructionObligation,
ReconstructionObligationLedger,
ReconstructionObligationManifest,
} from "../domain/reconstructionObligationLedgerSchemas.js";
type DiagnosticCode = ReconstructionObligation["diagnostics"][number]["code"];
export const obligationLedgerCoverage = (
incomplete: boolean,
): ReconstructionObligationLedger["coverage"] => ({
status: incomplete ? "partial" : "complete",
});
export const summarizeObligations = (
obligations: readonly ReconstructionObligation[],
): ReconstructionObligationLedger["summary"] => ({
total: obligations.length,
required: obligations.filter(({ required }) => required).length,
verified: obligations.filter(({ status }) => status === "verified").length,
required_open: obligations.filter(
({ required, status }) => required && status !== "verified",
).length,
by_status: countBy(
obligations,
[
"unowned",
"characterized",
"implemented",
"verified",
"contradicted",
"blocked",
"out-of-scope",
"unknown",
],
({ status }) => status,
),
by_application_layer: countBy(
obligations,
[
"cli",
"protocol",
"electron",
"persistence",
"process",
"runtime",
"packaging",
"native-abi",
"application",
"other",
],
({ application_layer: layer }) => layer,
),
by_evidence_authority: countBy(
obligations,
["candidate", "observed", "reviewed", "unknown"],
({ source_state: state }) => state,
),
});
export const obligationReports = (
obligations: readonly ReconstructionObligation[],
): ReconstructionObligationLedger["reports"] => ({
missing_owner_obligation_ids: obligationIdsWith(obligations, "missing-owner"),
missing_verifier_obligation_ids: obligationIdsWith(
obligations,
"missing-verifier",
),
contradicted_obligation_ids: obligationIdsWith(obligations, "contradiction"),
residual_unknown_ids: [
...new Set(obligations.flatMap(({ residual_unknown_ids: ids }) => ids)),
].sort(),
});
import { reconstructionObligationEvidenceIds } from "../domain/reconstructionObligationLedgerIntegrity.js";
import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js";
export const obligationLedgerEvidenceLinks = (
obligations: readonly ReconstructionObligation[],
@@ -85,9 +11,7 @@ export const obligationLedgerEvidenceLinks = (
): string[] =>
[
...new Set([
...obligations.flatMap(({ authority_references: references }) =>
references.map(({ evidence_id: id }) => id),
),
...reconstructionObligationEvidenceIds(obligations),
...manifest.bindings.flatMap((binding) => [
...binding.original_cases.map(({ evidence_id: id }) => id),
...binding.fixtures.flatMap(({ evidence_ids: ids }) => ids),
@@ -97,62 +21,4 @@ export const obligationLedgerEvidenceLinks = (
]),
...manifest.contradictions.flatMap(({ evidence_ids: ids }) => ids),
]),
].sort();
export const obligationLedgerStatus = (
obligations: readonly ReconstructionObligation[],
): ReconstructionObligationLedger["status"] => {
if (obligations.length === 0) return "unknown";
const requiredOpen = obligations.filter(
({ required, status }) => required && status !== "verified",
);
if (
requiredOpen.some(
({ status, diagnostics }) =>
status === "contradicted" ||
hasDiagnostic(diagnostics, "verifier-failed"),
)
)
return "failed";
if (requiredOpen.some(({ status }) => status === "unknown")) return "unknown";
return requiredOpen.length === 0 ? "ready" : "open";
};
export const obligationEdgeOrder = (
left: { readonly obligation_id: string },
right: { readonly obligation_id: string },
): number =>
compareUnicodeCodePoints(canonicalJson(left), canonicalJson(right));
export const digestObligationLedgerValue = (value: unknown): string => {
const encoded = canonicalize(value);
if (encoded === undefined)
throw new TypeError(
"Reconstruction obligation ledger is not canonical JSON",
);
return createHash("sha256").update(encoded).digest("hex");
};
const countBy = <Value extends string, Item>(
items: readonly Item[],
values: readonly Value[],
valueFor: (item: Item) => Value,
): { key: Value; count: number }[] =>
values.map((value) => ({
key: value,
count: items.filter((item) => valueFor(item) === value).length,
}));
const obligationIdsWith = (
obligations: readonly ReconstructionObligation[],
code: DiagnosticCode,
): string[] =>
obligations
.filter(({ diagnostics }) => hasDiagnostic(diagnostics, code))
.map(({ obligation_id: id }) => id)
.sort();
const hasDiagnostic = (
diagnostics: ReconstructionObligation["diagnostics"],
code: DiagnosticCode,
): boolean => diagnostics.some((diagnostic) => diagnostic.code === code);
].sort(compareUnicodeCodePoints);
src/application/RegularFileRead.ts
+24 -2
View File
@@ -16,6 +16,7 @@ import type { AnalysisCleanupObservation } from "../domain/analysisErrorBase.js"
export interface RegularFileReadOptions {
readonly signal?: AbortSignal | undefined;
readonly symlinks?: "follow" | "reject";
readonly resources?: ArtifactResourceScope;
}
export type RegularFileReadOutcome<Value> =
@@ -68,6 +69,15 @@ export const withRegularFile = async <Value>(
path: string,
read: (handle: FileHandle, stats: Stats) => Promise<Value>,
options: RegularFileReadOptions = {},
): Promise<Value> =>
options.resources === undefined
? withRegularFileOwned(path, read, options)
: options.resources.run(() => withRegularFileOwned(path, read, options));
const withRegularFileOwned = async <Value>(
path: string,
read: (handle: FileHandle, stats: Stats) => Promise<Value>,
options: RegularFileReadOptions,
): Promise<Value> => {
const { signal } = options;
let handle: FileHandle;
@@ -78,12 +88,15 @@ export const withRegularFile = async <Value>(
});
} catch (cause: unknown) {
if (!(cause instanceof RegularFileAdmissionFailure)) throw cause;
throw new RegularFileCleanupFailure(
const failure = new RegularFileCleanupFailure(
path,
cause.owner,
{ kind: "failed", cause: cause.cause },
cause.cleanupCause,
);
if (options.resources === undefined) throw failure;
await retryRegularFileCleanup(failure, options.resources);
throw cause.cause;
}
const owner = new OwnedFileHandle(handle);
let outcome: RegularFileReadOutcome<Value>;
@@ -108,7 +121,16 @@ export const withRegularFile = async <Value>(
try {
await owner.close();
} catch (cleanupCause: unknown) {
throw new RegularFileCleanupFailure(path, owner, outcome, cleanupCause);
const failure = new RegularFileCleanupFailure(
path,
owner,
outcome,
cleanupCause,
);
if (options.resources === undefined) throw failure;
const cleanup = await retryRegularFileCleanup(failure, options.resources);
if (cleanup.outcome.kind === "failed") throw cleanup.outcome.cause;
return cleanup.outcome.value;
}
if (outcome.kind === "failed") throw outcome.cause;
return outcome.value;
src/application/SetupClientConfiguration.ts
+18 -2
View File
@@ -12,6 +12,7 @@ import {
withClientServers,
type ClientConfigurationDocument,
} from "./ClientConfigurationDocument.js";
import { dshDocument, DshCordisDocument } from "./DshCordisPatch.js";
import { constants as fsConstants } from "node:fs";
import { copyFile, mkdir, rm, writeFile } from "node:fs/promises";
import { dirname } from "node:path";
@@ -89,7 +90,10 @@ const configureClientDocument = async (
let parsed: ClientConfigurationDocument;
try {
parsed = parseClientConfiguration(
original ?? (format === "toml" || format === "grok" ? "" : "{}"),
original ??
(format === "toml" || format === "grok" || format === "dsh"
? ""
: "{}"),
format,
);
const policy = await readClientPolicyBlock(
@@ -396,6 +400,7 @@ const registrationCurrent = (
parsed: ClientConfigurationDocument,
desired: unknown,
): boolean =>
(parsed.dialect !== "dsh" || dshDocument(parsed.document).ready) &&
clientConfigurationValuesEqual(
parsed.servers[PRODUCT_IDENTITY.mcpServerKey],
desired,
@@ -410,6 +415,17 @@ const clientConfigurationDesired = (
parsed?: ClientConfigurationDocument,
) => {
const format = parsed?.dialect ?? client.format ?? "json";
const launcher =
command.length === 0 ? [PRODUCT_IDENTITY.cliBinary, "mcp"] : command;
if (format === "dsh") {
const desired = (
parsed === undefined
? new DshCordisDocument("")
: dshDocument(parsed.document)
).desired(launcher, providerEnvironment);
if (!desired.ok) throw desired.error;
return desired.value;
}
// Preserve explicitly configured server settings, never the ambient process
// environment. Newly discovered provider paths win only for their own keys.
const environment = clientRegistrationEnvironment(
@@ -419,7 +435,7 @@ const clientConfigurationDesired = (
if (!environment.ok) throw environment.error;
const registration = clientRegistrationEntry(
format,
command.length === 0 ? [PRODUCT_IDENTITY.cliBinary, "mcp"] : command,
launcher,
environment.value,
);
return {
src/application/SetupHost.ts
+23 -6
View File
@@ -122,7 +122,7 @@ export const systemSetupHost = (
nodeVersion: process.versions.node,
macosVersion: () => doctorHost.macosVersion(),
linuxDistribution: readLinuxDistribution,
close: () => closeSetupResources(resources, macHopperHost),
close: () => closeSetupResources(resources, macHopperHost, doctorHost),
initialSetupState: async (
scope?: DoctorScope,
): Promise<SetupInitialState> => {
@@ -181,9 +181,17 @@ export const systemSetupHost = (
clientIds,
environment,
platform,
resources,
),
installSkill: (clientIds) =>
installCanonicalSkill(homeDirectory, clientIds, environment, platform),
installCanonicalSkill(
homeDirectory,
clientIds,
environment,
platform,
undefined,
resources,
),
doctor: (scope) => runDoctor(undefined, doctorHost, scope),
};
};
@@ -226,11 +234,14 @@ const clientConfigurationOperations = (resources: ArtifactResourceScope) => ({
const closeSetupResources = async (
resources: ArtifactResourceScope,
macHopperHost: ReturnType<typeof systemMacHopperInstallHost> | undefined,
doctorHost: DoctorHost,
): Promise<string | undefined> => {
const [resourceCleanup, hopperCleanup] = await Promise.allSettled([
resources.close(),
Promise.resolve().then(() => macHopperHost?.close()),
]);
const [resourceCleanup, hopperCleanup, doctorCleanup] =
await Promise.allSettled([
resources.close(),
Promise.resolve().then(() => macHopperHost?.close()),
Promise.resolve().then(() => doctorHost.close?.()),
]);
const failures: string[] = [];
if (hopperCleanup.status === "rejected")
failures.push(
@@ -238,6 +249,12 @@ const closeSetupResources = async (
);
else if (hopperCleanup.value !== undefined)
failures.push(hopperCleanup.value);
if (doctorCleanup.status === "rejected")
failures.push(
`Doctor host cleanup failed: ${errorMessage(doctorCleanup.reason)}`,
);
else if (doctorCleanup.value !== undefined)
failures.push(doctorCleanup.value);
if (resourceCleanup.status === "rejected")
failures.push(errorMessage(resourceCleanup.reason));
return failures.length === 0 ? undefined : failures.join("; ");
src/application/SetupSkill.ts
+68 -16
View File
@@ -4,13 +4,18 @@ import { dirname, join } from "node:path";
import writeFileAtomic from "write-file-atomic";
import { err, ok, type Result } from "../domain/result.js";
import { readBoundedFileBytes } from "../process/BoundedFileBytes.js";
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js";
import { PRODUCT_IDENTITY } from "../identity.js";
import {
NonRegularFileReadError,
RegularFileChangedError,
} from "../filesystem/RegularFile.js";
import { withRegularFile } from "./RegularFileRead.js";
import {
RegularFileCleanupFailure,
withRegularFile,
} from "./RegularFileRead.js";
import { clientSkillDirectories } from "./SupportedClients.js";
export const MANAGED_SKILL_FILES = [
@@ -27,15 +32,21 @@ export const isManagedSkillManifest = (content: string): boolean =>
content.startsWith(`---\nname: ${PRODUCT_IDENTITY.skillName}\n`) ||
content.startsWith(`---\r\nname: ${PRODUCT_IDENTITY.skillName}\r\n`);
export const readSkillFile = (path: string): Promise<string> =>
readSkillFileState(path).then(({ content }) => content);
export const readSkillFile = (
path: string,
resources?: ArtifactResourceScope,
): Promise<string> =>
readSkillFileState(path, resources).then(({ content }) => content);
interface SkillFileState {
readonly content: string;
readonly mode: number;
}
const readSkillFileState = async (path: string): Promise<SkillFileState> =>
const readSkillFileState = async (
path: string,
resources?: ArtifactResourceScope,
): Promise<SkillFileState> =>
withRegularFile(
path,
async (handle, stats) => {
@@ -49,7 +60,10 @@ const readSkillFileState = async (path: string): Promise<SkillFileState> =>
throw new RegularFileChangedError(path);
return { content: bytes.toString("utf8"), mode: stats.mode & 0o7777 };
},
{ symlinks: "reject" },
{
symlinks: "reject",
...(resources === undefined ? {} : { resources }),
},
);
interface CanonicalSkillFile {
@@ -173,6 +187,7 @@ export const existingSkillDestinations = async (
home: string,
environment: Readonly<NodeJS.ProcessEnv> = {},
platform: NodeJS.Platform = process.platform,
resources?: ArtifactResourceScope,
): Promise<readonly SkillDestination[]> => {
const existing: SkillDestination[] = [];
for (const destination of skillDestinations(
@@ -185,11 +200,16 @@ export const existingSkillDestinations = async (
const root = await lstat(destination.path);
if (root.isSymbolicLink() || !root.isDirectory()) continue;
const manifest = join(destination.path, "SKILL.md");
const content = await readSkillFile(manifest);
const content = await readSkillFile(manifest, resources);
if (isManagedSkillManifest(content)) existing.push(destination);
} catch (cause: unknown) {
if (cause instanceof Error && "code" in cause && cause.code === "ENOENT")
continue;
if (
cause instanceof RegularFileCleanupFailure ||
cause instanceof ArtifactReaderFailure
)
throw cause;
if (cause instanceof NonRegularFileReadError) continue;
throw cause;
}
@@ -199,9 +219,10 @@ export const existingSkillDestinations = async (
const readOptionalSkillState = async (
path: string,
resources?: ArtifactResourceScope,
): Promise<SkillFileState | undefined> => {
try {
return await readSkillFileState(path);
return await readSkillFileState(path, resources);
} catch (cause: unknown) {
if (cause instanceof Error && "code" in cause && cause.code === "ENOENT")
return undefined;
@@ -214,6 +235,7 @@ const canonicalSkillFiles = async (
clientIds: readonly string[],
environment: Readonly<NodeJS.ProcessEnv> = {},
platform: NodeJS.Platform = process.platform,
resources?: ArtifactResourceScope,
): Promise<readonly CanonicalSkillFile[]> =>
Promise.all(
skillDestinations(home, clientIds, environment, platform).flatMap(
@@ -229,7 +251,7 @@ const canonicalSkillFiles = async (
),
"utf8",
),
original: await readOptionalSkillState(destination),
original: await readOptionalSkillState(destination, resources),
};
}),
),
@@ -241,12 +263,24 @@ export const canonicalSkillNeedsInstall = async (
clientIds: readonly string[] = [],
environment: Readonly<NodeJS.ProcessEnv> = {},
platform: NodeJS.Platform = process.platform,
resources?: ArtifactResourceScope,
): Promise<boolean> => {
try {
return (
await canonicalSkillFiles(home, clientIds, environment, platform)
await canonicalSkillFiles(
home,
clientIds,
environment,
platform,
resources,
)
).some(({ content, original }) => original?.content !== content);
} catch (cause: unknown) {
if (
cause instanceof RegularFileCleanupFailure ||
cause instanceof ArtifactReaderFailure
)
throw cause;
// Unreadable skill state fails open to install so setup can repair it.
void cause;
return true;
@@ -277,7 +311,6 @@ const restoreSkillFiles = async (
return failures;
};
/** Transactionally install or upgrade the canonical REA skill and references. */
/** Transactionally install the canonical skill bundle and optional references. */
export const installCanonicalSkill = async (
home: string,
@@ -285,6 +318,7 @@ export const installCanonicalSkill = async (
environment: Readonly<NodeJS.ProcessEnv> = {},
platform: NodeJS.Platform = process.platform,
operations: SkillInstallFileOperations = systemSkillInstallFileOperations,
resources?: ArtifactResourceScope,
): Promise<SkillInstallResult> => {
let attempted: CanonicalSkillFile[] = [];
let stage: "prepare" | "backup" | "write" | "readback" = "prepare";
@@ -295,6 +329,7 @@ export const installCanonicalSkill = async (
clientIds,
environment,
platform,
resources,
);
const changed = canonical.filter(
({ content, original }) => original?.content !== content,
@@ -323,7 +358,7 @@ export const installCanonicalSkill = async (
stage = "readback";
for (const { destination, content } of changed) {
failingPath = destination;
if ((await readSkillFile(destination)) !== content)
if ((await readSkillFile(destination, resources)) !== content)
throw new Error(`skill readback mismatch: ${destination}`);
}
return ok("installed");
@@ -351,6 +386,7 @@ export const readInstalledSkillIdentity = async (
clientIds: readonly string[],
environment: Readonly<NodeJS.ProcessEnv> = {},
platform: NodeJS.Platform = process.platform,
resources?: ArtifactResourceScope,
): Promise<InstalledSkillIdentity | undefined> => {
const destinations = skillDestinations(
home,
@@ -361,22 +397,38 @@ export const readInstalledSkillIdentity = async (
let content: string | undefined;
for (const destination of destinations) {
try {
content = await readSkillFile(join(destination.path, "SKILL.md"));
content = await readSkillFile(
join(destination.path, "SKILL.md"),
resources,
);
break;
} catch {
} catch (cause: unknown) {
if (
cause instanceof RegularFileCleanupFailure ||
cause instanceof ArtifactReaderFailure
)
throw cause;
// Another selected copy may provide metadata; canonical validation below
// still marks missing or unreadable copies as stale.
}
}
if (content === undefined) return undefined;
const toolCount = /^\s{2}tool_count:\s*(\d+)\s*$/mu.exec(content)?.[1];
return {
canonical: !(await canonicalSkillNeedsInstall(
let canonical: boolean;
try {
canonical = !(await canonicalSkillNeedsInstall(
home,
clientIds,
environment,
platform,
)),
resources,
));
} catch (cause: unknown) {
if (!(cause instanceof ArtifactReaderFailure)) throw cause;
canonical = false;
}
return {
canonical,
version: /^\s{2}version:\s*"([^"]+)"\s*$/mu.exec(content)?.[1] ?? null,
toolCount: toolCount === undefined ? null : Number.parseInt(toolCount, 10),
};
src/application/SupportedClients.ts
+18
View File
@@ -38,6 +38,7 @@ export interface SetupClient {
| "omp"
| "pi"
| "hermes"
| "dsh"
| "unsupported";
}
@@ -129,6 +130,12 @@ const qwenCodeDirectory = ({ home, env }: ClientPathContext): string => {
const grokDirectory = ({ home, env }: ClientPathContext): string =>
env.GROK_HOME || join(home, ".grok");
const dshHomeDirectory = ({ home, env }: ClientPathContext): string =>
env.DSH_HOME ?? join(home, ".dsh");
const dshPatchDirectory = (context: ClientPathContext): string =>
join(dshHomeDirectory(context), "cordis.patch.yml");
const hermesDirectory = ({ home, platform, env }: ClientPathContext): string =>
resolveHermesClientHome(home, platform, env);
@@ -492,6 +499,17 @@ export const SUPPORTED_CLIENT_DEFINITIONS = [
markerPath: hermesDirectory,
format: "hermes",
},
{
name: "dsh",
displayName: "DeepSeek Harness",
// The home Cordis patch applies to every profile; the skill directory is
// shared agent storage Harness discovers.
skillPath: ({ home, env }: ClientPathContext) =>
join(env.DSH_AGENTS_HOME ?? join(home, ".agents"), "skills"),
configPath: dshPatchDirectory,
markerPath: dshHomeDirectory,
format: "dsh",
},
{
name: "grok_bot",
displayName: "Grok Bot",
src/application/Uninstall.ts
+88 -35
View File
@@ -22,8 +22,10 @@ import {
skillDestinations,
} from "./SetupSkill.js";
import { isOwnedClientRegistrationCommand } from "./ClientRegistrationIdentity.js";
import { dshDocument } from "./DshCordisPatch.js";
import { resolveClientConfigTransactionPath } from "./ClientConfigPath.js";
import { readRegularFileText } from "./RegularFileRead.js";
import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
import {
manualRegistrationRemediation,
supportedClients,
@@ -50,9 +52,11 @@ export interface UninstallFileSystem {
remove(path: string, recursive?: boolean): Promise<void>;
}
const systemFileSystem: UninstallFileSystem = {
readText: (path) => readRegularFileText(path),
readRegularText: readSkillFile,
const systemFileSystem = (
resources: ArtifactResourceScope,
): UninstallFileSystem => ({
readText: (path) => readRegularFileText(path, { resources }),
readRegularText: (path) => readSkillFile(path, resources),
copy: (source, destination) =>
copyFile(source, destination, fsConstants.COPYFILE_EXCL),
writeText: (path, contents) =>
@@ -60,7 +64,7 @@ const systemFileSystem: UninstallFileSystem = {
stat: (path) => lstat(path),
realpath: (path) => realpath(path),
remove: (path, recursive = true) => rm(path, { recursive }),
};
});
/** One explicitly classified uninstall action. */
export interface UninstallItem {
@@ -83,6 +87,7 @@ export interface UninstallHost {
removeClient(client: SetupClient): Promise<UninstallItem>;
removeSkill(): Promise<UninstallItem>;
purgeData(): Promise<readonly UninstallItem[]>;
close?(): Promise<string | undefined>;
}
/**
@@ -95,35 +100,66 @@ export const runUninstall = async (
purgeData: boolean,
host: UninstallHost = systemUninstallHost(),
): Promise<UninstallResult> => {
const clients = await host.clients();
const blocked: UninstallItem[] = [];
for (const client of clients) {
const blocking = await host.inspectClient(client);
if (blocking !== undefined) blocked.push(blocking);
let result: UninstallResult | undefined;
let primaryFailure: unknown;
let failed = false;
try {
const clients = await host.clients();
const blocked: UninstallItem[] = [];
for (const client of clients) {
const blocking = await host.inspectClient(client);
if (blocking !== undefined) blocked.push(blocking);
}
const items: UninstallItem[] =
blocked.length > 0
? [
...blocked,
item(
"uninstall",
"skipped",
"No REA registration, skill, or data was removed. Repair each failed configuration, then rerun uninstall.",
),
]
: await removeAll(clients, purgeData, host);
items.push({
name: "analysis_engine",
status: "retained",
detail: "Hopper is not owned by REA uninstall.",
});
const reported = collapseUninstallItems(items);
result = {
status: reported.some(({ status }) => status === "failed")
? "failed"
: "complete",
items: reported,
};
} catch (cause: unknown) {
failed = true;
primaryFailure = cause;
}
const items: UninstallItem[] =
blocked.length > 0
? [
...blocked,
item(
"uninstall",
"skipped",
"No REA registration, skill, or data was removed. Repair each failed configuration, then rerun uninstall.",
),
]
: await removeAll(clients, purgeData, host);
items.push({
name: "analysis_engine",
status: "retained",
detail: "Hopper is not owned by REA uninstall.",
});
const reported = collapseUninstallItems(items);
return {
status: reported.some(({ status }) => status === "failed")
? "failed"
: "complete",
items: reported,
};
let cleanupFailure: string | undefined;
try {
cleanupFailure = await host.close?.();
} catch (cause: unknown) {
cleanupFailure = `Uninstall resource cleanup failed: ${cause instanceof Error ? cause.message : String(cause)}`;
}
if (failed) {
if (cleanupFailure !== undefined)
throw new AggregateError(
[primaryFailure, new Error(cleanupFailure)],
"Uninstall failed and its resource cleanup remains incomplete.",
{ cause: primaryFailure },
);
throw primaryFailure;
}
if (result === undefined)
throw new Error("Uninstall result was not produced.");
if (cleanupFailure === undefined) return result;
const items = collapseUninstallItems([
...result.items,
item("uninstall_cleanup", "failed", cleanupFailure),
]);
return { status: "failed", items };
};
/**
@@ -157,12 +193,14 @@ const removeAll = async (
/** Create uninstall effects contained to detected client configs and REA-owned paths. */
export const systemUninstallHost = (
selectedHome: string | undefined = undefined,
fileSystem: UninstallFileSystem = systemFileSystem,
selectedFileSystem: UninstallFileSystem | undefined = undefined,
environment: Readonly<NodeJS.ProcessEnv> = process.env,
platform: NodeJS.Platform = process.platform,
): UninstallHost => {
const home =
selectedHome ?? homeDirectoryFromEnvironment(environment, platform);
const resources = new ArtifactResourceScope();
const fileSystem = selectedFileSystem ?? systemFileSystem(resources);
return {
clients: () =>
Promise.resolve(
@@ -185,6 +223,14 @@ export const systemUninstallHost = (
removeSkill: () =>
removeManagedSkills(home, fileSystem, environment, platform),
purgeData: () => purgeManagedData(join(home, ".rea"), fileSystem),
close: async () => {
try {
await resources.close();
return undefined;
} catch (cause: unknown) {
return `Uninstall resource cleanup failed: ${cause instanceof Error ? cause.message : String(cause)}`;
}
},
};
};
@@ -260,10 +306,15 @@ const removeClient = async (
// Beside a native OpenCode V2 table, OpenCode also loads a V1 `mcp.rea`.
const legacyPath = legacyClientServerPath(parsed, name);
const hasNative = Object.hasOwn(parsed.servers, name);
if (!hasNative && legacyPath === undefined)
// Harness ownership is the stable entry id. A stale or malformed launcher
// still belongs to that id, and so does a direct patch that projects no row.
const carriesDshEntry =
parsed.dialect === "dsh" && dshDocument(parsed.document).carriesEntry;
if (!hasNative && legacyPath === undefined && !carriesDshEntry)
return item(client.name, "skipped", "REA registration is absent.");
const removeNative =
hasNative && isOwnedRegistration(parsed.servers[name], client);
carriesDshEntry ||
(hasNative && isOwnedRegistration(parsed.servers[name], client));
const removeLegacy =
legacyPath !== undefined &&
isOwnedRegistration(parsed.legacyServers[name], client);
@@ -311,6 +362,8 @@ const removeClient = async (
);
if (
(removeNative && Object.hasOwn(readback.servers, name)) ||
(readback.dialect === "dsh" &&
dshDocument(readback.document).carriesEntry) ||
(removeLegacy && Object.hasOwn(readback.legacyServers, name))
)
throw new Error("registration readback mismatch");
src/application/Update.test.ts
+18
View File
@@ -18,6 +18,7 @@ import {
type NpmInstallationHost,
} from "./UpdateRuntime.js";
import type { IntegrationMaintenance } from "./UpdateMaintenance.js";
import { isUpdateFailure } from "./UpdateFailure.js";
class FakeUpdateHost implements UpdateHost {
release: Result<string, ReleaseLookupFailure> = ok("3.3.0");
@@ -37,6 +38,7 @@ class FakeUpdateHost implements UpdateHost {
verifiedInstallation: UpdateInstallation | undefined;
maintenanceInspected = false;
output: UpdateOutput | undefined;
cleanupFailure: string | undefined;
installation = (): Promise<UpdateInstallation | undefined> => {
this.installationInspected = true;
@@ -63,6 +65,8 @@ class FakeUpdateHost implements UpdateHost {
this.maintenanceInspected = true;
return Promise.resolve(this.maintenance);
};
close = (): Promise<string | undefined> =>
Promise.resolve(this.cleanupFailure);
}
describe("REA update", () => {
@@ -79,6 +83,20 @@ describe("REA update", () => {
expect(host.maintenanceInspected).toBe(false);
});
it("preserves update status while marking incomplete resource cleanup as failure", async () => {
const host = new FakeUpdateHost();
host.cleanupFailure = "Regular file remains open: /fixture/skill/SKILL.md";
const result = await runUpdate("3.3.0", host);
expect(result).toMatchObject({
status: "current",
currentVersion: "3.3.0",
cleanupFailure: host.cleanupFailure,
});
expect(isUpdateFailure(result)).toBe(true);
});
it("reports an unknown current installation honestly", async () => {
const host = new FakeUpdateHost();
host.installationResult = undefined;
src/application/Update.ts
+37 -2
View File
@@ -49,10 +49,11 @@ export interface UpdateHost {
planMaintenance(
installation: UpdateInstallation,
): Promise<IntegrationMaintenance>;
close?(): Promise<string | undefined>;
}
/** Verified update outcome, separate from the unapplied integration plan. */
export type UpdateResult =
export type UpdateResult = (
| {
readonly status: "current";
readonly currentVersion: string;
@@ -80,7 +81,8 @@ export type UpdateResult =
| "verification";
readonly installedVersion?: string;
readonly remediation: string;
};
}
) & { readonly cleanupFailure?: string };
/** Resolve the exact package-manager command for the identified installation and release. */
export const updateInstallCommand = (
@@ -141,6 +143,39 @@ export const runUpdate = async (
currentVersion: string,
host: UpdateHost,
output: UpdateOutput = "human",
): Promise<UpdateResult> => {
let result: UpdateResult | undefined;
let primaryFailure: unknown;
let failed = false;
try {
result = await runUpdateWorkflow(currentVersion, host, output);
} catch (cause: unknown) {
failed = true;
primaryFailure = cause;
}
let cleanupFailure: string | undefined;
try {
cleanupFailure = await host.close?.();
} catch (cause: unknown) {
cleanupFailure = `Update resource cleanup failed: ${cause instanceof Error ? cause.message : String(cause)}`;
}
if (failed) {
if (cleanupFailure !== undefined)
throw new AggregateError(
[primaryFailure, new Error(cleanupFailure)],
"Update failed and its resource cleanup remains incomplete.",
{ cause: primaryFailure },
);
throw primaryFailure;
}
if (result === undefined) throw new Error("Update result was not produced.");
return cleanupFailure === undefined ? result : { ...result, cleanupFailure };
};
const runUpdateWorkflow = async (
currentVersion: string,
host: UpdateHost,
output: UpdateOutput,
): Promise<UpdateResult> => {
const installation = await host.installation();
const release = await host.latestVersion(installation);
src/application/UpdateFailure.ts
+1
View File
@@ -2,6 +2,7 @@ import type { UpdateResult } from "./Update.js";
/** Whether the update command left the requested release unapplied. */
export const isUpdateFailure = (result: UpdateResult): boolean => {
if (result.cleanupFailure !== undefined) return true;
const status = result.status;
switch (status) {
case "current":
src/application/UpdateMaintenance.ts
+20 -3
View File
@@ -14,6 +14,7 @@ import { existingSkillDestinations } from "./SetupSkill.js";
import type { SetupAction } from "./SetupTypes.js";
import type { Result } from "../domain/result.js";
import { readRegularFileText } from "./RegularFileRead.js";
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
/** Existing REA integrations selected for maintenance, without discovering new targets. */
export interface MaintenanceScope {
@@ -58,9 +59,11 @@ export const existingMaintenanceScope = async (
home: string,
entryPoint: string,
environment: Readonly<NodeJS.ProcessEnv>,
resources?: ArtifactResourceScope,
): Promise<MaintenanceScope> => {
const registrations = await readClientRegistrationStatuses(home, entryPoint, {
environment,
...(resources === undefined ? {} : { resources }),
});
const supported = supportedClients(home, process.platform, environment);
const clients: string[] = [];
@@ -73,7 +76,10 @@ export const existingMaintenanceScope = async (
const client = supported.find(({ name }) => name === registration.client);
if (client === undefined) continue;
const parsed = parseClientConfiguration(
await readRegularFileText(client.configPath),
await readRegularFileText(
client.configPath,
resources === undefined ? {} : { resources },
),
client.format,
);
const enabled = z
@@ -91,7 +97,12 @@ export const existingMaintenanceScope = async (
)
clients.push(client.name);
}
const skillDestinations = await existingSkillDestinations(home, environment);
const skillDestinations = await existingSkillDestinations(
home,
environment,
process.platform,
resources,
);
return {
clients,
skill: skillDestinations.length > 0,
@@ -105,9 +116,15 @@ export const planIntegrationMaintenance = async (
entryPoint: string,
environment: Readonly<NodeJS.ProcessEnv>,
execute: (command: readonly string[]) => Promise<Result<string, string>>,
resources?: ArtifactResourceScope,
): Promise<IntegrationMaintenance> => {
try {
const scope = await existingMaintenanceScope(home, entryPoint, environment);
const scope = await existingMaintenanceScope(
home,
entryPoint,
environment,
resources,
);
const skillDestinations = scope.skillDestinations ?? [];
if (scope.clients.length === 0 && !scope.skill)
return { status: "current", plannedActions: [] };
src/application/UpdateRuntime.ts
+79 -66
View File
@@ -6,6 +6,7 @@ import spawn from "cross-spawn";
import { lt, valid } from "semver";
import { PRODUCT_IDENTITY } from "../identity.js";
import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
import { err, ok, type Result } from "../domain/result.js";
import { safeParseJson } from "../domain/safeJson.js";
import {
@@ -292,74 +293,86 @@ export const systemUpdateHost = (
packageRoot = fileURLToPath(new URL("../..", import.meta.url)),
home: string | undefined = undefined,
environment: Readonly<NodeJS.ProcessEnv> = process.env,
): UpdateHost => ({
installation: async () =>
(await detectBunInstallation(
packageRoot,
bunGlobalDirectory(environment, process.platform),
{ canonicalPath: realpath, isFile },
)) ??
detectNpmInstallation(packageRoot, {
canonicalPath: realpath,
globalRoot: () =>
requireCommandOutput(["npm", "root", "--global"], environment),
globalPrefix: () =>
requireCommandOutput(["npm", "prefix", "--global"], environment),
}),
latestVersion: async (installation) => {
const context = packageManagerContext(installation, environment);
if (installation?.manager === "bun") {
const version = await runUpdateCommand(
["bun", "--version"],
): UpdateHost => {
const resources = new ArtifactResourceScope();
return {
installation: async () =>
(await detectBunInstallation(
packageRoot,
bunGlobalDirectory(environment, process.platform),
{ canonicalPath: realpath, isFile },
)) ??
detectNpmInstallation(packageRoot, {
canonicalPath: realpath,
globalRoot: () =>
requireCommandOutput(["npm", "root", "--global"], environment),
globalPrefix: () =>
requireCommandOutput(["npm", "prefix", "--global"], environment),
}),
latestVersion: async (installation) => {
const context = packageManagerContext(installation, environment);
if (installation?.manager === "bun") {
const version = await runUpdateCommand(
["bun", "--version"],
context.environment,
"structured",
context.workingDirectory,
);
if (!version.ok)
return err({ kind: "unavailable", detail: version.error });
const bunVersion = valid(version.value);
// bun info was introduced in 1.2.17; older releases reserve the command.
if (bunVersion === null || lt(bunVersion, "1.2.17"))
return err({
kind: "unsupported-package-manager",
detail: `REA updates for Bun global installations require Bun 1.2.17 or later; bun --version returned ${JSON.stringify(version.value)}. Upgrade Bun manually, then rerun rea update.`,
});
}
return npmReleaseLookup(
await runUpdateCommand(
installation?.manager === "bun"
? BUN_LATEST_VERSION_COMMAND
: npmLatestVersionCommand(installation),
context.environment,
"structured",
context.workingDirectory,
),
);
},
installVersion: async (installation, version, output) => {
const context = packageManagerContext(installation, environment);
const result = await runUpdateCommand(
updateInstallCommand(installation, version),
context.environment,
"structured",
output,
context.workingDirectory,
);
if (!version.ok)
return err({ kind: "unavailable", detail: version.error });
const bunVersion = valid(version.value);
// bun info was introduced in 1.2.17; older releases reserve the command.
if (bunVersion === null || lt(bunVersion, "1.2.17"))
return err({
kind: "unsupported-package-manager",
detail: `REA updates for Bun global installations require Bun 1.2.17 or later; bun --version returned ${JSON.stringify(version.value)}. Upgrade Bun manually, then rerun rea update.`,
});
}
return npmReleaseLookup(
await runUpdateCommand(
installation?.manager === "bun"
? BUN_LATEST_VERSION_COMMAND
: npmLatestVersionCommand(installation),
context.environment,
"structured",
context.workingDirectory,
return result.ok ? ok(undefined) : result;
},
installedVersion: (installation) =>
runUpdateCommand(
[
process.execPath,
join(installation.packageRoot, "scripts", "rea.mjs"),
"--version",
],
environment,
),
);
},
installVersion: async (installation, version, output) => {
const context = packageManagerContext(installation, environment);
const result = await runUpdateCommand(
updateInstallCommand(installation, version),
context.environment,
output,
context.workingDirectory,
);
return result.ok ? ok(undefined) : result;
},
installedVersion: (installation) =>
runUpdateCommand(
[
process.execPath,
planMaintenance: (installation) =>
planIntegrationMaintenance(
home ?? homeDirectoryFromEnvironment(environment, process.platform),
join(installation.packageRoot, "scripts", "rea.mjs"),
"--version",
],
environment,
),
planMaintenance: (installation) =>
planIntegrationMaintenance(
home ?? homeDirectoryFromEnvironment(environment, process.platform),
join(installation.packageRoot, "scripts", "rea.mjs"),
environment,
(command) => runUpdateCommand(command, environment),
),
});
environment,
(command) => runUpdateCommand(command, environment),
resources,
),
close: async () => {
try {
await resources.close();
return undefined;
} catch (cause: unknown) {
return `Update resource cleanup failed: ${cause instanceof Error ? cause.message : String(cause)}`;
}
},
};
};

Some files were not shown because too many files have changed in this diff Show More