chore: merge main into intrinsic invocation fixes

This commit is contained in:
morluto
2026-10-13 20:09:52 +08:00
1678 changed files with 219782 additions and 47605 deletions
+271 -148
View File
@@ -1,19 +1,14 @@
# JavaScript artifact reconstruction
REA can project an operator-supplied Electron/JavaScript application directory
or ASAR into [JavaScript Application Graph](javascript-application-graph.md)
without executing application code. The target-free
`analyze_javascript_application` MCP tool, dedicated
`rea analyze-javascript-application` CLI command, and generic
`rea analyze PATH` directory/ASAR route expose the same application
service and return an Evidence envelope containing the graph and an Electron
boundary summary.
Inspect a local Electron/JavaScript directory or ASAR without executing its code.
`analyze_javascript_application` and the CLI routes below return an Evidence
record with the [application graph](javascript-application-graph.md), static
Electron observations, statistics, limitations, and coverage.
The resulting Evidence can be combined with passive web or Electron capture
Evidence through
[`reconcile_javascript_runtime`](javascript-runtime-reconciliation.md). That
second operation does not rerun static analysis or contact CDP; it preserves
separate static, runtime, and cross-layer inference authorities.
Start with the [public workflow](#public-workflow), select sources when useful,
then [read the analysis](#read-the-analysis). Use [application workflows](javascript-application-workflows.md)
for tracing and comparisons, or [partial recovery](#recover-partial-analysis)
when an operation is interrupted.
## Public workflow
@@ -66,121 +61,46 @@ code-point order. Dependency manifests remain package observations but do not
declare Electron main or renderer entries. If only dependency manifests are
present, the inventory artifact remains the graph root.
## ASAR integrity
## Select sources before analysis
ASAR inventory checks Electron integrity metadata for embedded archive entries
and supplied `.asar.unpacked` companion files. An integrity failure identifies
the logical path, declared and calculated SHA-256 values, and whether the entry
was unpacked. By default, a mismatch is returned as a failure with its artifact
context. Application analysis can explicitly select `record-and-continue` to
analyze observed bytes while retaining each contradiction's declared and
observed hashes and `observed-untrusted` trust. Such results mark application
graph coverage partial and include every contradicted path in the limitations.
Mismatched nested ASARs stay opaque rather than being expanded. An unpacked
entry whose companion bytes were not supplied remains `unavailable`.
For a question about known entry sources, supply exact artifact-relative file
paths in `source_paths`, or repeat `--source-path` on either CLI route:
An unpacked entry whose companion bytes were not supplied remains
`unavailable`. REA continues analyzing embedded JavaScript and records the
missing native/resource bytes as unknown. See [what is reconstructed](#what-is-reconstructed)
for the inventory fields and [MCP integrity handling](mcp-contracts.md#integrity-record-and-continue)
for the tool-result contract.
```bash
rea analyze /absolute/path/to/app.asar \
--source-path package.json --source-path main.js --json
```
## Large results
```json
{
"input_path": "/absolute/path/to/app.asar",
"source_paths": ["package.json", "main.js"],
"detail": "summary"
}
```
Application analysis projects semantic relationships one source file at a time.
After projection, it retains only the lexical module facts and exported return
shapes needed to build application relationships. Full source IR does not stay
resident for every file together. Callable ownership and contained-reference
queries use per-file range indexes, while call sites use exact-range lookups.
Existing node budgets and coverage reporting remain in effect.
REA inventories and verifies the complete artifact, including unselected bytes,
but allocates source text and performs static analysis only for selected files.
Include supporting manifests, HTML or source maps explicitly when needed;
imports do not expand the selection. Paths must match relevant inventoried
files exactly, including nested ASAR prefixes. Unknown paths, aliases and globs
return an `invalid_request` correction. Selected files are analyzed in request
order before unselected scopes are projected. Repeated paths preserve the
caller's request and first-occurrence priority, analyzing the file once.
Omit the selection for the existing complete
application workflow.
Progress identifies the source file being parsed and projected. Cancellation is
checked between files and throughout result transfer. The owned analyzer can
be stopped while its synchronous parser or semantic extractor is active;
the CLI/MCP process remains available for control requests.
The result and summary retain `source_selection.requested_paths` and
`source_selection.unselected_files`. Every relevant file keeps its original
inventory identity; unselected files report `text_status:
"outside-analysis-scope"`. Their relationships remain unknown, and application
coverage is partial while any file remains outside scope. Selection does not
claim whole-application completeness or change integrity policy. Summary detail
only reduces delivery; source selection reduces text acquisition and analysis.
Explicit native provider or snapshot routing rejects `--source-path` rather
than ignoring it.
Graph and Evidence identifiers hash canonical JSON incrementally, without
assembling a single string for the whole graph. The canonical bytes and existing
identifiers remain unchanged. The opt-in regression check is
`npm run verify:javascript:digests`;
it hashes a value larger than the running Node engine's single-string limit.
Both application CLI routes stream complete JSON and JSONL output in bounded
chunks, including field filters and `--full-output`. They wait for each stdout
write instead of constructing or reparsing a document-sized string. The opt-in
`npm run verify:javascript:output` checks the actual CLI formatting boundary with
a temporary output file larger than the running engine's string limit, verifies
its bytes and digest, and removes it. Use `-- jsonl` for the compact JSONL check.
Each check needs space for one output file plus a 1 GiB free-space reserve.
Other CLI formats and `--token-count` still assemble whole strings.
Field selection remains useful when the caller needs a
smaller view, for example `--format json --filter-output
evidence_id,normalized_result.statistics`. Streaming output does not bound the
memory needed to construct the analysis graph itself.
JavaScript source parsing, static analysis and semantic projection run in one
owned analysis process, reused sequentially across files. Target code is never
executed. The analyzer defaults to 1024 MiB of V8 old-space and a 300000 ms
deadline per source, including startup and fact transfer. Select
`max_heap_mb` and `analysis_timeout_ms` through MCP, or `--max-heap-mb` and
`--analysis-timeout-ms` on the dedicated CLI command. These choices control the
analyzer; the caller's `NODE_OPTIONS` still controls the CLI/MCP process heap.
V8 old-space limits do not impose a process RSS limit. When REA runs on Bun,
the analyzer ignores the V8 heap flags; resource failures then report
`worker_heap_limit_bytes` as null instead of a requested or V8-reported limit.
On Windows, the analyzer's owned Job Object instead limits its committed
memory to `max_heap_mb` plus a 512 MiB reserve for Bun itself (1536 MiB by
default). Reaching it fails the current source with the same memory resource
failure and reports `worker_process_memory_limit_bytes`, also a coverage limit;
the next source starts a new analyzer. Bun elsewhere enforces no analyzer limit.
A source that exhausts the analyzer heap does not terminate the CLI/MCP
process. Completed static facts are retained, other files can still be analyzed,
and unexamined semantic relationships remain explicit unknowns. The parent
also checks transfer expansion and accumulated results against its remaining
heap before accepting semantic facts. A resource-limited result has partial
coverage, the observed limits and an `analyze-javascript-source` unknown scope
with the specific failure, including captured diagnostics when available.
Resource failures do not increment syntax parse failures. There is no fixed
source-length cutoff.
Static observations also need memory when expanded into nodes, relationships
and Evidence. If that expansion does not fit, the file node retains
`static_analysis` and any admitted `semantic_module_analysis` observations,
including their exact values and source locations. Use the public modules item
view to read them. The graph marks those unexpanded relationships as unknown.
If cumulative result capacity is exhausted, analysis stops; inventoried but
unexamined files remain present with explicit unknown scopes.
A timeout stops the remaining analysis and returns typed failure details with
completed partial Evidence. MCP failure details return
`partial_observation: { kind: "retained-evidence", evidence_id: "..." }`
after recording succeeds, so `inspect_analysis_view` can open that analysis
directly without serializing the entire partial graph into the error response.
CLI failure output retains the complete partial Evidence.
Cancellation after source analysis begins likewise
retains completed facts; an SDK client may reject its cancelled request before
receiving that reply. On the same MCP connection, inspect the Evidence bundle
and use the retained analysis ID with `inspect_analysis_view`. Owned analyzer
cleanup is verified before its private files are released. Cleanup uncertainty
is reported separately and retains ownership for retry.
CLI workflows parse larger JSON input files incrementally from one verified
regular-file handle, without constructing a document-sized string. Smaller
files retain native JSON parsing. Both paths require strict JSON and valid
UTF-8, and return one complete value; the assembled object still needs memory.
An individual JSON string exceeding the native length limit or the available
heap headroom for assembly returns
`resource_constraint` with `input_reason: "too-large"`. Supply a smaller valid
value; for Evidence workflows, re-analyze a smaller selection of the original
target and use its complete Evidence. Splitting JSON text or trimming Evidence
fields does not produce a valid workflow input.
Evidence bundle and analysis snapshot file readers still decode whole
documents. Their runtime string-limit failures report a resource constraint
with the selected path, observed bytes and UTF-16 limit, rather than malformed
JSON.
## Read the analysis
`analyze_javascript_application` returns the complete analysis Evidence by
default. Select `"detail": "summary"` to retain that complete Evidence in the
@@ -192,18 +112,82 @@ workflows as a retained reference. The summary does not repeat or truncate the
analysis; a server without session retention refuses summary detail with
`capability_unavailable`. CLI output is unchanged and always complete.
MCP prepares the complete repeated response incrementally against the pinned
SDK's 10 MiB stdio receive-buffer budget. Oversized results return an actionable
transport constraint and the exact same-session Evidence reference. Use
`inspect_analysis_view` with the retained Evidence ID for a summary, one
module, or a stable page of module identities. Use `trace_application_feature`
once a module, route, or string seed is known, or `export_evidence_bundle` to
write the complete canonical bundle without a document-sized allocation.
Same-session analysis reads reuse authenticated immutable snapshots; foreign
inline Evidence is still parsed and authenticated.
See [MCP tool results](mcp-contracts.md#tool-results) for larger client buffers
and `REA_MCP_MAX_RESPONSE_BYTES`. Follow-up results remain complete and can also
exceed the transport budget.
With a connected MCP `client` and the caller-selected absolute `inputPath`, this
recipe analyzes once and pages module identities using the returned ID:
```js
const analysis = await client.callTool({
name: "analyze_javascript_application",
arguments: { input_path: inputPath, detail: "summary" },
});
if (analysis.isError) throw new Error(JSON.stringify(analysis.content));
const parentId =
analysis.structuredContent.normalized_result.parent_evidence_id;
const modules = await client.callTool({
name: "inspect_analysis_view",
arguments: {
source: { kind: "retained-evidence", evidence_id: parentId },
view: { kind: "page", collection: "modules", offset: 0, limit: 16 },
},
});
if (modules.isError) throw new Error(JSON.stringify(modules.content));
```
Select a returned module's exact `node_id` for an item view. A path can identify
several graph roles; module items preserve their recorded properties and source
locations. Follow `coverage.next_offset` when another page matters. For feature
tracing, version comparison, and retained-input examples, use the
[application workflows](javascript-application-workflows.md).
For supplied passive browser or Electron observations, use
[runtime reconciliation](javascript-runtime-reconciliation.md). It compares
existing Evidence without rerunning static analysis or contacting CDP.
## ASAR integrity
ASAR inventory checks Electron integrity metadata for embedded archive entries
and supplied `.asar.unpacked` companion files. An integrity failure identifies
the logical path, declared and calculated SHA-256 values, and whether the entry
was unpacked. By default, a mismatch is returned as a failure with its artifact
context. Application analysis can explicitly select `record-and-continue` to
analyze observed bytes while retaining each contradiction's declared and
observed hashes and `observed-untrusted` trust. Such results mark application
graph coverage partial and include every contradicted path in the limitations.
Mismatched nested ASARs stay opaque rather than being expanded.
An unpacked entry whose companion bytes were not supplied remains
`unavailable`. REA continues analyzing embedded JavaScript and records the
missing native/resource bytes as unknown. The occurrence keeps
`hash_status: unavailable`; no content-addressed child artifact is created for
those missing bytes. See [MCP integrity handling](mcp-contracts.md#integrity-record-and-continue)
for the tool-result contract.
## Large results
Use MCP `detail: "summary"` for a large application, then read selected views of
the retained analysis. Summary changes delivery only. To reduce source parsing,
select exact [source paths](#select-sources-before-analysis).
If a complete response exceeds the transport budget, use its retained Evidence
ID for a summary, module page, or item, or export the bundle. See
[MCP tool results](mcp-contracts.md#tool-results) for delivery limits, export, and
matching server/client buffer settings. Follow-up results also need to fit that
budget. [Analyzer limits](#analyzer-and-result-admission) control analysis memory,
source deadlines, and semantic capacity separately.
## Recover partial analysis
A source timeout stops the remaining analysis and returns completed facts as
partial Evidence. Cancellation after source analysis begins can also retain
completed facts, including when interrupted during cleanup, graph construction,
or Evidence sealing. Unexamined relationships remain unknown.
MCP returns a `partial_observation` retained reference after recording succeeds;
open it with `inspect_analysis_view` on the same connection. If the SDK stopped
waiting before receiving the reply, follow [MCP cancellation recovery](mcp-contracts.md#progress-and-cancellation)
to find the record. CLI failure output contains the complete partial Evidence.
Owned analyzer cleanup is verified before private files are released; uncertain
cleanup retains ownership for retry and is reported separately.
## What is reconstructed
@@ -220,17 +204,14 @@ execute them.
JavaScript and HTML source ranges retain an initial UTF-8 BOM as one UTF-16
code unit, matching the original bytes identified by the artifact digest.
If an ASAR declares an unpacked companion entry but the corresponding
`<archive>.unpacked` file is absent from the operator-supplied artifact set, REA
keeps the ASAR occurrence with `hash_status: unavailable`, records an explicit
limitation, and does not create a content-addressed child artifact for those
missing bytes. This allows static JavaScript/Electron reconstruction to proceed
for the embedded files while preserving the missing native/resource bytes as an
unknown instead of silently treating them as absent or verified.
Plain `.ts` artifact sources use TypeScript syntax without JSX, including
angle-bracket type assertions. `.tsx` and `.jsx` retain JSX parsing.
Relative TypeScript imports can name their emitted `.js`, `.mjs`, or `.cjs`
counterparts while resolving to inventoried source files. This substitution
requires import or require semantics. Literal Electron preload, renderer, and
utility-process runtime paths retain the runtime filename selected by the caller.
Selected bounded text is then parsed as inert data to recover:
- `package.json` metadata and declared main or renderer entrypoints;
@@ -320,9 +301,30 @@ Source-map contents are parsed as local input. The graph stores original source
Only explicitly present BrowserWindow values are observations. REA does not
substitute version-dependent Electron defaults for omitted `webPreferences`.
Dynamic option objects, bridge keys, API objects, and IPC channel expressions
Unresolved option objects, bridge keys, API objects, and IPC channel expressions
remain unknown and make coverage partial.
Recognized literal module-table bootstraps also support bounded lexical
substitution through module exports, import helper bodies, local wrapper calls,
returned closures, object spreads, and callback parameters. Recovered Electron
paths retain source ranges, module identities, literal channels and branch
conditions as static candidates. Unknown spreads, reassigned bindings, accessors,
dynamic channels and unsupported syntax do not become exact implementations.
This pass neither executes the bundle nor proves that runtime guards pass.
An ambiguous or unresolved handler retains its channel registration but does
not acquire a path to one arbitrarily selected callback. Unknown factory
conditions retain both return branches, including fallthrough. Short-circuit
selection distinguishes truthiness from nullishness; uncertain `??` paths
retain a nullish guard rather than a falsy guard.
These paths appear as contextual `javascript-expression` nodes. An exact `api`
seed such as `desktop.copy` can trace an exposed member through a unique IPC
pairing to an external API call. `addListener` is normalized to `on`. Semantic
projection retains these boundary paths before bulk function definitions, within
the existing per-file node budget; omitted general semantics remain explicitly
partial. Wrapper expansion is bounded by one million visits/substitutions and
128 nested substitutions, with exhaustion reported as unknown coverage.
Electron API identity follows proven lexical module aliases. Namespace interop
also preserves identity when the complete Rolldown helper operations, lexical
roles, live getters, and captured intrinsic origins match, or when a stable
@@ -373,7 +375,7 @@ do not establish that an application has no Electron boundary usage.
IPC pairing is an inference, not an observation. A renderer `invoke` pairs only
with one unique `ipcMain.handle`/`handleOnce` candidate on the same exact literal
channel; a renderer send pairs only with one unique `ipcMain.on`/`once`
channel; a renderer send pairs only with one unique `ipcMain.on`/`once`/`addListener`
candidate. Dynamic channels are never paired, and multiple compatible handlers
are reported as ambiguous without adding a caller-to-handler edge. A matching
channel still does not prove registration order, reachability, or runtime use.
@@ -435,6 +437,110 @@ and inventoried containment edge is retained. If parsing or artifact safety
boundaries omit input, coverage reports that omission; when an exact omission
count is not knowable, `omitted_count` is `null` rather than a guessed value.
### Analyzer and result admission
Application analysis projects semantic relationships one source file at a time.
After projection, it retains only the lexical module facts and exported return
shapes needed to build application relationships. Full source IR does not stay
resident for every file together. Callable ownership and contained-reference
queries use per-file range indexes, while call sites use exact-range lookups.
Existing node budgets and coverage reporting remain in effect.
Progress identifies the source file being parsed and projected. Cancellation is
checked between files and throughout result transfer. The owned analyzer can
be stopped while its synchronous parser or semantic extractor is active;
the CLI/MCP process remains available for control requests.
Graph and Evidence identifiers hash canonical JSON incrementally, without
assembling a single string for the whole graph. The canonical bytes and existing
identifiers remain unchanged. The opt-in regression check is
`npm run verify:javascript:digests`;
it hashes a value larger than the running Node engine's single-string limit.
Both application CLI routes stream complete JSON and JSONL output in bounded
chunks, including field filters and `--full-output`. They wait for each stdout
write instead of constructing or reparsing a document-sized string. The opt-in
`npm run verify:javascript:output` checks the actual CLI formatting boundary with
a temporary output file larger than the running engine's string limit, verifies
its bytes and digest, and removes it. Use `-- jsonl` for the compact JSONL check.
Each check needs space for one output file plus a 1 GiB free-space reserve.
Other CLI formats and `--token-count` still assemble whole strings.
Field selection remains useful when the caller needs a
smaller view, for example `--format json --filter-output
evidence_id,normalized_result.statistics`. Streaming output does not bound the
memory needed to construct the analysis graph itself.
JavaScript source parsing, static analysis and semantic projection run in one
owned analysis process, reused sequentially across files. Target code is never
executed. The analyzer defaults to 1024 MiB of V8 old-space and a 300000 ms
deadline per source, including startup and fact transfer. Select
`max_heap_mb` and `analysis_timeout_ms` through MCP, or `--max-heap-mb` and
`--analysis-timeout-ms` on the dedicated CLI command. These choices control the
analyzer; the caller's `NODE_OPTIONS` still controls the CLI/MCP process heap.
V8 old-space limits do not impose a process RSS limit. When REA runs on Bun,
the analyzer ignores the V8 heap flags; resource failures then report
`worker_heap_limit_bytes` as null instead of a requested or V8-reported limit.
On Windows, the analyzer's owned Job Object instead limits its committed
memory to `max_heap_mb` plus a 512 MiB reserve for Bun itself (1536 MiB by
default). Reaching it fails the current source with the same memory resource
failure and reports `worker_process_memory_limit_bytes`, also a coverage limit;
the next source starts a new analyzer. Bun elsewhere enforces no analyzer limit.
The default semantic capacities are 100000 nodes across the application and
20000 per source. Large bundles may explicitly select `max_semantic_nodes` and
`max_semantic_nodes_per_file` (CLI: `--max-semantic-nodes` and
`--max-semantic-nodes-per-file`), each from 1 to 1000000. The per-file offer is
also bounded by the remaining application capacity. Raising these controls does
not bypass worker heap, execution deadline or parent result admission. Truncated
coverage reports both effective node ceilings; omitted relationships remain
unknown.
While sources remain, the sequential worker reserves one next compact static
checkpoint at its 8 MiB bound with the same 6x retention expansion used at
admission. Earlier checkpoints are already charged to retained bytes; the
remaining inventory does not multiply the reserve. Streamed file transfers keep the default 512 MiB
allowance, and can grow with the parent's offered retention budget (8x byte
expansion), up to 2 GiB. Individual records remain bounded to 8 MiB. The parent
still independently checks actual bytes, record overhead, digests and source
identities before adopting results.
A source that exhausts the analyzer heap does not terminate the CLI/MCP
process. Completed static facts are retained, other files can still be analyzed,
and unexamined semantic relationships remain explicit unknowns. The parent
also checks transfer expansion and accumulated results against its remaining
heap before accepting semantic facts. A resource-limited result has partial
coverage, the observed limits and an `analyze-javascript-source` unknown scope
with the specific failure, including captured diagnostics when available.
Resource failures do not increment syntax parse failures. There is no fixed
source-length cutoff.
Static observations also need memory when expanded into nodes, relationships
and Evidence. If that expansion does not fit, the file node retains
`static_analysis` and any admitted `semantic_module_analysis` observations,
including their exact values and source locations. Use the public modules item
view to read them. The graph marks those unexpanded relationships as unknown.
If cumulative result capacity is exhausted, analysis stops; inventoried but
unexamined files remain present with explicit unknown scopes.
### Large local JSON inputs
CLI workflows parse larger JSON input files incrementally from one verified
regular-file handle, without constructing a document-sized string. Smaller
files retain native JSON parsing. Both paths require strict JSON and valid
UTF-8, and return one complete value; the assembled object still needs memory.
An individual JSON string exceeding the native length limit or the available
heap headroom for assembly returns
`resource_constraint` with `input_reason: "too-large"`. Supply a smaller valid
value; for Evidence workflows, re-analyze a smaller selection of the original
target and use its complete Evidence. Splitting JSON text or trimming Evidence
fields does not produce a valid workflow input.
Evidence bundle and analysis snapshot file readers still decode whole
documents. Their runtime string-limit failures report a resource constraint
with the selected path, observed bytes and UTF-16 limit, rather than malformed
JSON.
## Verification boundary
The source-owned fixture covers an extracted directory, a direct ASAR with an
@@ -466,6 +572,12 @@ from unmatched nested targets rather than reporting array indices.
For imports and requires, the active Node conditions include `module-sync`;
earlier active conditions retain precedence over later ones.
Package self-references use the importer's enclosing package scope before
installed dependencies. ESM requires the parsed package name to equal the
metadata name; a scope-only name such as `@scope` cannot capture `@scope/pkg`.
CommonJS retains Node's raw-request name/prefix matching. Both paths preserve
the original specifier and select exports under their own loader conditions.
Exact package-private `#alias` specifiers resolve against the importer's
enclosing `package.json` `imports` map — the nearest same-container package
scope, stopping at a `node_modules` directory so a nested package cannot inherit
@@ -481,13 +593,24 @@ URL targets before selecting a later package-local file; an unsupported bare
package or recursive alias keeps its unresolved outcome rather than selecting
a later fallback.
ESM relative module paths and selected package exports targets use URL suffix handling
and one percent-decoding pass. CommonJS relative paths and legacy package main
ESM relative module paths, legacy package subpaths without exports, and selected
package exports targets use URL suffix handling and one percent-decoding pass.
Bare specifiers retain their complete query/fragment text for exports key matching.
For dependency lookup, ESM constructs a package metadata URL instead of treating
the parsed package name as a literal directory. A root request such as `legacy?x`
does not load either `node_modules/legacy?x` or `node_modules/legacy`; an exact
self-reference with that metadata name can still select its exports first.
Without an exports map, ESM subpaths select exact files; directory imports and
extension fallback are refused. CommonJS relative paths and legacy package main
fields retain literal filename punctuation. Exports targets must start with
`./` and exclude `.`, `..`, and `node_modules` path segments, including their
encoded forms, before URL normalization. Arrays skip invalid exports targets
in declared order; URL decoding and file lookup failures for a selected target
do not select a later entry. Rejected references retain the package metadata
in declared order, including URLs that escape the package root after normalization.
Selected package targets use the normalized URL path for lookup; tabs and
newlines in the URL text are removed by the URL parser. CommonJS package targets
reject encoded separators anywhere in the selected URL; ESM checks its pathname.
URL decoding and file lookup failures for a selected target do not select a
later entry. Rejected references retain the package metadata
path, original target, failed constraint, and importing source location.
HTML script references resolve to exact inventoried files after applying the