Compare commits
103
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c865e20a0b | ||
|
|
0fd4021842 | ||
|
|
4db6ff461f | ||
|
|
ce58aebc0e | ||
|
|
80f6365523 | ||
|
|
dabcd783d5 | ||
|
|
3d34185171 | ||
|
|
21650455a9 | ||
|
|
844fa52772 | ||
|
|
e36c82becd | ||
|
|
092de8f4b3 | ||
|
|
3f4ea32189 | ||
|
|
966dc3acde | ||
|
|
91f6840c35 | ||
|
|
d9b4ae182d | ||
|
|
8be7ad496c | ||
|
|
01762b10c6 | ||
|
|
29ff887f65 | ||
|
|
693ed332b5 | ||
|
|
d3088c3795 | ||
|
|
a82d14e4db | ||
|
|
5f4c7ff6df | ||
|
|
13e03e9436 | ||
|
|
4353089c67 | ||
|
|
57c5acac2e | ||
|
|
0ecf3f2438 | ||
|
|
a56f7bb668 | ||
|
|
ee46323b49 | ||
|
|
a99857df23 | ||
|
|
6b043dc3de | ||
|
|
4003151ec1 | ||
|
|
92dac97102 | ||
|
|
89ea2cad19 | ||
|
|
ea06264a56 | ||
|
|
59b4a3c319 | ||
|
|
ea1c164e5e | ||
|
|
dc676b4d85 | ||
|
|
ad3e41d46e | ||
|
|
17f704050f | ||
|
|
c8820e03db | ||
|
|
ab80ec640a | ||
|
|
9ce6ef895e | ||
|
|
b3e977e6d0 | ||
|
|
a99b2fd3b5 | ||
|
|
4e4b15b825 | ||
|
|
21dbcf7a14 | ||
|
|
76296c5a38 | ||
|
|
e38550462e | ||
|
|
80896a1d64 | ||
|
|
e5bc4e85ec | ||
|
|
38221cb12e | ||
|
|
3749ccec47 | ||
|
|
a67dbf6a48 | ||
|
|
bd3a9bbbaf | ||
|
|
74050978be | ||
|
|
b212e92f9b | ||
|
|
990e1ad8c1 | ||
|
|
76d8f68cb9 | ||
|
|
fe574c0c9f | ||
|
|
3a7a3074e3 | ||
|
|
83e52559b8 | ||
|
|
824ecf109d | ||
|
|
8b4c3de3b3 | ||
|
|
f998fcb1cc | ||
|
|
f88c99021b | ||
|
|
49e6f00ed0 | ||
|
|
af88184eee | ||
|
|
c1efa6467d | ||
|
|
1e51c732bb | ||
|
|
7956934965 | ||
|
|
db4e360846 | ||
|
|
7bb4ac219c | ||
|
|
3df67c640a | ||
|
|
044aa34ce6 | ||
|
|
31bfff6fdf | ||
|
|
026126eafe | ||
|
|
29cc1b8746 | ||
|
|
ad41afa2b5 | ||
|
|
69a8b3a954 | ||
|
|
4733fc810a | ||
|
|
d21132fc97 | ||
|
|
66c344e289 | ||
|
|
28bd0d990a | ||
|
|
2612e474b6 | ||
|
|
a11fbc0df9 | ||
|
|
639b73097b | ||
|
|
bb074d84cb | ||
|
|
204e2c42a1 | ||
|
|
87edf14940 | ||
|
|
890dfa49f6 | ||
|
|
bc7846db91 | ||
|
|
5485483f7d | ||
|
|
c916850852 | ||
|
|
26038912f6 | ||
|
|
aa7c5b7b5e | ||
|
|
095fad9406 | ||
|
|
8b2f8514c5 | ||
|
|
c57e759ae6 | ||
|
|
7b25a73ac4 | ||
|
|
aa257ea4b1 | ||
|
|
4999e4be2c | ||
|
|
92cc0edd8e | ||
|
|
573598c632 |
@@ -0,0 +1,134 @@
|
||||
---
|
||||
name: rea-changelog-update
|
||||
description: Prepare or rewrite REA release changelogs and GitHub release notes from pinned Git history, with verified contributor thanks and Release Please synchronization.
|
||||
---
|
||||
|
||||
# REA changelog update
|
||||
|
||||
Use this for release-note preparation and contributor attribution. Read
|
||||
[docs/releasing.md](../../../docs/releasing.md) from the repository root for
|
||||
publication authority and the existing checkpoint checks. This skill prepares
|
||||
notes; it does not authorize merging, tagging, or publishing.
|
||||
|
||||
## Inventory before prose
|
||||
|
||||
Fetch complete history and tags. Select the published baseline tag from
|
||||
`.release-please-manifest.json` on the release PR's **base branch**, and pin that
|
||||
branch's exact SHA as the editorial target. REA tags are `rea-agents-VERSION`.
|
||||
Use the proposed version, including its prerelease suffix; do not change the
|
||||
`always-bump-minor` strategy to imitate OpenClaw's calendar versions.
|
||||
|
||||
```bash
|
||||
node scripts/release-notes.mjs inventory \
|
||||
--repository morluto/rea --base rea-agents-PREVIOUS \
|
||||
--target FULL_BASE_BRANCH_SHA --version VERSION \
|
||||
--manifest /tmp/rea-VERSION.contributions.json
|
||||
node scripts/release-notes.mjs record \
|
||||
--manifest /tmp/rea-VERSION.contributions.json \
|
||||
--summary \
|
||||
--output /tmp/rea-VERSION.record.md
|
||||
```
|
||||
|
||||
Read the inventory, not just the existing bot draft. It includes side-branch
|
||||
commits, GitHub-associated PRs whose merge commits are in the range, commit
|
||||
authors/co-authors, and confirmed closing issues with their reporters. GitHub
|
||||
connections are paginated. Network errors stop generation; rerun into a new
|
||||
output file after recovery. Existing output files are never overwritten.
|
||||
|
||||
Unresolved commit identities are retained as names/emails with `user: null`;
|
||||
never guess a handle or thank the merger as a substitute. The report preserves
|
||||
unknowns. This is a reviewed metadata snapshot, not independent proof that a
|
||||
human authored a patch. For unattributed identities, unusual merge strategies,
|
||||
backports, reversions, or references that GitHub did not associate, inspect
|
||||
the source PRs and diffs and report the limitation. The current collector uses
|
||||
confirmed PR-closing relationships for issue-reporter credit; unrelated issue
|
||||
mentions are not automatically credited. Do not claim that merely inventoried
|
||||
work still ships after a revert.
|
||||
|
||||
## Write for users and retain credit
|
||||
|
||||
Rewrite only the newest `## [VERSION]` section in `CHANGELOG.md`, preserving its
|
||||
version/date/link heading. Use single-line Markdown list items under:
|
||||
|
||||
- `### Highlights`: a few release-defining user outcomes; no quota for small releases.
|
||||
- `### Changes`: capabilities and meaningful behavior changes.
|
||||
- `### Fixes`: reliability, correctness, compatibility, and safety fixes.
|
||||
- `### ⚠ BREAKING CHANGES`: retain when applicable, with migrations and original
|
||||
PR/commit references required by the existing checkpoint validator.
|
||||
|
||||
Group related work by analyst workflow or user impact. Explain the resulting
|
||||
behavior in short, concrete entries. Highlights summarize the release; avoid
|
||||
repeating their full descriptions in Changes. Internal cleanup, tests, CI, and
|
||||
routine documentation generally belong only in the provenance file. Read ambiguous PRs/diffs before making
|
||||
claims. Distinguish simulated-provider checks from real-provider verification.
|
||||
|
||||
Credit each represented contribution once in the public notes with its link
|
||||
and `Thanks @...` for its verified human author, known co-authors, and confirmed
|
||||
closing-issue reporters. Grouping preserves all those credits in one line.
|
||||
When Highlights, detailed entries or migrations reference the same work,
|
||||
put its thanks in one of those entries, not all of them. Include issue links.
|
||||
Direct-commit bullets carry the commit link and known human credit.
|
||||
Exclude GitHub bots and known agent accounts. REA's observed Anthropic and
|
||||
Cursor co-author trailers resolve to `@claude` and `@cursoragent` with GitHub
|
||||
type `User`; the credit policy excludes them while retaining their metadata.
|
||||
At the REA maintainer's request, omit `@morluto` self-thanks while preserving
|
||||
authorship in the inventory. Other human contributors remain eligible.
|
||||
Use explicit GitHub links for cross-repository references to avoid ambiguous
|
||||
`#NNN` numbers. Unknown handles stay unknown; they are not fabricated credit.
|
||||
|
||||
Append the generated `record --summary` provenance link verbatim at the end
|
||||
of this release. The linked inventory accounts for every discovered in-range
|
||||
PR and direct contribution, including internal work omitted from the prose.
|
||||
Do not paste the full `record` output into public notes: it duplicates thanks
|
||||
and obscures the user-facing changes. Full records remain available for review.
|
||||
Do not hand-edit generated credits to make validation pass. Refresh the inventory
|
||||
when product history changes; a prose-only correction can reuse the snapshot.
|
||||
|
||||
## Finalize with Release Please
|
||||
|
||||
Before changing the live release PR, have authorization for that PR update.
|
||||
Local preparation and validation can proceed independently.
|
||||
|
||||
1. Add `rea:release-notes-finalized` to the selected open release PR **before**
|
||||
editing its branch/body. Main proposal pushes respect this label. This is
|
||||
our workflow guard, not Release Please's `autorelease: snooze` mechanism.
|
||||
A proposal already running must finish before editing. Do not dispatch
|
||||
manual preparation over finalized notes.
|
||||
2. On that release branch, save the reviewed inventory as
|
||||
`docs/releases/VERSION.contributions.json`. This is durable public release
|
||||
provenance, not a temporary planning file. Do not add credentials or local
|
||||
environment dumps. Keep the existing version artifacts aligned.
|
||||
3. Run the existing checkpoint validator and the new checker:
|
||||
|
||||
```bash
|
||||
node scripts/release-notes.mjs check \
|
||||
--manifest docs/releases/VERSION.contributions.json
|
||||
node scripts/release-notes.mjs render \
|
||||
--manifest docs/releases/VERSION.contributions.json \
|
||||
--output /tmp/rea-VERSION.notes.md
|
||||
```
|
||||
|
||||
The checker supports inline PR bodies, including their surrounding metadata,
|
||||
up to GitHub's 65,536-character limit. It fails on larger notes rather than
|
||||
truncating credit; Release Please's overflow-link bodies are not integrated.
|
||||
|
||||
4. Update the release PR body with the rendered section while preserving its
|
||||
Release Please header/footer and component/version structure. Use a body
|
||||
file for `gh pr edit --body-file`; changing only `CHANGELOG.md` does not
|
||||
change Release Please's release body. Push only authorized changes.
|
||||
5. Freshly fetch the PR body to a file and run `check --pr-body FILE`. Review
|
||||
the complete diff and wait for current CI. Publication checks the body
|
||||
again before release creation. Avoid further body edits while publishing.
|
||||
|
||||
The finalization label freezes bot refreshes, not main. New product changes
|
||||
after the inventory target require refreshing the notes before publication.
|
||||
The workflow checks target ancestry and allows only release metadata paths
|
||||
after that target. It also requires the inventory target to equal the reviewed
|
||||
merge's first parent, so intervening main commits require a fresh inventory.
|
||||
Review diffs within the allowed release metadata paths too: the path guard does
|
||||
not prove that package dependency/script edits are version-only changes.
|
||||
To resume automatic proposals, remove the finalization label and let the bot
|
||||
regenerate; expect editorial changes to be replaced.
|
||||
|
||||
Existing releases without a contribution inventory retain their current
|
||||
publication path. Never retag or republish a package to correct release prose.
|
||||
@@ -135,6 +135,16 @@ session to a selected path. The analysis remains complete in the ledger; avoid
|
||||
repeating it merely to request the same oversized response. Complete CLI JSON
|
||||
output also streams without a single MCP frame.
|
||||
|
||||
When the retained parent ID is unknown, use `get_evidence_bundle` with
|
||||
`detail: "summary"` and exact filters for operation, target SHA-256,
|
||||
analysis-profile digest or canonical procedure address. Select its returned
|
||||
Evidence ID for the next view instead of requesting the complete bundle.
|
||||
`retention: "complete-record"` means the whole record is stored; native
|
||||
value-flow truncation and unknown metadata remain separate. For an exported
|
||||
portable bundle, use `rea inspect-evidence-bundle` with JSON containing `path`,
|
||||
`detail: "summary"` and the same `filters`. Reading that file still requires
|
||||
validating its full contents within the local memory budget.
|
||||
|
||||
A client can also truncate a successful tool result without a REA transport
|
||||
error. Treat that preview as incomplete; use the retained parent ID for smaller
|
||||
views or export when the full record is needed. Do not parse the preview as
|
||||
|
||||
@@ -38,3 +38,11 @@ or absence:
|
||||
Process captures are opt-in behavioral evidence, not a security sandbox. V3
|
||||
captures cannot be upgraded to V4; rerun the original scenario. Distinguish root
|
||||
exit from descendant settlement and require freshness when the task needs it.
|
||||
A timed-out target that must write a final report needs `finalization_ms`: REA
|
||||
sends SIGTERM to the captured root, keeps observing, and sends SIGKILL once
|
||||
only if the target has not exited by the end of the interval.
|
||||
`exit.finalization.signals` lists each attempt with its delivery result; the
|
||||
observed exit signal stays in `exit.signal`, `exit.code` is null for every
|
||||
deadline exit, and the code of a finalizer that exited on its own is in
|
||||
`exit.finalization.exit_code` (null on a signal exit or when no exit was
|
||||
observed).
|
||||
|
||||
@@ -9,6 +9,22 @@ that static analysis. Read findings, coverage, limitations and unknowns, then
|
||||
follow a specific unanswered question. Older servers without `detail` return
|
||||
the complete application Evidence inline when it fits.
|
||||
|
||||
When the connected schema advertises `max_heap_mb` and `analysis_timeout_ms`,
|
||||
these select the isolated source analyzer's old-space heap and per-source
|
||||
deadline. Defaults are 1024 MiB and 300000 ms. They do not change the MCP
|
||||
process heap. Resource-limited analyses preserve completed static facts and
|
||||
report partial semantic coverage with specific unknown scopes; this does not
|
||||
mean the source is malformed. Inspect those limits before retrying a source.
|
||||
When expanded relationships exceed the parent's available memory, inspect the
|
||||
file's modules item view for retained `static_analysis` and
|
||||
`semantic_module_analysis`. Their facts remain available even when graph links
|
||||
are unknown. Cumulative result exhaustion can stop analysis before every file
|
||||
is examined; the inventory and remaining unknown scopes distinguish that case.
|
||||
Timeout or cancellation after source analysis begins retains partial Evidence
|
||||
in the same session. A returned `error.details.partial_observation` reference
|
||||
can be supplied directly to `inspect_analysis_view`. If the SDK cancels locally, retrieve its recorded ID from
|
||||
the Evidence bundle before inspecting the partial analysis.
|
||||
|
||||
## Read a module without repeating analysis
|
||||
|
||||
Use the actual returned parent ID, on the same connection, as `source.evidence_id`.
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
self-hosted-runner:
|
||||
labels:
|
||||
- hopper
|
||||
- ghidra-12-1-2
|
||||
# Hosted label introduced after the pinned actionlint release.
|
||||
- macos-15-intel
|
||||
+188
-81
@@ -5,6 +5,10 @@ on:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
branches: [main, "release/*"]
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ci-${{ github.event.pull_request.number || github.ref }}
|
||||
@@ -36,37 +40,39 @@ jobs:
|
||||
name: Classify changes
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
code: ${{ steps.package.outputs.required }}
|
||||
package: ${{ steps.package.outputs.required }}
|
||||
code: ${{ steps.plan.outputs.code }}
|
||||
runtime: ${{ steps.plan.outputs.runtime }}
|
||||
package: ${{ steps.plan.outputs.package }}
|
||||
apple: ${{ steps.plan.outputs.apple }}
|
||||
fixtures: ${{ steps.plan.outputs.fixtures }}
|
||||
readiness: ${{ steps.plan.outputs.readiness }}
|
||||
docs: ${{ steps.plan.outputs.docs }}
|
||||
workflows: ${{ steps.plan.outputs.workflows }}
|
||||
browser: ${{ steps.plan.outputs.browser }}
|
||||
binary_layout: ${{ steps.plan.outputs.binary_layout }}
|
||||
evm: ${{ steps.plan.outputs.evm }}
|
||||
javascript_recovery: ${{ steps.plan.outputs.javascript_recovery }}
|
||||
recorded_crash: ${{ steps.plan.outputs.recorded_crash }}
|
||||
web_network: ${{ steps.plan.outputs.web_network }}
|
||||
web_runtime: ${{ steps.plan.outputs.web_runtime }}
|
||||
web_source_map: ${{ steps.plan.outputs.web_source_map }}
|
||||
nativeaot: ${{ steps.plan.outputs.nativeaot }}
|
||||
website: ${{ steps.plan.outputs.website }}
|
||||
matrix: ${{ steps.plan.outputs.matrix }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- name: Select package and Windows lanes
|
||||
id: package
|
||||
shell: bash
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version-file: .nvmrc
|
||||
- name: Select verification lanes
|
||||
id: plan
|
||||
env:
|
||||
BASE_SHA: ${{ github.event.pull_request.base.sha }}
|
||||
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
|
||||
run: |
|
||||
if [[ "${GITHUB_EVENT_NAME}" != "pull_request" ]]; then
|
||||
echo "required=true" >> "${GITHUB_OUTPUT}"
|
||||
# Compare the PR's changes, excluding unrelated updates on its base.
|
||||
elif git diff --quiet "${BASE_SHA}...${HEAD_SHA}" -- \
|
||||
. \
|
||||
':(exclude)docs/**' \
|
||||
':(top,exclude,glob)README*.md' \
|
||||
':(exclude)AGENTS.md' \
|
||||
':(exclude)CONTRIBUTING.md'; then
|
||||
echo "required=false" >> "${GITHUB_OUTPUT}"
|
||||
else
|
||||
diff_status=$?
|
||||
# A failed comparison must fail classification, not report a scope.
|
||||
if [[ "${diff_status}" -ne 1 ]]; then
|
||||
exit "${diff_status}"
|
||||
fi
|
||||
echo "required=true" >> "${GITHUB_OUTPUT}"
|
||||
fi
|
||||
CI_FULL: ${{ contains(github.event.pull_request.labels.*.name, 'ci:full') }}
|
||||
run: node scripts/ci/plan.mjs
|
||||
|
||||
static:
|
||||
name: Static checks
|
||||
@@ -84,17 +90,17 @@ jobs:
|
||||
node-version-file: .nvmrc
|
||||
cache: npm
|
||||
- run: npm ci
|
||||
- if: github.event_name == 'push' || needs.changes.outputs.code == 'true'
|
||||
- if: needs.changes.outputs.code == 'true'
|
||||
run: npm run check:ci
|
||||
- if: github.event_name == 'push' || needs.changes.outputs.code == 'true'
|
||||
- if: needs.changes.outputs.code == 'true'
|
||||
run: npm run verify:test-discovery
|
||||
- if: github.event_name == 'pull_request' && needs.changes.outputs.code != 'true'
|
||||
- if: needs.changes.outputs.code != 'true'
|
||||
run: npm run format:check
|
||||
|
||||
test-shard:
|
||||
name: Test shard (${{ matrix.shard }})
|
||||
needs: [changes, build]
|
||||
if: github.event_name == 'push' || needs.changes.outputs.code == 'true'
|
||||
if: needs.changes.outputs.code == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
@@ -109,14 +115,8 @@ jobs:
|
||||
- run: npm ci
|
||||
- uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: rea-dist
|
||||
path: dist
|
||||
- name: Generate test artifacts from the downloaded runtime
|
||||
run: |
|
||||
node scripts/generate-mcp-tool-catalog.mjs
|
||||
node scripts/generate-skill-metadata.mjs
|
||||
node scripts/generate-product-catalog.mjs
|
||||
node scripts/generate-completion-ledger.mjs
|
||||
name: rea-test-runtime
|
||||
path: .
|
||||
- run: npm run test:ci:shard:run -- --shard=${{ matrix.shard }}
|
||||
- uses: actions/upload-artifact@v4
|
||||
if: ${{ !cancelled() }}
|
||||
@@ -131,7 +131,7 @@ jobs:
|
||||
build:
|
||||
name: Build package
|
||||
needs: changes
|
||||
if: github.event_name == 'push' || needs.changes.outputs.code == 'true'
|
||||
if: needs.changes.outputs.code == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
@@ -140,11 +140,17 @@ jobs:
|
||||
node-version-file: .nvmrc
|
||||
cache: npm
|
||||
- run: npm ci
|
||||
- run: npm run build:cached
|
||||
- run: npm run test:prepare
|
||||
- uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: rea-dist
|
||||
path: dist
|
||||
name: rea-test-runtime
|
||||
path: |
|
||||
dist/
|
||||
skills/
|
||||
.cache/mcp-tool-catalog.json
|
||||
docs/public/product-catalog.json
|
||||
docs/verification/managed-conformance-*.json
|
||||
include-hidden-files: true
|
||||
if-no-files-found: error
|
||||
retention-days: 7
|
||||
|
||||
@@ -224,7 +230,7 @@ jobs:
|
||||
conformance-fixtures:
|
||||
name: Source-built conformance fixtures
|
||||
needs: changes
|
||||
if: github.event_name == 'push' || needs.changes.outputs.code == 'true'
|
||||
if: needs.changes.outputs.fixtures == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
@@ -235,7 +241,7 @@ jobs:
|
||||
windows-native-build:
|
||||
name: Build bundled Windows x64 native controls
|
||||
needs: changes
|
||||
if: github.event_name == 'push' || needs.changes.outputs.package == 'true'
|
||||
if: needs.changes.outputs.runtime == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
@@ -265,24 +271,14 @@ jobs:
|
||||
package-e2e:
|
||||
name: Native package baseline (${{ matrix.platform }} ${{ matrix.architecture }})
|
||||
needs: [changes, windows-native-build]
|
||||
if: github.event_name == 'push' || needs.changes.outputs.package == 'true'
|
||||
if: needs.changes.outputs.runtime == 'true'
|
||||
strategy:
|
||||
fail-fast: false
|
||||
max-parallel: 2
|
||||
matrix:
|
||||
include:
|
||||
- os: ubuntu-latest
|
||||
platform: linux
|
||||
architecture: x64
|
||||
- os: ubuntu-24.04-arm
|
||||
platform: linux
|
||||
architecture: arm64
|
||||
- os: macos-15
|
||||
platform: darwin
|
||||
architecture: arm64
|
||||
- os: macos-15-intel
|
||||
platform: darwin
|
||||
architecture: x64
|
||||
# The four platform jobs use independent hosted runners; keep each
|
||||
# runner's existing heap and threadpool limits while avoiding a queued
|
||||
# second platform wave.
|
||||
max-parallel: 4
|
||||
matrix: ${{ fromJSON(needs.changes.outputs.matrix) }}
|
||||
runs-on: ${{ matrix.os }}
|
||||
timeout-minutes: 20
|
||||
env:
|
||||
@@ -317,7 +313,7 @@ jobs:
|
||||
apple-artifacts:
|
||||
name: Real Apple artifact CLI/MCP E2E
|
||||
needs: changes
|
||||
if: github.event_name == 'push' || needs.changes.outputs.code == 'true'
|
||||
if: needs.changes.outputs.apple == 'true'
|
||||
runs-on: macos-15
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
@@ -327,30 +323,39 @@ jobs:
|
||||
node-version-file: .nvmrc
|
||||
cache: npm
|
||||
- run: npm ci
|
||||
- run: npm run verify:keyed-archive
|
||||
- run: npm run verify:asset-catalog
|
||||
- run: npm run verify:macos-bundle
|
||||
- name: Legacy simulator platform verification
|
||||
run: npm run verify:legacy-simulator
|
||||
- name: Native installer and compiled NIB hierarchy regressions
|
||||
run: npm run test:focused -- tests/boundary/filesystem/installScript.test.ts tests/boundary/filesystem/nibNativeHierarchy.test.ts tests/boundary/filesystem/interfaceBuilderBundle.test.ts
|
||||
- name: Real binary metadata and Interface Builder integration
|
||||
run: npm run verify:apple-dispatch && npm run verify:interface-builder
|
||||
- name: Native plist CLI selection regressions
|
||||
run: npm run test:focused -- tests/boundary/cli/nativePlistCli.test.ts tests/boundary/cli/nativePlistPrototypeKeyCli.test.ts tests/boundary/cli/nativePlistSelectionFailureCli.test.ts
|
||||
- name: Native Swift demangling CLI and MCP regressions
|
||||
run: npm run test:focused -- tests/boundary/cli/nativeDemangleCli.test.ts tests/boundary/mcp/nativeDemangleMcp.test.ts
|
||||
- name: Native signature target binding and CLI/MCP recovery regressions
|
||||
run: npm run test:focused -- tests/boundary/providers/native/nativeSignatureTargetBinding.test.ts tests/boundary/cli/nativeSignatureCli.test.ts tests/boundary/mcp/nativeSignatureTargetBindingMcp.test.ts
|
||||
- name: Process filesystem snapshot descriptor regressions
|
||||
run: npm run test:focused -- src/process/capture/FilesystemSnapshot.test.ts
|
||||
- name: Native Darwin ownership and process cleanup regressions
|
||||
run: npm run test:focused -- tests/boundary/process/darwinProcessRunTokenReader.test.ts tests/boundary/process/directAnalysisSignals.test.ts tests/boundary/process/processCaptureLifecycle.test.ts tests/boundary/process/processCli.test.ts tests/boundary/mcp/sessionMcp.test.ts
|
||||
- name: Verify Apple artifact formats and metadata
|
||||
run: |
|
||||
npm run verify:keyed-archive
|
||||
npm run verify:asset-catalog
|
||||
npm run verify:macos-bundle
|
||||
npm run verify:legacy-simulator
|
||||
npm run verify:apple-dispatch
|
||||
npm run verify:interface-builder
|
||||
- name: Run native Apple and process boundary regressions
|
||||
run: >-
|
||||
npm run test:focused --
|
||||
tests/boundary/filesystem/installScript.test.ts
|
||||
tests/boundary/filesystem/nibNativeHierarchy.test.ts
|
||||
tests/boundary/filesystem/interfaceBuilderBundle.test.ts
|
||||
tests/boundary/cli/nativePlistCli.test.ts
|
||||
tests/boundary/cli/nativePlistPrototypeKeyCli.test.ts
|
||||
tests/boundary/cli/nativePlistSelectionFailureCli.test.ts
|
||||
tests/boundary/cli/nativeDemangleCli.test.ts
|
||||
tests/boundary/mcp/nativeDemangleMcp.test.ts
|
||||
tests/boundary/providers/native/nativeSignatureTargetBinding.test.ts
|
||||
tests/boundary/cli/nativeSignatureCli.test.ts
|
||||
tests/boundary/mcp/nativeSignatureTargetBindingMcp.test.ts
|
||||
src/process/capture/FilesystemSnapshot.test.ts
|
||||
tests/boundary/process/darwinProcessRunTokenReader.test.ts
|
||||
tests/boundary/process/directAnalysisSignals.test.ts
|
||||
tests/boundary/process/processCaptureLifecycle.test.ts
|
||||
tests/boundary/process/processCli.test.ts
|
||||
tests/boundary/mcp/sessionMcp.test.ts
|
||||
|
||||
reconstruction-readiness:
|
||||
name: Reconstruction readiness
|
||||
needs: changes
|
||||
if: github.event_name == 'push' || needs.changes.outputs.code == 'true'
|
||||
if: needs.changes.outputs.readiness == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
@@ -379,7 +384,7 @@ jobs:
|
||||
windows-curated:
|
||||
name: Windows capability and package lane
|
||||
needs: [changes, windows-native-build]
|
||||
if: github.event_name == 'push' || needs.changes.outputs.package == 'true'
|
||||
if: needs.changes.outputs.package == 'true'
|
||||
runs-on: windows-latest
|
||||
timeout-minutes: 25
|
||||
steps:
|
||||
@@ -476,7 +481,8 @@ jobs:
|
||||
|
||||
docs:
|
||||
name: Docs Generation
|
||||
if: github.event_name == 'pull_request'
|
||||
needs: changes
|
||||
if: needs.changes.outputs.docs == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
@@ -496,3 +502,104 @@ jobs:
|
||||
docs/verification/managed-conformance-*.json
|
||||
skills/reverse-engineer-anything
|
||||
if-no-files-found: error
|
||||
|
||||
workflows:
|
||||
name: Validate workflows
|
||||
needs: changes
|
||||
if: needs.changes.outputs.workflows == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Validate workflow syntax and expressions
|
||||
run: go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.7 -shellcheck= -pyflakes=
|
||||
|
||||
browser:
|
||||
needs: changes
|
||||
if: needs.changes.outputs.browser == 'true'
|
||||
uses: ./.github/workflows/real-browser.yml
|
||||
|
||||
binary_layout:
|
||||
needs: changes
|
||||
if: needs.changes.outputs.binary_layout == 'true'
|
||||
uses: ./.github/workflows/real-binary-layout.yml
|
||||
|
||||
evm:
|
||||
needs: changes
|
||||
if: needs.changes.outputs.evm == 'true'
|
||||
uses: ./.github/workflows/real-evm-interface.yml
|
||||
|
||||
javascript_recovery:
|
||||
needs: changes
|
||||
if: needs.changes.outputs.javascript_recovery == 'true'
|
||||
uses: ./.github/workflows/real-javascript-recovery.yml
|
||||
|
||||
recorded_crash:
|
||||
needs: changes
|
||||
if: needs.changes.outputs.recorded_crash == 'true'
|
||||
uses: ./.github/workflows/real-recorded-crash.yml
|
||||
|
||||
web_network:
|
||||
needs: changes
|
||||
if: needs.changes.outputs.web_network == 'true'
|
||||
uses: ./.github/workflows/real-web-network-captures.yml
|
||||
|
||||
web_runtime:
|
||||
needs: changes
|
||||
if: needs.changes.outputs.web_runtime == 'true'
|
||||
uses: ./.github/workflows/real-web-runtime.yml
|
||||
|
||||
web_source_map:
|
||||
needs: changes
|
||||
if: needs.changes.outputs.web_source_map == 'true'
|
||||
uses: ./.github/workflows/real-web-source-map.yml
|
||||
|
||||
nativeaot:
|
||||
needs: changes
|
||||
if: needs.changes.outputs.nativeaot == 'true'
|
||||
uses: ./.github/workflows/nativeaot-fixtures.yml
|
||||
|
||||
website:
|
||||
needs: changes
|
||||
if: needs.changes.outputs.website == 'true'
|
||||
uses: ./.github/workflows/website-check.yml
|
||||
|
||||
required:
|
||||
name: CI required
|
||||
if: ${{ always() }}
|
||||
needs:
|
||||
[
|
||||
changes,
|
||||
pr-title,
|
||||
static,
|
||||
build,
|
||||
test-shard,
|
||||
test,
|
||||
conformance-fixtures,
|
||||
windows-native-build,
|
||||
package-e2e,
|
||||
apple-artifacts,
|
||||
reconstruction-readiness,
|
||||
windows-curated,
|
||||
docs,
|
||||
workflows,
|
||||
browser,
|
||||
binary_layout,
|
||||
evm,
|
||||
javascript_recovery,
|
||||
recorded_crash,
|
||||
web_network,
|
||||
web_runtime,
|
||||
web_source_map,
|
||||
nativeaot,
|
||||
website,
|
||||
]
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version-file: .nvmrc
|
||||
- name: Require every selected lane to pass
|
||||
env:
|
||||
CI_JOBS: ${{ toJSON(needs) }}
|
||||
run: node scripts/ci/required.mjs
|
||||
|
||||
@@ -2,12 +2,7 @@ name: Source-built NativeAOT fixtures
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
paths:
|
||||
- "bridge/ghidra/extensions/nativeaot/**"
|
||||
- "scripts/generate-nativeaot-fixtures.mjs"
|
||||
- "tests/fixtures/nativeaot/**"
|
||||
- ".github/workflows/nativeaot-fixtures.yml"
|
||||
workflow_call:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
@@ -7,36 +7,7 @@ on:
|
||||
description: "Also verify the complete 65,281-section result over MCP (slow)"
|
||||
type: boolean
|
||||
default: false
|
||||
pull_request:
|
||||
paths:
|
||||
- ".github/workflows/real-binary-layout.yml"
|
||||
- "bridge/pwntools/**"
|
||||
- "src/native/pwntools/**"
|
||||
- "src/domain/native/binaryLayout*"
|
||||
- "src/application/binaryDiagnostics/**"
|
||||
- "src/contracts/native/binaryDiagnosticsToolContracts.ts"
|
||||
- "src/composition/binaryDiagnostics.ts"
|
||||
- "src/server/registerBinaryDiagnosticsTools.ts"
|
||||
- "src/server/createServer.ts"
|
||||
- "src/cli/binaryDiagnosticsCommands.ts"
|
||||
- "src/cli.ts"
|
||||
- "src/cliCommandNames.ts"
|
||||
- "src/contracts/toolContracts.ts"
|
||||
- "src/contracts/toolEffects.ts"
|
||||
- "src/process/OwnedCommand.ts"
|
||||
- "src/process/ProviderProcess.ts"
|
||||
- "src/process/ProviderDeadline.ts"
|
||||
- "src/process/ProcessOwnership*.ts"
|
||||
- "src/process/PrivateRuntimeRoot.ts"
|
||||
- "src/artifacts/readStableArtifact*.ts"
|
||||
- "src/artifacts/ArtifactReader.ts"
|
||||
- "src/domain/analysisError*.ts"
|
||||
- "scripts/verify-binary-layout.mjs"
|
||||
- "scripts/lib/elf-section-name-fixtures.mjs"
|
||||
- "scripts/lib/large-result-mcp.mjs"
|
||||
- "scripts/fixtures/binary-layout*"
|
||||
- "package.json"
|
||||
- "package-lock.json"
|
||||
workflow_call:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
@@ -1,40 +1,7 @@
|
||||
name: Real browser verification
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- ".github/workflows/real-browser.yml"
|
||||
- "scripts/verify-real-browser.mjs"
|
||||
- "scripts/verify-browser-network.mjs"
|
||||
- "scripts/verify-browser-scripts.mjs"
|
||||
- "scripts/verify-browser-modules.mjs"
|
||||
- "scripts/verify-browser-dom-destinations.mjs"
|
||||
- "scripts/lib/browser-*.mjs"
|
||||
- "scripts/fixtures/browser-network-site.mjs"
|
||||
- "scripts/fixtures/browser-script-site.mjs"
|
||||
- "scripts/fixtures/browser-module-site.mjs"
|
||||
- "scripts/fixtures/browser-verifier-site.mjs"
|
||||
- "scripts/fixtures/page-cdp-proxy.mjs"
|
||||
- "src/browser/**"
|
||||
- "src/domain/browser*.ts"
|
||||
- "src/domain/webScript*.ts"
|
||||
- "src/domain/webModule*.ts"
|
||||
- "src/application/WebModule*.ts"
|
||||
- "src/composition/webModules.ts"
|
||||
- "src/artifacts/readStableArtifact.ts"
|
||||
- "src/artifacts/ArtifactPaths.ts"
|
||||
- "src/application/WebScriptExportService.ts"
|
||||
- "src/contracts/webScriptToolContracts.ts"
|
||||
- "src/server/registerWebScriptTool.ts"
|
||||
- "src/server/registerWebModuleTool.ts"
|
||||
- "src/server/createServer.ts"
|
||||
- "src/cli/webModuleCommands.ts"
|
||||
- "src/cli.ts"
|
||||
- "src/cliCommandNames.ts"
|
||||
- "src/contracts/toolEffects.ts"
|
||||
- "src/cli/webScriptCommands.ts"
|
||||
- "package.json"
|
||||
- "package-lock.json"
|
||||
workflow_call:
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
|
||||
@@ -2,33 +2,7 @@ name: Real offline EVM interface
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
paths:
|
||||
- ".github/workflows/real-evm-interface.yml"
|
||||
- "src/evm/**"
|
||||
- "src/domain/evm/**"
|
||||
- "src/application/evm/**"
|
||||
- "src/contracts/evm/**"
|
||||
- "src/composition/evm.ts"
|
||||
- "src/cli/evmCommands.ts"
|
||||
- "src/server/registerEvmTools.ts"
|
||||
- "src/server/createServer.ts"
|
||||
- "src/cli.ts"
|
||||
- "src/cliCommandNames.ts"
|
||||
- "src/contracts/toolContracts.ts"
|
||||
- "src/contracts/toolEffects.ts"
|
||||
- "src/process/OwnedCommand.ts"
|
||||
- "src/process/ProviderProcess.ts"
|
||||
- "src/process/ProcessOwnership*.ts"
|
||||
- "src/process/ProviderDeadline.ts"
|
||||
- "src/process/PrivateRuntimeRoot.ts"
|
||||
- "src/artifacts/readStableArtifact*.ts"
|
||||
- "src/artifacts/ArtifactReader.ts"
|
||||
- "scripts/verify-evm-interface.mjs"
|
||||
- "scripts/fixtures/evm-interface.sol"
|
||||
- "third_party/evmole/**"
|
||||
- "package.json"
|
||||
- "package-lock.json"
|
||||
workflow_call:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
@@ -1,32 +1,7 @@
|
||||
name: Real JavaScript recovery
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- ".github/workflows/real-javascript-recovery.yml"
|
||||
- "src/javascript/recovery/**"
|
||||
- "src/domain/javascript/**"
|
||||
- "src/application/javascript/JavaScriptRecovery*.ts"
|
||||
- "src/composition/javascriptRecovery.ts"
|
||||
- "src/contracts/javascript/javascriptRecoveryToolContracts.ts"
|
||||
- "src/cli/javascriptRecoveryCommands.ts"
|
||||
- "src/cli/javascriptApplicationAnalysis.ts"
|
||||
- "src/server/registerJavaScriptRecoveryTool.ts"
|
||||
- "src/server/createServer.ts"
|
||||
- "src/server/sessionAvailabilityPolicy.ts"
|
||||
- "src/application/CapabilityInventory.ts"
|
||||
- "src/cli.ts"
|
||||
- "src/cliCommandNames.ts"
|
||||
- "src/contracts/toolContracts.ts"
|
||||
- "src/contracts/toolEffects.ts"
|
||||
- "scripts/rea.mjs"
|
||||
- "src/process/**"
|
||||
- "src/artifacts/SafeOutputTree.ts"
|
||||
- "scripts/verify/javascript/recovery.mjs"
|
||||
- "scripts/fixtures/javascript-recovery/**"
|
||||
- "third_party/wakaru"
|
||||
- "package.json"
|
||||
- "package-lock.json"
|
||||
workflow_call:
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
|
||||
@@ -2,34 +2,7 @@ name: Real recorded crash evidence
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
paths:
|
||||
- ".github/workflows/real-recorded-crash.yml"
|
||||
- "bridge/pwntools/**"
|
||||
- "bridge/pwndbg/**"
|
||||
- "src/native/pwntools/**"
|
||||
- "src/native/pwndbg/**"
|
||||
- "src/domain/native/recordedCrash*"
|
||||
- "src/application/binaryDiagnostics/**"
|
||||
- "src/contracts/native/binaryDiagnosticsToolContracts.ts"
|
||||
- "src/contracts/toolEffects.ts"
|
||||
- "src/contracts/toolOutputSchemaGroups.ts"
|
||||
- "src/composition/binaryDiagnostics.ts"
|
||||
- "src/server/registerRecordedCrashTools.ts"
|
||||
- "src/server/createServer.ts"
|
||||
- "src/server/sessionAvailabilityPolicy.ts"
|
||||
- "src/cli/binaryDiagnosticsCommands.ts"
|
||||
- "src/cliCommandNames.ts"
|
||||
- "src/process/**"
|
||||
- "src/artifacts/readStableArtifact*.ts"
|
||||
- "src/domain/analysisError*.ts"
|
||||
- "scripts/verify-recorded-crash.mjs"
|
||||
- "scripts/verify/recorded-crash/**"
|
||||
- "scripts/lib/recorded-core-fixtures.py"
|
||||
- "scripts/fixtures/recorded-crash.c"
|
||||
- "third_party/pwndbg/**"
|
||||
- "package.json"
|
||||
- "package-lock.json"
|
||||
workflow_call:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
name: Real Termux browser verification
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: real-termux-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
env:
|
||||
HUSKY: "0"
|
||||
|
||||
jobs:
|
||||
real-termux:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 45
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Download the pinned Termux app and verify its release checksum
|
||||
shell: bash
|
||||
run: |
|
||||
mkdir -p "$RUNNER_TEMP/rea-termux"
|
||||
cd "$RUNNER_TEMP/rea-termux"
|
||||
release=https://github.com/termux/termux-app/releases/download/v0.118.3
|
||||
apk=termux-app_v0.118.3+github-debug_x86_64.apk
|
||||
sums=termux-app_v0.118.3+github-debug_sha256sums
|
||||
curl --fail --location --retry 3 --output "$apk" "$release/$apk"
|
||||
curl --fail --location --retry 3 --output "$sums" "$release/$sums"
|
||||
awk -v apk="$apk" '$2 == apk || $2 == "*" apk' "$sums" > selected.sha256
|
||||
test "$(wc -l < selected.sha256)" -eq 1
|
||||
sha256sum --check selected.sha256
|
||||
- name: Enable emulator acceleration
|
||||
run: |
|
||||
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
|
||||
sudo udevadm control --reload-rules
|
||||
sudo udevadm trigger --name-match=kvm
|
||||
- name: Build and capture a page inside Termux
|
||||
uses: reactivecircus/android-emulator-runner@4c44018e59b437e86cdfc41da381398f93ed8808 # v2
|
||||
with:
|
||||
api-level: 30
|
||||
arch: x86_64
|
||||
target: google_apis
|
||||
ram-size: 4096M
|
||||
heap-size: 512M
|
||||
disk-size: 8192M
|
||||
disable-animations: true
|
||||
script: bash scripts/verify/termux/emulator.sh
|
||||
- name: Upload Termux and Android diagnostics
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: termux-browser-diagnostics
|
||||
path: ${{ runner.temp }}/rea-termux/diagnostics
|
||||
if-no-files-found: warn
|
||||
@@ -2,38 +2,7 @@ name: Real historical web network captures
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
paths:
|
||||
- ".github/workflows/real-web-network-captures.yml"
|
||||
- "bridge/mitmproxy/**"
|
||||
- "src/browser/history/**"
|
||||
- "src/domain/webNetworkCapture*.ts"
|
||||
- "src/domain/analysisError*.ts"
|
||||
- "src/contracts/errorSchemas.ts"
|
||||
- "src/domain/explicitSensitive*.ts"
|
||||
- "src/application/WebNetworkCapture*"
|
||||
- "src/composition/webNetworkCaptures.ts"
|
||||
- "src/cli/webNetworkCaptureCommands.ts"
|
||||
- "src/server/registerWebNetworkCaptureTool.ts"
|
||||
- "src/server/createServer.ts"
|
||||
- "src/server/toolRegistrationOptions.ts"
|
||||
- "src/cli.ts"
|
||||
- "src/cliCommandNames.ts"
|
||||
- "src/contracts/toolContracts.ts"
|
||||
- "src/contracts/toolEffects.ts"
|
||||
- "src/process/OwnedCommand.ts"
|
||||
- "src/process/ProviderProcess.ts"
|
||||
- "src/process/ProviderDeadline.ts"
|
||||
- "src/process/ProcessOwnership*.ts"
|
||||
- "src/process/PrivateRuntimeRoot.ts"
|
||||
- "src/process/capture/ProcessCaptureLifecycle.ts"
|
||||
- "src/artifacts/readStableArtifact*.ts"
|
||||
- "src/artifacts/ArtifactReader.ts"
|
||||
- "src/contracts/webNetworkCaptureToolContracts.ts"
|
||||
- "scripts/verify-web-network-captures.mjs"
|
||||
- "scripts/fixtures/generate-mitmproxy-capture.py"
|
||||
- "package.json"
|
||||
- "package-lock.json"
|
||||
workflow_call:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
@@ -1,34 +1,6 @@
|
||||
name: Real website runtime attribution
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- ".github/workflows/real-web-runtime.yml"
|
||||
- "scripts/verify-browser-runtime.mjs"
|
||||
- "scripts/lib/browser-runtime-*.mjs"
|
||||
- "src/process/ProviderProcess.ts"
|
||||
- "src/process/ProcessOwnership*.ts"
|
||||
- "scripts/fixtures/browser-runtime-site.mjs"
|
||||
- "src/browser/execution/**"
|
||||
- "src/browser/CdpConnection.ts"
|
||||
- "src/browser/CdpEndpoint.ts"
|
||||
- "src/browser/CdpTarget*.ts"
|
||||
- "src/browser/CdpCaptureValues.ts"
|
||||
- "src/browser/CdpAuthorized*.ts"
|
||||
- "src/domain/webRuntime.ts"
|
||||
- "src/domain/webExecution.ts"
|
||||
- "src/domain/webEventListeners.ts"
|
||||
- "src/domain/browserObservationError*.ts"
|
||||
- "src/application/WebRuntime*.ts"
|
||||
- "src/composition/webRuntime.ts"
|
||||
- "src/contracts/webRuntimeToolContracts.ts"
|
||||
- "src/contracts/toolEffects.ts"
|
||||
- "src/server/registerWebRuntimeTools.ts"
|
||||
- "src/server/createServer.ts"
|
||||
- "src/cli/webRuntimeCommands.ts"
|
||||
- "src/cli/commandCancellation.ts"
|
||||
- "src/cli.ts"
|
||||
- "package.json"
|
||||
- "package-lock.json"
|
||||
workflow_call:
|
||||
workflow_dispatch:
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
@@ -1,27 +1,6 @@
|
||||
name: Real website source-map verification
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- ".github/workflows/real-web-source-map.yml"
|
||||
- "scripts/verify-browser-source-maps.mjs"
|
||||
- "scripts/lib/browser-source-map-e2e.mjs"
|
||||
- "scripts/fixtures/browser-source-map-site.mjs"
|
||||
- "src/javascript/sourceMaps/**"
|
||||
- "src/browser/assets/**"
|
||||
- "src/domain/webSource*.ts"
|
||||
- "src/domain/webScript*.ts"
|
||||
- "src/application/WebSourceLocation*.ts"
|
||||
- "src/application/ValidateSelectedWebScript.ts"
|
||||
- "src/composition/webSourceLocations.ts"
|
||||
- "src/server/registerWebSourceLocationTool.ts"
|
||||
- "src/server/createServer.ts"
|
||||
- "src/contracts/webScriptToolContracts.ts"
|
||||
- "src/process/**"
|
||||
- "src/cli/webSourceLocationCommands.ts"
|
||||
- "src/cli.ts"
|
||||
- "src/artifacts/readStableArtifact.ts"
|
||||
- "package.json"
|
||||
- "package-lock.json"
|
||||
workflow_call:
|
||||
workflow_dispatch:
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
@@ -35,7 +35,21 @@ jobs:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Respect finalized release notes
|
||||
id: editorial
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }}
|
||||
GH_REPO: ${{ github.repository }}
|
||||
run: |
|
||||
finalized=$(gh api --paginate "repos/${GH_REPO}/pulls?state=open&base=main&per_page=100" \
|
||||
--jq '.[] | select(.head.repo.full_name == .base.repo.full_name) | select(.head.ref | startswith("release-please--branches--main--")) | select(any(.labels[]; .name == "rea:release-notes-finalized")) | .number')
|
||||
if [[ -n "${finalized}" ]]; then
|
||||
echo "Finalized release notes on PR ${finalized}; automatic proposal refresh is paused." >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "frozen=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- uses: googleapis/release-please-action@v4
|
||||
if: steps.editorial.outputs.frozen != 'true'
|
||||
with:
|
||||
token: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }}
|
||||
target-branch: main
|
||||
@@ -80,6 +94,16 @@ jobs:
|
||||
node scripts/verify-release-checkpoint.mjs \
|
||||
--phase publish --release-branch main --source-sha "$SOURCE_SHA"
|
||||
|
||||
- name: Validate finalized notes before release creation
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }}
|
||||
GH_REPO: ${{ github.repository }}
|
||||
RELEASE_PR: ${{ github.event.pull_request.number }}
|
||||
SOURCE_SHA: ${{ github.event.pull_request.merge_commit_sha }}
|
||||
run: |
|
||||
node scripts/release-notes.mjs publication --repository "$GH_REPO" \
|
||||
--source "$SOURCE_SHA" --pr "$RELEASE_PR"
|
||||
|
||||
- uses: googleapis/release-please-action@v4
|
||||
id: release
|
||||
with:
|
||||
@@ -192,6 +216,16 @@ jobs:
|
||||
--phase "$RELEASE_PHASE" --release-branch "$RELEASE_BRANCH"
|
||||
--source-sha "$SOURCE_SHA"
|
||||
|
||||
- name: Validate manual finalized notes before release creation
|
||||
if: inputs.phase == 'publish'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }}
|
||||
GH_REPO: ${{ github.repository }}
|
||||
SOURCE_SHA: ${{ steps.branch.outputs.sha }}
|
||||
run: |
|
||||
node scripts/release-notes.mjs publication --repository "$GH_REPO" \
|
||||
--source "$SOURCE_SHA"
|
||||
|
||||
- uses: googleapis/release-please-action@v4
|
||||
id: release
|
||||
with:
|
||||
|
||||
@@ -1,12 +1,7 @@
|
||||
name: Website checks
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main, "release/*"]
|
||||
paths:
|
||||
- "website/**"
|
||||
- "scripts/*website*.py"
|
||||
- ".github/workflows/**"
|
||||
workflow_call:
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
@@ -35,5 +30,9 @@ jobs:
|
||||
sudo apt-get update
|
||||
sudo apt-get install --no-install-recommends -y libcairo2 fonts-dejavu-core
|
||||
python3 -m pip install -r website/requirements.txt
|
||||
- name: Set up Node.js for website script tests
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version-file: .nvmrc
|
||||
- name: Check shared HTML, links, assets and contributor workflows
|
||||
run: python3 scripts/website.py check
|
||||
|
||||
@@ -59,8 +59,15 @@ jobs:
|
||||
sudo apt-get update
|
||||
sudo apt-get install --no-install-recommends -y libcairo2 fonts-dejavu-core
|
||||
python3 -m pip install -r website/requirements.txt
|
||||
- name: Set up Node.js for website script tests
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version-file: .nvmrc
|
||||
- name: Check shared HTML, links, assets and contributor workflows
|
||||
run: python3 scripts/website.py check
|
||||
# An out-of-date translation only warns in checks; publish it with its notice.
|
||||
- name: Refresh translation notices
|
||||
run: python3 scripts/website.py sync
|
||||
- name: Record the published commit
|
||||
run: |
|
||||
python3 - <<'PY'
|
||||
|
||||
+2
-1
@@ -22,7 +22,8 @@
|
||||
],
|
||||
"max-params": ["warn", 4],
|
||||
"no-var": "warn",
|
||||
"no-var-requires": "error"
|
||||
"no-var-requires": "error",
|
||||
"typescript/no-import-type-side-effects": "error"
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
{
|
||||
".": "6.3.0"
|
||||
".": "6.4.0"
|
||||
}
|
||||
|
||||
@@ -1,5 +1,69 @@
|
||||
# Changelog
|
||||
|
||||
## [6.4.0](https://github.com/morluto/rea/compare/rea-agents-6.3.0...rea-agents-6.4.0) (2026-10-11)
|
||||
|
||||
### Highlights
|
||||
|
||||
- Android: added device inspection, split-APK acquisition, logs, screenshots, file transfers and explicit app lifecycle actions through ADB, plus manifest and localized-string decoding with Apktool. ([#1552](https://github.com/morluto/rea/pull/1552), [#1556](https://github.com/morluto/rea/pull/1556)) Thanks @3QL.
|
||||
- JEB: browse project units and decompile selected items through a caller-started local JEB MCP server. ([#1536](https://github.com/morluto/rea/pull/1536)) Thanks @3QL.
|
||||
- Large analyses: discover retained Evidence without loading every payload, and inspect JavaScript graphs, binary layouts and function dossiers through summaries and selected views. ([#1682](https://github.com/morluto/rea/pull/1682), [#1568](https://github.com/morluto/rea/pull/1568), [#1570](https://github.com/morluto/rea/pull/1570), [#1345](https://github.com/morluto/rea/pull/1345), [#1507](https://github.com/morluto/rea/pull/1507), [#1247](https://github.com/morluto/rea/issues/1247)) Thanks @jhuang-tw, @kaoru0822-kitauji, @N0zoM1z0.
|
||||
- JavaScript: isolate source analysis in an owned process with configurable heap and timeout limits; retain completed facts and explicit unknowns when analysis runs out of resources. ([#1651](https://github.com/morluto/rea/pull/1651), [#1563](https://github.com/morluto/rea/issues/1563), [#1565](https://github.com/morluto/rea/issues/1565)) Thanks @N0zoM1z0, @peter-research.
|
||||
- Ghidra: configure import language/compiler specifications and seed functions, code and labels before auto-analysis. ([#1667](https://github.com/morluto/rea/pull/1667)) Thanks @sqmagellan.
|
||||
- RPC discovery: recognize supported tRPC calls, HTTP-link URLs and named GraphQL operations in JavaScript applications and web bundles. ([#1670](https://github.com/morluto/rea/pull/1670)) Thanks @pompiliodonatoapp.
|
||||
|
||||
### Changes
|
||||
|
||||
- Added Ghidra support for Windows PE DLLs and bounded MIPS ELF analysis, including carved MIPS inputs with incomplete section tables. Windows P0 remains read-only. ([#1303](https://github.com/morluto/rea/pull/1303), [#1362](https://github.com/morluto/rea/pull/1362), [#1629](https://github.com/morluto/rea/pull/1629), [#1290](https://github.com/morluto/rea/issues/1290), [#718](https://github.com/morluto/rea/issues/718), [#1619](https://github.com/morluto/rea/issues/1619)) Thanks @bqw371, @jaelliot, @kvalls, @N0zoM1z0.
|
||||
- Enabled `set_address_name` and `set_addresses_names` for Ghidra sessions with mutation authority, including namespace-aware data labels and per-address batch results. ([#1635](https://github.com/morluto/rea/pull/1635), [#1634](https://github.com/morluto/rea/issues/1634)) Thanks @giri256.
|
||||
- Recovered additional signed AArch64 relative jump-table patterns. ([#1592](https://github.com/morluto/rea/pull/1592)) Thanks @RishTheFish21.
|
||||
- Added PE resource inspection and read-only NativeAOT metadata parsing for supported x64 PE32+/.NET 8 RTR 9.1 inputs. The NativeAOT parser’s Windows-host Ghidra workflow remains unverified on a real Windows host. ([#1482](https://github.com/morluto/rea/pull/1482), [76d8f68](https://github.com/morluto/rea/commit/76d8f68cb918dfb43869f2b649a0b6c0c1cf266d), [#1254](https://github.com/morluto/rea/issues/1254)) Thanks @N0zoM1z0, @thangldw.
|
||||
- Added Pi, Hermes and Qwen Code setup, including Hermes active-profile support. ([#1376](https://github.com/morluto/rea/pull/1376), [#1403](https://github.com/morluto/rea/pull/1403), [#1415](https://github.com/morluto/rea/pull/1415), [#1409](https://github.com/morluto/rea/issues/1409)) Thanks @rasatpetabit, @yiliang114, @zaydiscold.
|
||||
- Added `REA_MCP_INPUT_SCHEMA_PROFILE=compact` for size-capped MCP clients such as Kimi/Moonshot, while retaining canonical runtime validation. ([#1514](https://github.com/morluto/rea/pull/1514), [#1580](https://github.com/morluto/rea/pull/1580), [#1484](https://github.com/morluto/rea/issues/1484), [#1581](https://github.com/morluto/rea/issues/1581)) Thanks @haythemsellami, @N0zoM1z0.
|
||||
- Added live process-capture progress and optional `finalization_ms` to capture shutdown output before timeout escalation; cancellation remains immediate. ([#1483](https://github.com/morluto/rea/pull/1483), [#1583](https://github.com/morluto/rea/pull/1583), [#1252](https://github.com/morluto/rea/issues/1252), [#1253](https://github.com/morluto/rea/issues/1253), [modelcontextprotocol/typescript-sdk#2997](https://github.com/modelcontextprotocol/typescript-sdk/issues/2997)) Thanks @N0zoM1z0, @nikolay-dementiev, @sathariels.
|
||||
- Improved Codex onboarding and recovery from observed client, PATH and provider blockers. ([#1613](https://github.com/morluto/rea/pull/1613)) Thanks @N0zoM1z0.
|
||||
- Reduced redirected JSON and terminal output size, and deferred heavy imports until their commands need them. ([#1413](https://github.com/morluto/rea/pull/1413), [#1450](https://github.com/morluto/rea/pull/1450), [#1439](https://github.com/morluto/rea/pull/1439), [21dbcf7](https://github.com/morluto/rea/commit/21dbcf7a1458219989ad291e44ce7d017378b727)) Thanks @kaoru0822-kitauji.
|
||||
|
||||
### Fixes
|
||||
|
||||
- Electron analysis: follow renamed and default-imported APIs and optional calls, ignore overwritten identities and media preload attributes, and resolve packaged `app.asar` pages and unpacked companions. ([#1391](https://github.com/morluto/rea/pull/1391), [#1393](https://github.com/morluto/rea/pull/1393), [#1400](https://github.com/morluto/rea/pull/1400), [#1664](https://github.com/morluto/rea/pull/1664), [#1673](https://github.com/morluto/rea/pull/1673)) Thanks @kaoru0822-kitauji, @marnixpluim, @N0zoM1z0, @suninrain086.
|
||||
- JavaScript data flow: preserve returned/yielded reference uncertainty, invalidate property-stored aliases correctly, and retain bundled-module runtime scope. ([#1437](https://github.com/morluto/rea/pull/1437), [#1445](https://github.com/morluto/rea/pull/1445), [#1471](https://github.com/morluto/rea/pull/1471), [#1577](https://github.com/morluto/rea/pull/1577), [#1428](https://github.com/morluto/rea/issues/1428), [#1442](https://github.com/morluto/rea/issues/1442), [#1449](https://github.com/morluto/rea/issues/1449), [#1576](https://github.com/morluto/rea/issues/1576)) Thanks @CassieFox23, @dakjdakd, @N0zoM1z0.
|
||||
- JavaScript packages: resolve exact exports subpaths, preserve trailing-slash directory requests and exact CommonJS keys, and respect shadowed `require` bindings. ([#1424](https://github.com/morluto/rea/pull/1424), [#1615](https://github.com/morluto/rea/pull/1615), [5485483](https://github.com/morluto/rea/commit/5485483f7da2f6cb66441bb784aaf2a1a5f8731a), [8b2f851](https://github.com/morluto/rea/commit/8b2f8514c50e469c3add8ce9ba7ebebef888b14f), [f998fcb](https://github.com/morluto/rea/commit/f998fcb1cca9994c3589fb70d6e0127fc215f76f), [1e51c73](https://github.com/morluto/rea/commit/1e51c732bbf08191688d6526bf58ab2650b601e3), [#1365](https://github.com/morluto/rea/issues/1365)) Thanks @akshitkhokhani, @AWUQAQ, @immanuwell, @rudycelekli.
|
||||
- Large JavaScript inputs: handle oversized inline source maps, deep returns and reassignment loops; avoid argument expansion limits and yield during large analysis passes. ([#1386](https://github.com/morluto/rea/pull/1386), [#1368](https://github.com/morluto/rea/pull/1368), [#1390](https://github.com/morluto/rea/pull/1390), [#1539](https://github.com/morluto/rea/pull/1539), [#1525](https://github.com/morluto/rea/pull/1525), [#1542](https://github.com/morluto/rea/pull/1542), [#1545](https://github.com/morluto/rea/pull/1545), [#1557](https://github.com/morluto/rea/pull/1557), [#1380](https://github.com/morluto/rea/issues/1380), [#1401](https://github.com/morluto/rea/issues/1401), [#1538](https://github.com/morluto/rea/issues/1538), [#1462](https://github.com/morluto/rea/issues/1462)) Thanks @cold-atom, @KadirJeager, @kaoru0822-kitauji, @marnixpluim, @N0zoM1z0, @rudycelekli.
|
||||
- JavaScript receivers: expand escaped methods once, index returned-object targets, and avoid replaying earlier writes or duplicating retained graph allocations. ([#1517](https://github.com/morluto/rea/pull/1517), [#1537](https://github.com/morluto/rea/pull/1537), [#1444](https://github.com/morluto/rea/pull/1444), [#1601](https://github.com/morluto/rea/pull/1601), [#1495](https://github.com/morluto/rea/issues/1495)) Thanks @CassieFox23, @geekwolf, @kaoru0822-kitauji, @N0zoM1z0, @zeynelacikgoez.
|
||||
- Browser captures: retain WebSocket URLs, empty connections and frame order; compare only common known payload coverage and preserve incomplete WebMCP replay coverage. ([db4e360](https://github.com/morluto/rea/commit/db4e3608465486466591b594cf77037ac150c988), [aa257ea](https://github.com/morluto/rea/commit/aa257ea4b16f2d2a35c12df63359d650b48b1154), [e52c0a7](https://github.com/morluto/rea/commit/e52c0a72c59c740315e89149881a27ce4d89a11c))
|
||||
- Browser scenarios: seed storage in the correct partition, normalize loopback attach identities, retain declared-secret redaction and require verified lineage before Windows Electron signals. ([#1604](https://github.com/morluto/rea/pull/1604), [#1590](https://github.com/morluto/rea/pull/1590), [#1588](https://github.com/morluto/rea/pull/1588), [8b4c3de](https://github.com/morluto/rea/commit/8b4c3de3b3a3b580adec266f94f29ccedf1da228), [#1497](https://github.com/morluto/rea/issues/1497), [#1591](https://github.com/morluto/rea/issues/1591), [#1589](https://github.com/morluto/rea/issues/1589)) Thanks @Adolanium, @akushonkamen, @dakjdakd.
|
||||
- Added Termux Android browser-scenario support. ([#1655](https://github.com/morluto/rea/pull/1655)) Thanks @tyloryang.
|
||||
- Web inputs: accept BOM-prefixed HAR, preserve the first duplicate DOM attribute and resolve SVG/XLink URLs in HTML and XHTML. ([#1509](https://github.com/morluto/rea/pull/1509), [#1516](https://github.com/morluto/rea/pull/1516), [#1602](https://github.com/morluto/rea/pull/1602), [#1683](https://github.com/morluto/rea/pull/1683), [#1496](https://github.com/morluto/rea/issues/1496), [#1586](https://github.com/morluto/rea/issues/1586)) Thanks @CassieFox23, @kaoru0822-kitauji, @sathariels, @taljeri.
|
||||
- Screenshot comparisons: validate PNG chunks, distinguish corruption from unsupported encodings and bound comparison memory. ([4e4b15b](https://github.com/morluto/rea/commit/4e4b15b8250b29e176d4191c9610a3931b693474), [b3e977e](https://github.com/morluto/rea/commit/b3e977e6d073ab1783cece8581279f87fc360411)) Thanks @rudycelekli.
|
||||
- Archives: preserve case-distinct names, report collisions at their diverging segment, detect nested formats from bytes and reject Windows alternate-stream extraction paths. ([#1480](https://github.com/morluto/rea/pull/1480), [#1505](https://github.com/morluto/rea/pull/1505), [#1553](https://github.com/morluto/rea/pull/1553), [573598c](https://github.com/morluto/rea/commit/573598c6324698cc48ceafd24ed3864670e08d00), [#1470](https://github.com/morluto/rea/issues/1470), [#1491](https://github.com/morluto/rea/issues/1491)) Thanks @CassieFox23, @kaoru0822-kitauji.
|
||||
- Cleanup: retain failed readers, ASAR handles, output trees and provider workers for retry without discarding completed output or partial observations. ([ffe0574](https://github.com/morluto/rea/commit/ffe0574e17b9d973860371083cb1a351be912131), [de0d341](https://github.com/morluto/rea/commit/de0d341524dc68250a45b41eb7e49fcecc1e32a9), [3f8c51e](https://github.com/morluto/rea/commit/3f8c51e9b506d88fc41aded7b1a22b9a09037c26), [d1235fa](https://github.com/morluto/rea/commit/d1235fadc901e1fb9ab3149d53063306c8545414), [026126e](https://github.com/morluto/rea/commit/026126eafe741e5a8b857433587fd615cdd44c19))
|
||||
- Providers: canonicalize IDA workspace roots before identity checks, respect IDA cancellation, serialize JEB acquisition and correct Ghidra no-return/imported-symbol recovery. ([#1481](https://github.com/morluto/rea/pull/1481), [#1419](https://github.com/morluto/rea/pull/1419), [5f30ea6](https://github.com/morluto/rea/commit/5f30ea6ff2dd89aab7af6da6e8dfa5e508ed459b), [f88c990](https://github.com/morluto/rea/commit/f88c99021b88ce67e513025901f57478d8107d0c), [49e6f00](https://github.com/morluto/rea/commit/49e6f00ed08b225abf9ddb7c762553607fdab115)) Thanks @AS9929, @JonahSimon, @Y0UW1SH.
|
||||
- Binary metadata: respect PE/.NET section extents, bound binary-plist expansion, scope keyed-archive traversal and decode app `Info.plist` files without `plutil`. ([#1678](https://github.com/morluto/rea/pull/1678), [#1515](https://github.com/morluto/rea/pull/1515), [#1378](https://github.com/morluto/rea/pull/1378), [#1476](https://github.com/morluto/rea/pull/1476), [975dff2](https://github.com/morluto/rea/commit/975dff2e25b14b634e14dd5e0db1537d0ca9815b), [76296c5](https://github.com/morluto/rea/commit/76296c5a3868ba3b54be6ed3896719283d6de5f8)) Thanks @jayhemnani9910, @kaoru0822-kitauji, @rudycelekli.
|
||||
- Windows runtimes: preserve drive-root paths when creating private provider directories. ([ee46323](https://github.com/morluto/rea/commit/ee46323b493994902b04824a7dc8aae2a8117903))
|
||||
- Compatibility: load Windows native controls under Node-API hosts with renamed executables, preserve dylib directory requirements, support Wakaru 1.x and reject directory-valued Hopper launchers. ([#1531](https://github.com/morluto/rea/pull/1531), [#1623](https://github.com/morluto/rea/pull/1623), [#1540](https://github.com/morluto/rea/pull/1540), [#1426](https://github.com/morluto/rea/pull/1426)) Thanks @evenafterallisaid, @Fazmin, @pionxzh, @xeropresence.
|
||||
- Reference imports: normalize internal symlink lookup consistently, preserve exact paths and exclusion provenance, report root access failures and reject FIFO replacements without blocking. ([#1535](https://github.com/morluto/rea/pull/1535), [2ca3ad2](https://github.com/morluto/rea/commit/2ca3ad238608adfca80a7ff1aaadea6e59a58225), [6636045](https://github.com/morluto/rea/commit/6636045c9c5b472bba51afee9d434394861e93a4), [3967cfa](https://github.com/morluto/rea/commit/3967cfa43e786c65a57e89c265343983d19ef9d0), [f1c58cc](https://github.com/morluto/rea/commit/f1c58cc53fbd83b968e11b3e2eccebca51299c54)) Thanks @anujsingh-cse.
|
||||
- Setup: honor active configuration and skill roots, preserve configured MCP environments, validate Hermes/OMP profiles before writing, recognize Omarchy and retain recovery backups. ([#1504](https://github.com/morluto/rea/pull/1504), [#1607](https://github.com/morluto/rea/pull/1607), [#1528](https://github.com/morluto/rea/pull/1528), [#1562](https://github.com/morluto/rea/pull/1562), [#1662](https://github.com/morluto/rea/pull/1662), [a011485](https://github.com/morluto/rea/commit/a01148553e2a37c72ae78405418ab648876a96a6), [#1606](https://github.com/morluto/rea/issues/1606), [#1521](https://github.com/morluto/rea/issues/1521), [#1561](https://github.com/morluto/rea/issues/1561)) Thanks @Adkid-Zephyr, @CassieFox23, @LuisTaffelli, @N0zoM1z0, @peter-research.
|
||||
- Process capture: retain descendant lineage after launcher exit, preserve identity uncertainty before signaling and release terminals after observation failures. ([80896a1](https://github.com/morluto/rea/commit/80896a1d64b216ef893907249cd4252f86215d49), [c916850](https://github.com/morluto/rea/commit/c9168508527302ac57bc86385ccf0d824697b5dc)) Thanks @Y0UW1SH.
|
||||
- CLI/MCP errors: accept JSON files for web comparisons, preserve native option help and union correction details, remove invalid aliases and validate doctor against its discovered output contract. ([#1397](https://github.com/morluto/rea/pull/1397), [#1395](https://github.com/morluto/rea/pull/1395), [#1425](https://github.com/morluto/rea/pull/1425), [#1461](https://github.com/morluto/rea/pull/1461), [8b4c96e](https://github.com/morluto/rea/commit/8b4c96e6c6b3ed6806e4cd72b556b5db6114304a), [#1421](https://github.com/morluto/rea/issues/1421), [#1454](https://github.com/morluto/rea/issues/1454)) Thanks @Fazmin, @kaoru0822-kitauji, @N0zoM1z0.
|
||||
|
||||
### ⚠ BREAKING CHANGES
|
||||
|
||||
- WebSocket results: use `network.websocket_connections` instead of `network.websocket_events`. Connections contain `request_id`, `url` and `events`; allow `payload_bytes: null` when size is unknown. No legacy alias remains. ([db4e360](https://github.com/morluto/rea/commit/db4e3608465486466591b594cf77037ac150c988))
|
||||
- Direct resource APIs: pass a caller-owned `ArtifactResourceScope` to inventory/extraction, JavaScript analysis, reference imports and web exports; retain it through cleanup/shutdown. For `readJavaScriptArtifactFiles`, scope is argument three and abort signal is argument four. Keep each root inventory source’s single cleanup owner. Use `OwnedFileHandle` owners, injected ASAR `fd`/`read` handles instead of `createReadStream`, VCS `Result` values, and `WebScriptExportError`’s options object. CLI/MCP inputs for the scoped operations are unchanged. ([ffe0574](https://github.com/morluto/rea/commit/ffe0574e17b9d973860371083cb1a351be912131), [53e7fad](https://github.com/morluto/rea/commit/53e7fadbd078422ba3d55b5499a70d237c5f4908), [684199b](https://github.com/morluto/rea/commit/684199ba0be7822266d57ddb8a40e95f087fdc42), [be75ed5](https://github.com/morluto/rea/commit/be75ed5e4b4bfb146af497f8fa5d172a1ea19462), [be6272f](https://github.com/morluto/rea/commit/be6272f26115bd8e94f2ab0df4cf092004587a18), [e0e49e9](https://github.com/morluto/rea/commit/e0e49e9dd53d9d1f1d2330b23f7f880dc1c81774), [95a77b6](https://github.com/morluto/rea/commit/95a77b6e25420b3879bd0e32bdbb14d452b38776))
|
||||
- Saved reference graphs: exclusions require their matching `pattern` and distinguish `project-ignored` from `default-ignored`. Re-import graphs missing that provenance. ([6585293](https://github.com/morluto/rea/commit/6585293c562853b95c3a70df02c12ad8cc17afc4))
|
||||
- Terminal captures: keep initial and resized buffers within 1,000,000 cells using `columns × (rows + scrollback)`; oversized scenarios are rejected before launch. ([f9ea9a9](https://github.com/morluto/rea/commit/f9ea9a9268c854d31048da9f8a50cfa37703fe5a))
|
||||
- Optional agent-verification reports: accept schema version 4, nullable `expectedFirstTool` and `targetClarificationPassed`. Production CLI/MCP contracts are unchanged. ([#1613](https://github.com/morluto/rea/pull/1613))
|
||||
- Capture comparisons: recapture both sides with matching `literal-replacements-v1` semantics when custom normalization patterns are used. Legacy unversioned pattern captures remain readable but cannot be compared with new pattern captures; captures without patterns are unaffected. ([#1456](https://github.com/morluto/rea/pull/1456), [2b6f14f](https://github.com/morluto/rea/commit/2b6f14f528297cf12c776c960af0859f5cfbefbf)) Thanks @Madhavan006, @N0zoM1z0.
|
||||
- Artifact graph edges: read child artifact IDs and paths from the occurrence selected by `occurrence_id`, not edge fields `child_artifact_id`/`logical_path`. Earlier inventory Evidence no longer validates against the new commitment. ([#1493](https://github.com/morluto/rea/pull/1493)) Thanks @kaoru0822-kitauji.
|
||||
- Semantic graph schema: nodes and relations no longer include `identifier_strategy`, and relation IDs no longer hash it. Earlier Evidence no longer validates against this schema. ([#1474](https://github.com/morluto/rea/pull/1474)) Thanks @kaoru0822-kitauji.
|
||||
- Semantic identities: select IDs from the parent graph; read exact event names from `event_name`, endpoints from `endpoint` and literal values from `properties.value`. New analyses use digest-based IDs for long/reserved event and endpoint names and `value-sha256` literal role keys, with shorter labels; string literal labels now read `string literal`. Previously stored graphs retain their original IDs and remain valid workflow inputs. ([#1427](https://github.com/morluto/rea/pull/1427), [#1423](https://github.com/morluto/rea/pull/1423), [#1407](https://github.com/morluto/rea/pull/1407)) Thanks @N0zoM1z0.
|
||||
|
||||
### Contributors
|
||||
|
||||
- Thanks also to @BaaRai, @commit-regrets, @coygeek, @Franky100-pig, @inii-sayaa, @jopemachine, @JosephDemarest, @keemsisi, @Totopo27 for contributions included in this release.
|
||||
|
||||
Full contribution record: [336 commits and 163 merged PRs](https://github.com/morluto/rea/blob/rea-agents-6.4.0/docs/releases/6.4.0.contributions.json).
|
||||
|
||||
## [6.3.0](https://github.com/morluto/rea/compare/rea-agents-6.2.0...rea-agents-6.3.0) (2026-10-09)
|
||||
|
||||
|
||||
|
||||
@@ -148,10 +148,15 @@ Generated fixtures and manifests remain ignored.
|
||||
Use the [release guide](docs/releasing.md). Main pushes automatically refresh a
|
||||
Release Please PR with the next version and changelog. Review its migration
|
||||
notes, wait for the current CI checks to pass, and merge it into main to publish.
|
||||
|
||||
The Release workflow validates and tags that merge; both publishers build its
|
||||
exact SHA. Ordinary main pushes cannot publish packages. An optional manual
|
||||
`release/VERSION` checkpoint retains explicit preparation and publication.
|
||||
|
||||
For grouped release notes with verified contributor thanks, use the repository's
|
||||
`rea-changelog-update` skill in `.agents/skills/rea-changelog-update/`. Its inventory,
|
||||
credit checks, and finalization process are described in the release guide.
|
||||
|
||||
Keep new implementation commits on main for the next release. The workflow
|
||||
owns packaged-artifact verification, npm publication, the published CLI/MCP
|
||||
canary, and MCP Registry publication. Sync manual checkpoint metadata back to
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
import ghidra.app.script.GhidraScript;
|
||||
import ghidra.program.model.address.Address;
|
||||
import ghidra.program.model.address.AddressSpace;
|
||||
import ghidra.program.model.listing.Function;
|
||||
import ghidra.program.model.listing.Program;
|
||||
import ghidra.program.model.mem.MemoryBlock;
|
||||
import ghidra.program.model.symbol.SourceType;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.LinkOption;
|
||||
import java.nio.file.Path;
|
||||
import java.security.MessageDigest;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HexFormat;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* Apply caller-declared analysis seeds before default auto-analysis.
|
||||
*
|
||||
* The single argument is REA's canonical seed snapshot: one
|
||||
* {@code 0x<hex>\t(function|code|label)[\tname]} line per seed. Code seeds are
|
||||
* applied after functions (highest address first), then labels. The outcome is stored as a
|
||||
* program property that the bridge reports during its handshake.
|
||||
*/
|
||||
public final class ReaGhidraApplySeeds extends GhidraScript {
|
||||
static final String REPORT_OPTION = "REA analysis seeds";
|
||||
|
||||
private record Seed(Address address, String kind, String name) {}
|
||||
|
||||
@Override
|
||||
public void run() throws Exception {
|
||||
String[] args = getScriptArgs();
|
||||
if (args.length != 1) {
|
||||
throw new IllegalArgumentException("REA seed script requires exactly one seed file");
|
||||
}
|
||||
Path path = Path.of(args[0]);
|
||||
if (!Files.isRegularFile(path, LinkOption.NOFOLLOW_LINKS)) {
|
||||
throw new IllegalArgumentException("REA seed file is not a regular file");
|
||||
}
|
||||
byte[] bytes = Files.readAllBytes(path);
|
||||
String sha256 = HexFormat.of().formatHex(
|
||||
MessageDigest.getInstance("SHA-256").digest(bytes)
|
||||
);
|
||||
AddressSpace space = currentProgram.getAddressFactory().getDefaultAddressSpace();
|
||||
List<Seed> seeds = new ArrayList<>();
|
||||
for (String line : new String(bytes, StandardCharsets.UTF_8).split("\n")) {
|
||||
if (line.isEmpty()) continue;
|
||||
String[] fields = line.split("\t", -1);
|
||||
if (fields.length < 2 || fields.length > 3 || !fields[0].startsWith("0x")) {
|
||||
throw new IllegalArgumentException("REA seed snapshot is malformed");
|
||||
}
|
||||
Address address;
|
||||
try {
|
||||
address = space.getAddress(Long.parseUnsignedLong(fields[0].substring(2), 16));
|
||||
} catch (ghidra.program.model.address.AddressOutOfBoundsException e) {
|
||||
address = null;
|
||||
}
|
||||
seeds.add(new Seed(address, fields[1], fields.length == 3 ? fields[2] : null));
|
||||
}
|
||||
|
||||
// Highest address first: a function body computed before analysis stops at
|
||||
// an existing function entry, so later creations cannot swallow earlier ones.
|
||||
seeds.sort((left, right) -> {
|
||||
if (left.address() == null) return right.address() == null ? 0 : 1;
|
||||
if (right.address() == null) return -1;
|
||||
return right.address().compareTo(left.address());
|
||||
});
|
||||
int unmapped = 0;
|
||||
int logged = 0;
|
||||
int codeDecoded = 0, codeFailed = 0;
|
||||
int functionCreated = 0, functionExisting = 0, functionFailed = 0;
|
||||
int labelApplied = 0, labelFailed = 0;
|
||||
for (String pass : new String[] { "function", "code", "label" }) {
|
||||
for (Seed seed : seeds) {
|
||||
if (!seed.kind().equals(pass)) continue;
|
||||
monitor.checkCancelled();
|
||||
MemoryBlock block = seed.address() == null
|
||||
? null
|
||||
: currentProgram.getMemory().getBlock(seed.address());
|
||||
if (block == null || (!pass.equals("label") && !block.isInitialized())) {
|
||||
unmapped++;
|
||||
continue;
|
||||
}
|
||||
switch (pass) {
|
||||
case "code" -> {
|
||||
if (decoded(seed.address())) codeDecoded++;
|
||||
else {
|
||||
codeFailed++;
|
||||
if (logged++ < 20) println("REA seed failed: code " + seed.address() + " " + why(seed.address()));
|
||||
}
|
||||
}
|
||||
case "function" -> {
|
||||
Function existing = getFunctionAt(seed.address());
|
||||
if (existing != null) {
|
||||
functionExisting++;
|
||||
if (seed.name() != null) {
|
||||
existing.setName(seed.name(), SourceType.USER_DEFINED);
|
||||
}
|
||||
} else if (decoded(seed.address()) &&
|
||||
createFunction(seed.address(), seed.name()) != null) {
|
||||
functionCreated++;
|
||||
} else {
|
||||
functionFailed++;
|
||||
if (logged++ < 20) println("REA seed failed: function " + seed.address() + " " + why(seed.address()));
|
||||
}
|
||||
}
|
||||
default -> {
|
||||
try {
|
||||
createLabel(seed.address(), seed.name(), true, SourceType.USER_DEFINED);
|
||||
labelApplied++;
|
||||
} catch (Exception e) {
|
||||
labelFailed++;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
String report = "{\"format\":\"rea-ghidra-seeds-v1\",\"sha256\":\"" + sha256 + "\"" +
|
||||
",\"entries\":" + seeds.size() +
|
||||
",\"function_created\":" + functionCreated +
|
||||
",\"function_existing\":" + functionExisting +
|
||||
",\"function_failed\":" + functionFailed +
|
||||
",\"code_decoded\":" + codeDecoded +
|
||||
",\"code_failed\":" + codeFailed +
|
||||
",\"label_applied\":" + labelApplied +
|
||||
",\"label_failed\":" + labelFailed +
|
||||
",\"unmapped\":" + unmapped + "}";
|
||||
currentProgram.getOptions(Program.PROGRAM_INFO).setString(REPORT_OPTION, report);
|
||||
println("REA seeds applied: " + report);
|
||||
}
|
||||
|
||||
private String why(Address address) {
|
||||
if (getDataAt(address) != null) return "defined data " + getDataAt(address).getDataType().getName();
|
||||
var containing = getInstructionContaining(address);
|
||||
if (containing != null && !containing.getAddress().equals(address))
|
||||
return "inside instruction at " + containing.getAddress();
|
||||
Function owner = getFunctionContaining(address);
|
||||
if (owner != null) return "inside function " + owner.getEntryPoint();
|
||||
return getInstructionAt(address) == null ? "undecodable" : "function creation refused";
|
||||
}
|
||||
|
||||
/** Decode one instruction flow at an address, keeping an existing decode. */
|
||||
private boolean decoded(Address address) {
|
||||
if (getInstructionAt(address) != null) return true;
|
||||
if (getDataAt(address) != null || getInstructionContaining(address) != null) {
|
||||
return false;
|
||||
}
|
||||
disassemble(address);
|
||||
return getInstructionAt(address) != null;
|
||||
}
|
||||
}
|
||||
+524
-216
@@ -27,6 +27,7 @@ import java.util.Comparator;
|
||||
import java.util.HashSet;
|
||||
import java.util.IdentityHashMap;
|
||||
import java.util.Iterator;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
@@ -80,12 +81,14 @@ import ghidra.program.database.mem.FileBytes;
|
||||
import ghidra.program.model.address.SegmentedAddress;
|
||||
import ghidra.program.model.reloc.Relocation;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import ghidra.program.model.symbol.Namespace;
|
||||
import ghidra.program.model.symbol.RefType;
|
||||
import ghidra.program.model.symbol.Reference;
|
||||
import ghidra.program.model.symbol.ReferenceIterator;
|
||||
import ghidra.program.model.symbol.SourceType;
|
||||
import ghidra.program.model.symbol.Symbol;
|
||||
import ghidra.program.model.symbol.SymbolIterator;
|
||||
import ghidra.program.model.symbol.SymbolTable;
|
||||
import ghidra.program.model.pcode.FunctionPrototype;
|
||||
import ghidra.program.model.pcode.HighFunction;
|
||||
import ghidra.program.model.pcode.HighParam;
|
||||
@@ -375,7 +378,10 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
case "inspect_native_instruction" -> inspectNativeInstruction(request.params);
|
||||
case "resolve_native_call_targets" -> resolveNativeCallTargets(request.params);
|
||||
case "xrefs" -> xrefs(request.params);
|
||||
case "analyze_function" -> analyzeFunction(request.params);
|
||||
case "analyze_function" -> {
|
||||
FunctionAnalysisObservation observation = analyzeFunctionObservation(request.params);
|
||||
yield functionObservation(observation.dossier(), observation.limitations());
|
||||
}
|
||||
case "set_address_name", "set_addresses_names" -> {
|
||||
if (!descriptor.transport.equals("unix-socket"))
|
||||
throw new RequestFailure("method_unavailable", "Windows P0 does not admit database mutation");
|
||||
@@ -432,6 +438,11 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
result.add("target", target);
|
||||
JsonArray extensions = analysisExtensions.reports();
|
||||
if (extensions.size() > 0) result.add("analysis_extensions", extensions);
|
||||
// Written by ReaGhidraApplySeeds before auto-analysis; REA checks it
|
||||
// against the committed profile.
|
||||
String seeds = currentProgram.getOptions(ghidra.program.model.listing.Program.PROGRAM_INFO)
|
||||
.getString("REA analysis seeds", null);
|
||||
if (seeds != null) result.add("analysis_seeds", JsonParser.parseString(seeds));
|
||||
return result;
|
||||
}
|
||||
|
||||
@@ -650,10 +661,14 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
private JsonElement procedurePseudocode(JsonObject params) throws Exception {
|
||||
requireKeys(params, Set.of("document", "procedure"));
|
||||
requireDocument(params);
|
||||
String value = decompiledText(
|
||||
decompile(resolveProcedure(requireString(params, "procedure")))
|
||||
Function function = resolveProcedure(requireString(params, "procedure"));
|
||||
String value = decompiledText(decompile(function));
|
||||
return functionObservation(
|
||||
value == null ? JsonNull.INSTANCE : GSON.toJsonTree(value),
|
||||
noReturnCallLimitations(
|
||||
currentProgram.getListing().getInstructions(function.getBody(), true)
|
||||
)
|
||||
);
|
||||
return value == null ? JsonNull.INSTANCE : GSON.toJsonTree(value);
|
||||
}
|
||||
|
||||
private JsonElement procedureAssembly(JsonObject params) throws Exception {
|
||||
@@ -661,7 +676,10 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
requireDocument(params);
|
||||
Function function = resolveProcedure(requireString(params, "procedure"));
|
||||
InstructionScan scan = scanInstructions(function, Integer.MAX_VALUE);
|
||||
return GSON.toJsonTree(renderAssembly(scan.instructions));
|
||||
return functionObservation(
|
||||
GSON.toJsonTree(renderAssembly(scan.instructions)),
|
||||
noReturnCallLimitations(scan.instructions)
|
||||
);
|
||||
}
|
||||
|
||||
private JsonObject readFunctionInstructions(JsonObject params) throws Exception {
|
||||
@@ -679,11 +697,13 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
limitations.add(
|
||||
"The fast path does not invoke the decompiler or scan whole-program names and strings."
|
||||
);
|
||||
List<String> boundaryLimitations = noReturnCallLimitations(scan.instructions);
|
||||
for (String limitation : boundaryLimitations) limitations.add(limitation);
|
||||
JsonObject result = new JsonObject();
|
||||
result.add("procedure", procedureIdentity(function));
|
||||
result.add("instructions", items);
|
||||
result.add("limitations", limitations);
|
||||
return result;
|
||||
return functionObservation(result, boundaryLimitations);
|
||||
}
|
||||
|
||||
private JsonArray procedureCalls(JsonObject params, boolean callers) throws Exception {
|
||||
@@ -716,7 +736,12 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
result.addProperty("signature", function.getPrototypeString(false, true));
|
||||
result.add("locals", functionLocals(function));
|
||||
result.add("classification", functionClassification(function));
|
||||
return result;
|
||||
return functionObservation(
|
||||
result,
|
||||
noReturnCallLimitations(
|
||||
currentProgram.getListing().getInstructions(function.getBody(), true)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
private JsonObject procedureReferences(JsonObject params) throws Exception {
|
||||
@@ -1105,17 +1130,6 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
return requested;
|
||||
}
|
||||
|
||||
private void requireAvailableAddressName(Address address,
|
||||
ghidra.program.model.symbol.Namespace namespace, String name)
|
||||
throws ghidra.util.exception.DuplicateNameException {
|
||||
for (Symbol owner : currentProgram.getSymbolTable().getSymbols(name, namespace)) {
|
||||
if (!owner.getAddress().equals(Address.NO_ADDRESS) && !owner.getAddress().equals(address))
|
||||
throw new ghidra.util.exception.DuplicateNameException(
|
||||
"Symbol name " + name + " is already owned at " +
|
||||
canonicalAddress(owner.getAddress()) + " in namespace " + namespace.getName(true));
|
||||
}
|
||||
}
|
||||
|
||||
private static String requireText(JsonObject params, String field) {
|
||||
JsonElement value = params.get(field);
|
||||
if (value == null || !value.isJsonPrimitive() || !value.getAsJsonPrimitive().isString())
|
||||
@@ -1139,6 +1153,10 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
}
|
||||
|
||||
private JsonObject analyzeFunction(JsonObject params) throws Exception {
|
||||
return analyzeFunctionObservation(params).dossier();
|
||||
}
|
||||
|
||||
private FunctionAnalysisObservation analyzeFunctionObservation(JsonObject params) throws Exception {
|
||||
requireKeys(params, Set.of("procedure"));
|
||||
Function function = resolveProcedure(requireString(params, "procedure"));
|
||||
InstructionScan scan = scanInstructions(function, Integer.MAX_VALUE);
|
||||
@@ -1196,12 +1214,16 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
limitations.add(
|
||||
"Synthetic Ghidra entry-point references without actionable memory sources are omitted."
|
||||
);
|
||||
List<String> boundaryLimitations = noReturnCallLimitations(scan.instructions);
|
||||
for (String limitation : boundaryLimitations) {
|
||||
limitations.add(limitation);
|
||||
}
|
||||
if (decompilation != null && decompilation.getErrorMessage() != null &&
|
||||
!decompilation.getErrorMessage().isBlank()) {
|
||||
limitations.add("Ghidra decompiler diagnostic: " + decompilation.getErrorMessage());
|
||||
}
|
||||
result.add("limitations", limitations);
|
||||
return result;
|
||||
return new FunctionAnalysisObservation(result, boundaryLimitations);
|
||||
}
|
||||
|
||||
private JsonObject nativeValueFlow(DecompileResults decompilation) throws Exception {
|
||||
@@ -1445,19 +1467,24 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
boolean commit = false;
|
||||
try {
|
||||
Function function = currentProgram.getFunctionManager().getFunctionAt(address);
|
||||
var table = currentProgram.getSymbolTable();
|
||||
SymbolTable table = currentProgram.getSymbolTable();
|
||||
Symbol primary = function == null ? table.getPrimarySymbol(address) : function.getSymbol();
|
||||
boolean existing = primary != null && !primary.isDynamic();
|
||||
String expected = existing ? symbolLeafName(primary, name) : name;
|
||||
requireAvailableAddressName(address,
|
||||
existing ? primary.getParentNamespace() : currentProgram.getGlobalNamespace(), expected);
|
||||
Namespace namespace = primary == null
|
||||
? currentProgram.getGlobalNamespace()
|
||||
: primary.getParentNamespace();
|
||||
String expected = primary == null ? name : symbolLeafName(primary, name);
|
||||
for (Symbol symbol : table.getLabelOrFunctionSymbols(expected, namespace)) {
|
||||
if (!address.equals(symbol.getAddress())) return false;
|
||||
}
|
||||
if (function != null) {
|
||||
function.setName(expected, SourceType.USER_DEFINED);
|
||||
} else if (existing) {
|
||||
primary.setName(expected, SourceType.USER_DEFINED);
|
||||
} else {
|
||||
Symbol created = table.createLabel(address, expected, SourceType.USER_DEFINED);
|
||||
created.setPrimary();
|
||||
if (primary != null && !primary.isDynamic()) {
|
||||
primary.setName(expected, SourceType.USER_DEFINED);
|
||||
} else {
|
||||
Symbol created = table.createLabel(address, expected, SourceType.USER_DEFINED);
|
||||
created.setPrimary();
|
||||
}
|
||||
}
|
||||
invalidateAnalysisCaches();
|
||||
Symbol readback = currentProgram.getSymbolTable().getPrimarySymbol(address);
|
||||
@@ -1483,16 +1510,33 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
requireDocument(params);
|
||||
if (!params.has("names") || !params.get("names").isJsonObject())
|
||||
throw new RequestFailure("invalid_request", "names must be an object of address -> name");
|
||||
Map<Address, List<AddressNameRequest>> requestsByAddress = new LinkedHashMap<>();
|
||||
JsonObject result = new JsonObject();
|
||||
for (Map.Entry<String, JsonElement> entry : params.getAsJsonObject("names").entrySet()) {
|
||||
boolean ok;
|
||||
try {
|
||||
ok = applyAddressName(parseReaAddress(entry.getKey()), entry.getValue().getAsString());
|
||||
Address address = parseReaAddress(entry.getKey());
|
||||
requestsByAddress.computeIfAbsent(address, ignored -> new ArrayList<>())
|
||||
.add(new AddressNameRequest(entry.getKey(), address, entry.getValue().getAsString()));
|
||||
}
|
||||
catch (RequestFailure failure) {
|
||||
ok = false;
|
||||
result.addProperty(entry.getKey(), false);
|
||||
}
|
||||
result.addProperty(entry.getKey(), ok);
|
||||
}
|
||||
for (List<AddressNameRequest> requests : requestsByAddress.values()) {
|
||||
String name = requests.get(0).name;
|
||||
boolean conflictingAliasNames = requests.stream()
|
||||
.anyMatch(request -> !request.name.equals(name));
|
||||
boolean success = false;
|
||||
if (!conflictingAliasNames) {
|
||||
try {
|
||||
success = applyAddressName(requests.get(0).address, name);
|
||||
}
|
||||
catch (RequestFailure failure) {
|
||||
// Keep per-address failures in the existing batch result shape.
|
||||
}
|
||||
}
|
||||
for (AddressNameRequest request : requests)
|
||||
result.addProperty(request.inputAddress, success);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
@@ -1960,9 +2004,20 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
);
|
||||
JumpTable.LoadTable[] loadTables = jumpTable.getLoadTables();
|
||||
RecoveredAArch64RelativeTable recoveredAArch64 =
|
||||
loadTables.length == 0
|
||||
? recoverAArch64RelativeTable(jumpTable)
|
||||
: null;
|
||||
recoverAArch64RelativeTable(jumpTable);
|
||||
if (recoveredAArch64 != null && loadTables.length > 0) {
|
||||
// A decoded table only adds evidence when Ghidra's own load table
|
||||
// describes the same bytes; any disagreement keeps Ghidra's view.
|
||||
boolean agrees = false;
|
||||
for (JumpTable.LoadTable loadTable : loadTables) {
|
||||
if (loadTable.getAddress().equals(recoveredAArch64.tableAddress) &&
|
||||
loadTable.getSize() == recoveredAArch64.entrySize &&
|
||||
loadTable.getNum() == recoveredAArch64.mappings.size()) {
|
||||
agrees = true;
|
||||
}
|
||||
}
|
||||
if (!agrees) recoveredAArch64 = null;
|
||||
}
|
||||
List<Address> referencedDataAddresses = loadTables.length == 0
|
||||
? dispatchPathDataReferences(jumpTable)
|
||||
: List.of();
|
||||
@@ -1996,7 +2051,7 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
source.add("evidence", evidence);
|
||||
dataSources.add(source);
|
||||
}
|
||||
if (recoveredAArch64 != null) {
|
||||
if (recoveredAArch64 != null && loadTables.length == 0) {
|
||||
JsonObject source = new JsonObject();
|
||||
source.addProperty(
|
||||
"address",
|
||||
@@ -2011,9 +2066,16 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
inferenceEvidence(
|
||||
"jump-table",
|
||||
"ghidra-instruction-and-memory",
|
||||
"Decoded " + recoveredAArch64.mappings.size() +
|
||||
"Decoded " + recoveredAArch64.mappings.size() + " " +
|
||||
(recoveredAArch64.signedEntries ? "signed" : "unsigned") +
|
||||
" entries of size " + recoveredAArch64.entrySize + " bytes from " + canonicalAddress(recoveredAArch64.tableAddress) +
|
||||
"; the AArch64 LDRB-or-LDRH/ADR/ADD/BR chain uses a selector guarded by an unsigned upper-bound branch."
|
||||
"; the AArch64 ADRP/ADD table, indexed load, ADR base, ADD" +
|
||||
(recoveredAArch64.offsetShift == 0 ? "" : " LSL #" + recoveredAArch64.offsetShift) +
|
||||
"/BR chain uses a selector bounded by CMP #" + recoveredAArch64.compareImmediate +
|
||||
" and " + recoveredAArch64.guardMnemonic + " on one straight-line path" +
|
||||
(recoveredAArch64.nullTarget == null
|
||||
? "."
|
||||
: "; zero entries are routed by CBZ to " + canonicalAddress(recoveredAArch64.nullTarget) + ".")
|
||||
)
|
||||
);
|
||||
source.add("evidence", evidence);
|
||||
@@ -2052,6 +2114,33 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
result.add("data_sources", dataSources);
|
||||
Address[] targets = jumpTable.getCases();
|
||||
TypedJumpTableCases typedCases = typedJumpTableCases(jumpTable, caseTokens);
|
||||
// Table indices identify the register bounded by CMP. Typed labels may
|
||||
// instead identify a selector before normalization (for example, SUB
|
||||
// index, selector, #10). Do not merge those distinct value identities.
|
||||
if (recoveredAArch64 != null && recoveredAArch64.priorSelectorDefinition != null) {
|
||||
typedCases.limitations.add(
|
||||
"REA decoded the relative table, but the compared selector has an earlier register definition at " +
|
||||
canonicalAddress(recoveredAArch64.priorSelectorDefinition) + "; its table indices are not proven to use the typed switch selector's case values. Ghidra's typed labels are retained and additional case labels remain unknown."
|
||||
);
|
||||
recoveredAArch64 = null;
|
||||
}
|
||||
if (recoveredAArch64 != null) {
|
||||
for (Map.Entry<Long, Address> typed : typedCases.cases.entrySet()) {
|
||||
long value = typed.getKey();
|
||||
if (value < 0 || value >= recoveredAArch64.mappings.size()) {
|
||||
typedCases.limitations.add("Typed case labels and decoded table indices do not use a verified common selector; Ghidra's typed labels are retained.");
|
||||
recoveredAArch64 = null;
|
||||
break;
|
||||
}
|
||||
RecoveredAArch64Mapping decoded = recoveredAArch64.mappings.get((int) value);
|
||||
boolean divertedZero = decoded.nullEntry && typed.getValue().equals(recoveredAArch64.unreachableZeroTarget);
|
||||
if (!typed.getValue().equals(decoded.targetAddress) && !divertedZero) {
|
||||
typedCases.limitations.add("Typed case evidence and the decoded relative table disagree on a destination; Ghidra's typed labels are retained.");
|
||||
recoveredAArch64 = null;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
JsonArray mappings = new JsonArray();
|
||||
JsonArray defaultTargets = new JsonArray();
|
||||
for (Address target : typedCases.defaults) {
|
||||
@@ -2059,27 +2148,30 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
}
|
||||
Set<Address> coveredTargets = new HashSet<>(typedCases.cases.values());
|
||||
coveredTargets.addAll(typedCases.defaults);
|
||||
if (recoveredAArch64 != null && recoveredAArch64.unreachableZeroTarget != null) {
|
||||
Address unreachable = recoveredAArch64.unreachableZeroTarget;
|
||||
coveredTargets.add(unreachable);
|
||||
for (RecoveredAArch64Mapping recovered : recoveredAArch64.mappings) {
|
||||
if (recovered.nullEntry && unreachable.equals(typedCases.cases.get((long) recovered.tableIndex))) {
|
||||
typedCases.cases.remove((long) recovered.tableIndex);
|
||||
}
|
||||
}
|
||||
typedCases.limitations.add(
|
||||
"Ghidra recovered destination " + canonicalAddress(unreachable) +
|
||||
" as the zero-offset decode of the relative table, but the CBZ before the dispatch diverts zero entries to " +
|
||||
canonicalAddress(recoveredAArch64.nullTarget) + ", so the indirect branch never reaches it for an in-range case."
|
||||
);
|
||||
}
|
||||
if (recoveredAArch64 != null) {
|
||||
for (int index = 0; index < recoveredAArch64.mappings.size(); index += 1) {
|
||||
monitor.checkCancelled();
|
||||
RecoveredAArch64Mapping recovered = recoveredAArch64.mappings.get(index);
|
||||
if (
|
||||
typedCases.unknownTargets.contains(recovered.targetAddress) ||
|
||||
typedCases.ambiguousValues.contains((long) recovered.caseValue)
|
||||
typedCases.ambiguousValues.contains((long) recovered.tableIndex)
|
||||
) continue;
|
||||
Address typedTarget = typedCases.cases.get((long) recovered.caseValue);
|
||||
if (typedTarget != null && !typedTarget.equals(recovered.targetAddress)) {
|
||||
typedCases.cases.remove((long) recovered.caseValue);
|
||||
typedCases.ambiguousValues.add((long) recovered.caseValue);
|
||||
typedCases.unknownTargets.add(typedTarget);
|
||||
typedCases.unknownTargets.add(recovered.targetAddress);
|
||||
typedCases.limitations.add(
|
||||
"Typed case evidence and the verified relative table disagree on a destination; both destinations remain unknown for that case."
|
||||
);
|
||||
continue;
|
||||
}
|
||||
JsonObject mapping = new JsonObject();
|
||||
mapping.addProperty("case_value", recovered.caseValue);
|
||||
mapping.addProperty("case_value", recovered.tableIndex);
|
||||
String targetAddress = canonicalAddress(recovered.targetAddress);
|
||||
mapping.addProperty("target_address", targetAddress);
|
||||
mapping.addProperty("confidence", "high");
|
||||
@@ -2088,14 +2180,18 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
inferenceEvidence(
|
||||
"jump-table",
|
||||
"ghidra-instruction-and-memory",
|
||||
"Case value " + recovered.caseValue + (recoveredAArch64.entrySize == 1 ? " indexes byte " : " indexes halfword at byte offset ") +
|
||||
(recovered.caseValue * recoveredAArch64.entrySize) + " at " + canonicalAddress(recoveredAArch64.tableAddress) +
|
||||
"; the loaded unsigned entry selects a four-byte instruction offset from " +
|
||||
canonicalAddress(recoveredAArch64.branchBase) + ", reaching " + targetAddress + "."
|
||||
"Case value " + recovered.tableIndex + (recoveredAArch64.entrySize == 1 ? " indexes byte " : " indexes halfword at byte offset ") +
|
||||
(recovered.tableIndex * recoveredAArch64.entrySize) + " at " + canonicalAddress(recoveredAArch64.tableAddress) +
|
||||
(recovered.nullEntry
|
||||
? "; the entry is zero, so the CBZ before the dispatch branches to " + targetAddress + "."
|
||||
: "; the loaded " + (recoveredAArch64.signedEntries ? "signed" : "unsigned") +
|
||||
" entry " + recovered.encodedEntry + " selects a " +
|
||||
(recoveredAArch64.offsetShift == 2 ? "four-byte instruction" : "byte") +
|
||||
" offset from " + canonicalAddress(recoveredAArch64.branchBase) + ", reaching " + targetAddress + ".")
|
||||
)
|
||||
);
|
||||
mapping.add("evidence", evidence);
|
||||
if (!typedCases.cases.containsKey((long) recovered.caseValue)) {
|
||||
if (!typedCases.cases.containsKey((long) recovered.tableIndex)) {
|
||||
mappings.add(mapping);
|
||||
}
|
||||
coveredTargets.add(recovered.targetAddress);
|
||||
@@ -2169,7 +2265,7 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
);
|
||||
}
|
||||
if (recoveredAArch64 == null && loadTables.length == 0 && currentProgram.getLanguageID().getIdAsString().startsWith("AARCH64:")) {
|
||||
limitations.add("REA did not verify a supported byte/halfword relative dispatch chain with exact unsigned bounds and register definitions; unsupported encodings and ambiguous bounds remain unknown.");
|
||||
limitations.add("REA did not verify a supported signed or unsigned byte/halfword relative dispatch chain with exact unsigned bounds and register definitions; unsupported encodings and ambiguous bounds remain unknown.");
|
||||
}
|
||||
if (targets.length == 0) {
|
||||
limitations.add(
|
||||
@@ -2299,12 +2395,21 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
}
|
||||
|
||||
private static final class RecoveredAArch64Mapping {
|
||||
private final int caseValue;
|
||||
private final int tableIndex;
|
||||
private final Address targetAddress;
|
||||
private final long encodedEntry;
|
||||
private final boolean nullEntry;
|
||||
|
||||
private RecoveredAArch64Mapping(int caseValue, Address targetAddress) {
|
||||
this.caseValue = caseValue;
|
||||
private RecoveredAArch64Mapping(
|
||||
int tableIndex,
|
||||
Address targetAddress,
|
||||
long encodedEntry,
|
||||
boolean nullEntry
|
||||
) {
|
||||
this.tableIndex = tableIndex;
|
||||
this.targetAddress = targetAddress;
|
||||
this.encodedEntry = encodedEntry;
|
||||
this.nullEntry = nullEntry;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2312,24 +2417,70 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
private final Address tableAddress;
|
||||
private final Address branchBase;
|
||||
private final int entrySize;
|
||||
private final Address priorSelectorDefinition;
|
||||
private final boolean signedEntries;
|
||||
private final int offsetShift;
|
||||
private final String guardMnemonic;
|
||||
private final long compareImmediate;
|
||||
private final Address nullTarget;
|
||||
private final Address unreachableZeroTarget;
|
||||
private final List<RecoveredAArch64Mapping> mappings;
|
||||
|
||||
private RecoveredAArch64RelativeTable(
|
||||
Address tableAddress,
|
||||
Address branchBase,
|
||||
int entrySize,
|
||||
Address priorSelectorDefinition,
|
||||
boolean signedEntries,
|
||||
int offsetShift,
|
||||
String guardMnemonic,
|
||||
long compareImmediate,
|
||||
Address nullTarget,
|
||||
Address unreachableZeroTarget,
|
||||
List<RecoveredAArch64Mapping> mappings
|
||||
) {
|
||||
this.tableAddress = tableAddress;
|
||||
this.branchBase = branchBase;
|
||||
this.entrySize = entrySize;
|
||||
this.priorSelectorDefinition = priorSelectorDefinition;
|
||||
this.signedEntries = signedEntries;
|
||||
this.offsetShift = offsetShift;
|
||||
this.guardMnemonic = guardMnemonic;
|
||||
this.compareImmediate = compareImmediate;
|
||||
this.nullTarget = nullTarget;
|
||||
this.unreachableZeroTarget = unreachableZeroTarget;
|
||||
this.mappings = mappings;
|
||||
}
|
||||
}
|
||||
|
||||
private static final Pattern AARCH64_DISPATCH_ADD = Pattern.compile(
|
||||
"(?i)^([wx]\\d+),\\s*([wx]\\d+),\\s*([wx]\\d+)(?:,\\s*LSL\\s*#(?:0x)?([0-9a-f]+))?$"
|
||||
);
|
||||
private static final Pattern AARCH64_TABLE_LOAD = Pattern.compile(
|
||||
"(?i)^([wx]\\d+),\\s*\\[([wx]\\d+),\\s*([wx]\\d+)(?:,\\s*(LSL|UXTW)\\s*(?:#(?:0x)?([0-9a-f]+))?)?\\s*\\]$"
|
||||
);
|
||||
private static final Pattern AARCH64_REGISTER_AND_ADDRESS = Pattern.compile(
|
||||
"(?i)^([wx]\\d+),\\s*(0x[0-9a-f]+)$"
|
||||
);
|
||||
private static final Pattern AARCH64_ADD_IMMEDIATE = Pattern.compile(
|
||||
"(?i)^([wx]\\d+),\\s*([wx]\\d+),\\s*#?(0x[0-9a-f]+|[0-9]+)$"
|
||||
);
|
||||
private static final Pattern AARCH64_COMPARE_IMMEDIATE = Pattern.compile(
|
||||
"(?i)^([wx]\\d+),\\s*#?(0x[0-9a-f]+|[0-9]+)$"
|
||||
);
|
||||
private static final Pattern AARCH64_MOVE_REGISTER = Pattern.compile(
|
||||
"(?i)^([wx]\\d+),\\s*([wx]\\d+)$"
|
||||
);
|
||||
private static final Set<String> AARCH64_FLAG_PRESERVING =
|
||||
Set.of("MOV", "ADRP", "ADR", "ADD");
|
||||
|
||||
/**
|
||||
* Recover only the directly verified AArch64 byte/halfword-indexed relative branch
|
||||
* table form. Other instruction sequences keep their case labels unknown.
|
||||
* Recover only directly verified AArch64 relative branch tables:
|
||||
* ADRP/ADD table base, LDRB/LDRH/LDRSB/LDRSH entry load with an exact
|
||||
* index scale, ADR branch base, optional CBZ on the loaded entry, and
|
||||
* ADD (byte or LSL #2 scale)/BR, with a selector bounded by CMP #imm and an
|
||||
* unsigned B.HI or B.CS/B.HS guard on one straight-line path. Any other
|
||||
* instruction sequence keeps its case labels unknown.
|
||||
*/
|
||||
private RecoveredAArch64RelativeTable recoverAArch64RelativeTable(
|
||||
JumpTable jumpTable
|
||||
@@ -2351,167 +2502,265 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
if (instruction.getAddress().equals(jumpTable.getSwitchAddress())) break;
|
||||
}
|
||||
int branchIndex = instructions.size() - 1;
|
||||
if (branchIndex < 1 ||
|
||||
!instructions.get(branchIndex).getMnemonicString().equalsIgnoreCase("BR")) {
|
||||
if (branchIndex < 1 || !mnemonic(instructions.get(branchIndex)).equals("BR")) {
|
||||
return null;
|
||||
}
|
||||
Set<String> recoveredTargetSet = new HashSet<>();
|
||||
for (Address target : jumpTable.getCases()) recoveredTargetSet.add(canonicalAddress(target));
|
||||
if (recoveredTargetSet.isEmpty()) return null;
|
||||
return matchAArch64RelativeDispatch(
|
||||
instructions,
|
||||
branchIndex,
|
||||
recoveredTargetSet,
|
||||
dispatchPathDataReferences(jumpTable)
|
||||
);
|
||||
}
|
||||
|
||||
private RecoveredAArch64RelativeTable matchAArch64RelativeDispatch(
|
||||
List<Instruction> instructions,
|
||||
int branchIndex,
|
||||
Set<String> recoveredTargetSet,
|
||||
List<Address> dataAddresses
|
||||
) throws Exception {
|
||||
String branchRegister = normalizedRegister(operands(instructions.get(branchIndex)));
|
||||
|
||||
// ADD Xd, Xn, Xm{, LSL #s}; BR Xd. LSL applies to Xm only.
|
||||
Instruction dispatchAdd = instructions.get(branchIndex - 1);
|
||||
if (!mnemonic(dispatchAdd).equals("ADD")) return null;
|
||||
java.util.regex.Matcher addMatcher = AARCH64_DISPATCH_ADD.matcher(operands(dispatchAdd));
|
||||
if (!addMatcher.matches() || !addMatcher.group(1).toLowerCase(Locale.ROOT).startsWith("x")) return null;
|
||||
if (!normalizedRegister(addMatcher.group(1)).equals(branchRegister)) return null;
|
||||
int offsetShift = addMatcher.group(4) == null ? 0 : Integer.parseInt(addMatcher.group(4), 16);
|
||||
if (offsetShift != 0 && offsetShift != 2) return null;
|
||||
String firstSource = normalizedRegister(addMatcher.group(2));
|
||||
String secondSource = normalizedRegister(addMatcher.group(3));
|
||||
if (firstSource.equals(secondSource)) return null;
|
||||
|
||||
// The entry register is the source whose latest definition is the
|
||||
// table load; the other source must be defined by ADR.
|
||||
int loadIndex = -1;
|
||||
String entryRegister = null;
|
||||
String baseRegister = null;
|
||||
for (String candidate : offsetShift == 0 ? List.of(secondSource, firstSource) : List.of(secondSource)) {
|
||||
int writer = latestWriter(instructions, branchIndex - 1, candidate);
|
||||
if (writer >= 0 && Set.of("LDRB", "LDRH", "LDRSB", "LDRSH").contains(mnemonic(instructions.get(writer)))) {
|
||||
loadIndex = writer;
|
||||
entryRegister = candidate;
|
||||
baseRegister = candidate.equals(firstSource) ? secondSource : firstSource;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (loadIndex < 0) return null;
|
||||
|
||||
Instruction load = instructions.get(loadIndex);
|
||||
String loadMnemonic = mnemonic(load);
|
||||
int entrySize = loadMnemonic.endsWith("B") ? 1 : 2;
|
||||
boolean signedEntries = loadMnemonic.startsWith("LDRS");
|
||||
java.util.regex.Matcher loadMatcher = AARCH64_TABLE_LOAD.matcher(operands(load));
|
||||
if (!loadMatcher.matches()) return null;
|
||||
// A signed load into a W register sign-extends only to 32 bits; the
|
||||
// 64-bit dispatch ADD would then read a zero-extended value.
|
||||
if (signedEntries && !loadMatcher.group(1).toLowerCase(Locale.ROOT).startsWith("x")) return null;
|
||||
String tableBaseRegister = normalizedRegister(loadMatcher.group(2));
|
||||
String indexRegister = normalizedRegister(loadMatcher.group(3));
|
||||
boolean wordIndex = loadMatcher.group(3).toLowerCase(Locale.ROOT).startsWith("w");
|
||||
String extend = loadMatcher.group(4) == null ? null : loadMatcher.group(4).toUpperCase(Locale.ROOT);
|
||||
int indexShift = loadMatcher.group(5) == null ? 0 : Integer.parseInt(loadMatcher.group(5), 16);
|
||||
if (wordIndex ? !"UXTW".equals(extend) : "UXTW".equals(extend)) return null;
|
||||
if (indexShift != (entrySize == 1 ? 0 : 1)) return null;
|
||||
|
||||
// Only ADR and one CBZ on the entry may sit between load and ADD.
|
||||
Address nullTarget = null;
|
||||
for (int index = loadIndex + 1; index < branchIndex - 1; index += 1) {
|
||||
Instruction instruction = instructions.get(index);
|
||||
String name = mnemonic(instruction);
|
||||
if (name.equals("ADR")) continue;
|
||||
if (name.equals("CBZ") && nullTarget == null) {
|
||||
String[] parts = operands(instruction).split(",");
|
||||
Address[] flows = instruction.getFlows();
|
||||
if (parts.length != 2 || flows.length != 1 ||
|
||||
!normalizedRegister(parts[0]).equals(entryRegister)) return null;
|
||||
nullTarget = flows[0];
|
||||
continue;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
Address[] recoveredTargets = jumpTable.getCases();
|
||||
Set<String> recoveredTargetSet = new HashSet<>();
|
||||
for (Address target : recoveredTargets) recoveredTargetSet.add(canonicalAddress(target));
|
||||
List<Address> dataAddresses = dispatchPathDataReferences(jumpTable);
|
||||
int baseIndex = latestWriter(instructions, branchIndex - 1, baseRegister);
|
||||
if (baseIndex < 0 || !mnemonic(instructions.get(baseIndex)).equals("ADR")) return null;
|
||||
java.util.regex.Matcher baseMatcher = AARCH64_REGISTER_AND_ADDRESS.matcher(operands(instructions.get(baseIndex)));
|
||||
if (!baseMatcher.matches()) return null;
|
||||
Address branchBase = addressFromOperand(baseMatcher.group(2));
|
||||
|
||||
for (int loadIndex = 0; loadIndex < branchIndex; loadIndex += 1) {
|
||||
Instruction load = instructions.get(loadIndex);
|
||||
int entrySize = load.getMnemonicString().equalsIgnoreCase("LDRB") ? 1 : load.getMnemonicString().equalsIgnoreCase("LDRH") ? 2 : 0;
|
||||
if (entrySize == 0 || branchIndex != loadIndex + 2 || loadIndex < 3 ||
|
||||
!instructions.get(loadIndex - 1).getMnemonicString().equalsIgnoreCase("ADR") ||
|
||||
!instructions.get(loadIndex - 2).getMnemonicString().equalsIgnoreCase("ADD") ||
|
||||
!instructions.get(loadIndex - 3).getMnemonicString().equalsIgnoreCase("ADRP")) continue;
|
||||
String loadOperands = operands(load);
|
||||
java.util.regex.Matcher loadMatcher = java.util.regex.Pattern
|
||||
.compile(entrySize == 1 ? "(?i)^[wx]\\d+,\\s*\\[([wx]\\d+),\\s*([wx]\\d+)\\]$" : "(?i)^[wx]\\d+,\\s*\\[([wx]\\d+),\\s*([wx]\\d+),\\s*LSL\\s*#(?:0x)?1\\]$")
|
||||
.matcher(loadOperands);
|
||||
if (!loadMatcher.matches()) continue;
|
||||
String loadedRegister = normalizedRegister(loadOperands.substring(0, loadOperands.indexOf(',')));
|
||||
String tableBaseRegister = normalizedRegister(loadMatcher.group(1));
|
||||
String indexRegister = normalizedRegister(loadMatcher.group(2));
|
||||
int tableAddIndex = latestWriter(instructions, loadIndex, tableBaseRegister);
|
||||
if (tableAddIndex < 0 || !mnemonic(instructions.get(tableAddIndex)).equals("ADD")) return null;
|
||||
java.util.regex.Matcher tableAddMatcher = AARCH64_ADD_IMMEDIATE.matcher(operands(instructions.get(tableAddIndex)));
|
||||
if (!tableAddMatcher.matches() ||
|
||||
!normalizedRegister(tableAddMatcher.group(1)).equals(tableBaseRegister) ||
|
||||
!normalizedRegister(tableAddMatcher.group(2)).equals(tableBaseRegister)) return null;
|
||||
int tablePageIndex = latestWriter(instructions, tableAddIndex, tableBaseRegister);
|
||||
if (tablePageIndex < 0 || !mnemonic(instructions.get(tablePageIndex)).equals("ADRP")) return null;
|
||||
java.util.regex.Matcher pageMatcher = AARCH64_REGISTER_AND_ADDRESS.matcher(operands(instructions.get(tablePageIndex)));
|
||||
if (!pageMatcher.matches() || !normalizedRegister(pageMatcher.group(1)).equals(tableBaseRegister)) return null;
|
||||
Address tableAddress = addressFromOperand(pageMatcher.group(2)).add(parseInteger(tableAddMatcher.group(3)));
|
||||
if (!currentProgram.getMemory().contains(tableAddress) ||
|
||||
(!dataAddresses.isEmpty() && !dataAddresses.contains(tableAddress))) return null;
|
||||
|
||||
Address resolvedTableAddress = null;
|
||||
for (int index = loadIndex - 3; index < loadIndex - 1; index += 1) {
|
||||
Instruction instruction = instructions.get(index);
|
||||
String mnemonic = instruction.getMnemonicString().toUpperCase(Locale.ROOT);
|
||||
String text = operands(instruction);
|
||||
if (mnemonic.equals("ADRP") &&
|
||||
text.matches("(?i)^[wx]\\d+,\\s*0x[0-9a-f]+$")) {
|
||||
String[] parts = text.split(",");
|
||||
if (normalizedRegister(parts[0]).equals(tableBaseRegister)) {
|
||||
resolvedTableAddress = addressFromOperand(parts[1]);
|
||||
}
|
||||
}
|
||||
else if (mnemonic.equals("ADD") && resolvedTableAddress != null &&
|
||||
text.matches("(?i)^[wx]\\d+,\\s*[wx]\\d+,\\s*#?(?:0x[0-9a-f]+|[0-9]+)$")) {
|
||||
String[] parts = text.split(",");
|
||||
if (normalizedRegister(parts[0]).equals(tableBaseRegister) &&
|
||||
normalizedRegister(parts[1]).equals(tableBaseRegister)) {
|
||||
resolvedTableAddress = resolvedTableAddress.add(parseInteger(parts[2].trim().replaceFirst("^#", "")));
|
||||
}
|
||||
}
|
||||
// Unsigned guard: CMP Rs, #imm, then B.HI (imm + 1 entries) or
|
||||
// B.CS/B.HS (imm entries), with only flag-preserving moves and
|
||||
// address arithmetic between the compare and the load.
|
||||
int guardIndex = -1;
|
||||
for (int index = loadIndex - 1; index >= 0; index -= 1) {
|
||||
String name = mnemonic(instructions.get(index));
|
||||
if (name.startsWith("B.")) {
|
||||
guardIndex = index;
|
||||
break;
|
||||
}
|
||||
if (resolvedTableAddress == null || !currentProgram.getMemory().contains(resolvedTableAddress) ||
|
||||
(!dataAddresses.isEmpty() && !dataAddresses.contains(resolvedTableAddress))) continue;
|
||||
Address tableAddress = resolvedTableAddress;
|
||||
|
||||
Address branchBase = null;
|
||||
boolean addressChain = false;
|
||||
for (int index = loadIndex - 1; index < branchIndex; index += 1) {
|
||||
Instruction instruction = instructions.get(index);
|
||||
String mnemonic = instruction.getMnemonicString().toUpperCase(Locale.ROOT);
|
||||
String text = operands(instruction);
|
||||
if (index < loadIndex && mnemonic.equals("ADR") && text.matches("(?i)^[wx]\\d+,\\s*0x[0-9a-f]+$")) {
|
||||
int comma = text.indexOf(',');
|
||||
String destination = normalizedRegister(text.substring(0, comma));
|
||||
if (!destination.equals(normalizedRegister(operands(instructions.get(branchIndex))))) continue;
|
||||
branchBase = addressFromOperand(text.substring(comma + 1));
|
||||
}
|
||||
if (index > loadIndex && mnemonic.equals("ADD") && branchBase != null &&
|
||||
text.matches("(?i)^[wx]\\d+,\\s*[wx]\\d+,\\s*[wx]\\d+,\\s*LSL\\s*#(?:0x)?2$")) {
|
||||
String[] parts = text.split(",");
|
||||
String destination = normalizedRegister(parts[0]);
|
||||
String base = normalizedRegister(parts[1]);
|
||||
String offset = normalizedRegister(parts[2]);
|
||||
if (destination.equals(base) &&
|
||||
base.equals(normalizedRegister(operands(instructions.get(branchIndex)))) &&
|
||||
offset.equals(loadedRegister)) {
|
||||
addressChain = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (!addressChain || branchBase == null) continue;
|
||||
|
||||
int entryCount = -1;
|
||||
Address defaultTarget = null;
|
||||
for (int compareIndex = 0; compareIndex < loadIndex; compareIndex += 1) {
|
||||
Instruction compare = instructions.get(compareIndex);
|
||||
if (!compare.getMnemonicString().equalsIgnoreCase("CMP")) continue;
|
||||
java.util.regex.Matcher compareMatcher = java.util.regex.Pattern
|
||||
.compile("(?i)^([wx]\\d+),\\s*#?(0x[0-9a-f]+|[0-9]+)$")
|
||||
.matcher(operands(compare));
|
||||
if (!compareMatcher.matches()) continue;
|
||||
String comparedRegister = normalizedRegister(compareMatcher.group(1));
|
||||
long upperBound = parseInteger(compareMatcher.group(2));
|
||||
if (upperBound < 0 || upperBound >= 4096) continue;
|
||||
for (int guardIndex = compareIndex + 1; guardIndex < loadIndex; guardIndex += 1) {
|
||||
Instruction guard = instructions.get(guardIndex);
|
||||
String mnemonic = guard.getMnemonicString().toUpperCase(Locale.ROOT);
|
||||
if (!mnemonic.equals("B.HI")) continue;
|
||||
Address[] flows = guard.getFlows();
|
||||
if (flows.length != 1 || !recoveredTargetSet.contains(canonicalAddress(flows[0]))) continue;
|
||||
boolean supportedSequence = true;
|
||||
for (int sequenceIndex = compareIndex + 1; sequenceIndex < loadIndex; sequenceIndex += 1) {
|
||||
String sequenceMnemonic = instructions.get(sequenceIndex)
|
||||
.getMnemonicString().toUpperCase(Locale.ROOT);
|
||||
if (!Set.of("B.HI", "MOV", "ADRP", "ADD", "ADR").contains(sequenceMnemonic)) {
|
||||
supportedSequence = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
Set<String> selectorAliases = new HashSet<>();
|
||||
selectorAliases.add(comparedRegister);
|
||||
Set<String> boundedIndexes = new HashSet<>();
|
||||
if (compareMatcher.group(1).toLowerCase(Locale.ROOT).startsWith("x")) boundedIndexes.add(comparedRegister);
|
||||
for (int sequenceIndex = compareIndex + 1; sequenceIndex < loadIndex; sequenceIndex++) {
|
||||
Instruction instruction = instructions.get(sequenceIndex);
|
||||
String[] parts = operands(instruction).split(",");
|
||||
if (parts.length == 0) continue;
|
||||
if (instruction.getMnemonicString().equalsIgnoreCase("MOV")) {
|
||||
String destination = normalizedRegister(parts[0]);
|
||||
String source = parts.length == 2 ? normalizedRegister(parts[1]) : "";
|
||||
boolean aliasesSelector = selectorAliases.contains(source);
|
||||
boolean fullWidthBounded = boundedIndexes.contains(source) ||
|
||||
(parts[0].trim().toLowerCase(Locale.ROOT).startsWith("w") && aliasesSelector);
|
||||
if (aliasesSelector) selectorAliases.add(destination);
|
||||
else selectorAliases.remove(destination);
|
||||
if (fullWidthBounded) boundedIndexes.add(destination);
|
||||
else boundedIndexes.remove(destination);
|
||||
} else if (Set.of("ADRP", "ADD", "ADR").contains(instruction.getMnemonicString().toUpperCase(Locale.ROOT))) {
|
||||
selectorAliases.remove(normalizedRegister(parts[0]));
|
||||
boundedIndexes.remove(normalizedRegister(parts[0]));
|
||||
}
|
||||
}
|
||||
boolean selectorFeedsTable = boundedIndexes.contains(indexRegister);
|
||||
if (compareIndex < loadIndex - 3 && selectorFeedsTable && supportedSequence) {
|
||||
entryCount = (int) upperBound + 1;
|
||||
defaultTarget = flows[0];
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (entryCount > 0) break;
|
||||
}
|
||||
if (entryCount <= 0 || entryCount > 4096 || defaultTarget == null) continue;
|
||||
|
||||
List<RecoveredAArch64Mapping> mappings = new ArrayList<>();
|
||||
Set<String> mappedTargets = new HashSet<>();
|
||||
boolean valid = true;
|
||||
for (int caseValue = 0; caseValue < entryCount; caseValue += 1) {
|
||||
Address entry = tableAddress.add((long) caseValue * entrySize);
|
||||
int encodedOffset = entrySize == 1 ? currentProgram.getMemory().getByte(entry) & 0xff : currentProgram.getMemory().getShort(entry) & 0xffff;
|
||||
Address target = branchBase.add((long) encodedOffset * 4L);
|
||||
String canonicalTarget = canonicalAddress(target);
|
||||
if (!recoveredTargetSet.contains(canonicalTarget)) {
|
||||
valid = false;
|
||||
break;
|
||||
}
|
||||
mappedTargets.add(canonicalTarget);
|
||||
mappings.add(new RecoveredAArch64Mapping(caseValue, target));
|
||||
}
|
||||
if (!valid || mappedTargets.contains(canonicalAddress(defaultTarget))) continue;
|
||||
Set<String> accountedTargets = new HashSet<>(mappedTargets);
|
||||
accountedTargets.add(canonicalAddress(defaultTarget));
|
||||
if (!accountedTargets.equals(recoveredTargetSet)) continue;
|
||||
return new RecoveredAArch64RelativeTable(tableAddress, branchBase, entrySize, List.copyOf(mappings));
|
||||
if (!AARCH64_FLAG_PRESERVING.contains(name)) return null;
|
||||
}
|
||||
return null;
|
||||
if (guardIndex < 1) return null;
|
||||
Instruction guard = instructions.get(guardIndex);
|
||||
String guardMnemonic = mnemonic(guard);
|
||||
if (!Set.of("B.HI", "B.CS", "B.HS").contains(guardMnemonic)) return null;
|
||||
Address[] guardFlows = guard.getFlows();
|
||||
if (guardFlows.length != 1) return null;
|
||||
Address defaultTarget = guardFlows[0];
|
||||
int compareIndex = -1;
|
||||
for (int index = guardIndex - 1; index >= 0; index -= 1) {
|
||||
String name = mnemonic(instructions.get(index));
|
||||
if (name.equals("CMP")) {
|
||||
compareIndex = index;
|
||||
break;
|
||||
}
|
||||
if (!AARCH64_FLAG_PRESERVING.contains(name)) return null;
|
||||
}
|
||||
if (compareIndex < 0) return null;
|
||||
java.util.regex.Matcher compareMatcher = AARCH64_COMPARE_IMMEDIATE.matcher(operands(instructions.get(compareIndex)));
|
||||
if (!compareMatcher.matches()) return null;
|
||||
String comparedRegister = normalizedRegister(compareMatcher.group(1));
|
||||
boolean comparedFullWidth = compareMatcher.group(1).toLowerCase(Locale.ROOT).startsWith("x");
|
||||
long compareImmediate = parseInteger(compareMatcher.group(2));
|
||||
long entryCountLong = guardMnemonic.equals("B.HI") ? compareImmediate + 1 : compareImmediate;
|
||||
if (entryCountLong <= 0 || entryCountLong > 4096) return null;
|
||||
int entryCount = (int) entryCountLong;
|
||||
|
||||
// The loaded index must carry the compared value: either the compared
|
||||
// register itself, unchanged since CMP, or a MOV copy made after CMP.
|
||||
// A W compare bounds only the low 32 bits, so an X index needs a
|
||||
// zero-extending W copy; a W index is zero-extended by UXTW.
|
||||
int indexWriter = latestWriter(instructions, loadIndex, indexRegister);
|
||||
boolean bounded = false;
|
||||
if (indexWriter < compareIndex) {
|
||||
bounded = indexRegister.equals(comparedRegister) && (wordIndex || comparedFullWidth);
|
||||
}
|
||||
else if (mnemonic(instructions.get(indexWriter)).equals("MOV")) {
|
||||
java.util.regex.Matcher moveMatcher = AARCH64_MOVE_REGISTER.matcher(operands(instructions.get(indexWriter)));
|
||||
if (moveMatcher.matches() &&
|
||||
normalizedRegister(moveMatcher.group(2)).equals(comparedRegister) &&
|
||||
latestWriter(instructions, indexWriter, comparedRegister) < compareIndex) {
|
||||
boolean wordCopy = moveMatcher.group(1).toLowerCase(Locale.ROOT).startsWith("w");
|
||||
bounded = wordCopy || comparedFullWidth;
|
||||
}
|
||||
}
|
||||
if (!bounded) return null;
|
||||
|
||||
// Every fact above holds only on one straight-line fall-through path
|
||||
// with no other entry into the window.
|
||||
int windowStart = Math.min(Math.min(tablePageIndex, baseIndex), compareIndex);
|
||||
for (int index = windowStart + 1; index <= branchIndex; index += 1) {
|
||||
monitor.checkCancelled();
|
||||
Instruction previous = instructions.get(index - 1);
|
||||
Instruction current = instructions.get(index);
|
||||
Address fallThrough = previous.getFallThrough();
|
||||
if (fallThrough == null || !fallThrough.equals(current.getAddress())) return null;
|
||||
ReferenceIterator incoming = currentProgram.getReferenceManager().getReferencesTo(current.getAddress());
|
||||
while (incoming.hasNext()) {
|
||||
if (incoming.next().getReferenceType().isFlow()) return null;
|
||||
}
|
||||
}
|
||||
|
||||
List<RecoveredAArch64Mapping> mappings = new ArrayList<>();
|
||||
Set<String> mappedTargets = new HashSet<>();
|
||||
for (int tableIndex = 0; tableIndex < entryCount; tableIndex += 1) {
|
||||
monitor.checkCancelled();
|
||||
Address entry = tableAddress.add((long) tableIndex * entrySize);
|
||||
if (!currentProgram.getMemory().contains(entry)) return null;
|
||||
long encoded;
|
||||
if (entrySize == 1) {
|
||||
byte value = currentProgram.getMemory().getByte(entry);
|
||||
encoded = signedEntries ? value : value & 0xffL;
|
||||
}
|
||||
else {
|
||||
short value = currentProgram.getMemory().getShort(entry);
|
||||
encoded = signedEntries ? value : value & 0xffffL;
|
||||
}
|
||||
if (encoded == 0 && nullTarget != null) {
|
||||
mappings.add(new RecoveredAArch64Mapping(tableIndex, nullTarget, encoded, true));
|
||||
continue;
|
||||
}
|
||||
Address target;
|
||||
try {
|
||||
target = branchBase.addNoWrap(encoded << offsetShift);
|
||||
}
|
||||
catch (ghidra.program.model.address.AddressOverflowException overflow) {
|
||||
return null;
|
||||
}
|
||||
String canonicalTarget = canonicalAddress(target);
|
||||
if (!recoveredTargetSet.contains(canonicalTarget)) return null;
|
||||
mappedTargets.add(canonicalTarget);
|
||||
mappings.add(new RecoveredAArch64Mapping(tableIndex, target, encoded, false));
|
||||
}
|
||||
if (mappedTargets.contains(canonicalAddress(defaultTarget))) return null;
|
||||
// Ghidra's recovered destinations must be fully accounted for by the
|
||||
// decoded entries, the guard default, and the CBZ null path.
|
||||
Set<String> accountedTargets = new HashSet<>(mappedTargets);
|
||||
accountedTargets.add(canonicalAddress(defaultTarget));
|
||||
if (nullTarget != null) accountedTargets.add(canonicalAddress(nullTarget));
|
||||
// Ghidra does not model the CBZ, so it may decode a zero entry as a
|
||||
// branch to the base itself. Only a zero entry encodes offset 0, and
|
||||
// the CBZ diverts every zero entry, so the BR never reaches the base.
|
||||
Address unreachableZeroTarget = null;
|
||||
boolean divertedZeroEntry = mappings.stream().anyMatch(mapping -> mapping.nullEntry);
|
||||
if (divertedZeroEntry &&
|
||||
recoveredTargetSet.contains(canonicalAddress(branchBase)) &&
|
||||
!mappedTargets.contains(canonicalAddress(branchBase)) &&
|
||||
!branchBase.equals(defaultTarget) &&
|
||||
!branchBase.equals(nullTarget)) {
|
||||
unreachableZeroTarget = branchBase;
|
||||
accountedTargets.add(canonicalAddress(branchBase));
|
||||
}
|
||||
if (!accountedTargets.containsAll(recoveredTargetSet)) return null;
|
||||
int selectorWriter = latestWriter(instructions, compareIndex, comparedRegister);
|
||||
return new RecoveredAArch64RelativeTable(
|
||||
tableAddress,
|
||||
branchBase,
|
||||
entrySize,
|
||||
selectorWriter < 0 ? null : instructions.get(selectorWriter).getAddress(),
|
||||
signedEntries,
|
||||
offsetShift,
|
||||
guardMnemonic,
|
||||
compareImmediate,
|
||||
nullTarget,
|
||||
unreachableZeroTarget,
|
||||
List.copyOf(mappings)
|
||||
);
|
||||
}
|
||||
|
||||
private static String mnemonic(Instruction instruction) {
|
||||
return instruction.getMnemonicString().toUpperCase(Locale.ROOT);
|
||||
}
|
||||
|
||||
/** Index of the last instruction before {@code before} that writes {@code register}, or -1. */
|
||||
private static int latestWriter(List<Instruction> instructions, int before, String register) {
|
||||
for (int index = before - 1; index >= 0; index -= 1) {
|
||||
for (Object result : instructions.get(index).getResultObjects()) {
|
||||
if (result instanceof Register written &&
|
||||
normalizedRegister(written.getBaseRegister().getName()).equals(register)) {
|
||||
return index;
|
||||
}
|
||||
}
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
private static String operands(Instruction instruction) {
|
||||
@@ -2658,6 +2907,63 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
return new InstructionScan(List.copyOf(instructions), iterator.hasNext());
|
||||
}
|
||||
|
||||
private JsonObject functionObservation(
|
||||
JsonElement value,
|
||||
List<String> limitations) {
|
||||
JsonObject result = new JsonObject();
|
||||
result.add("value", value);
|
||||
result.add("limitations", GSON.toJsonTree(limitations));
|
||||
return result;
|
||||
}
|
||||
|
||||
private List<String> noReturnCallLimitations(Iterable<Instruction> instructions)
|
||||
throws Exception {
|
||||
List<String> limitations = new ArrayList<>();
|
||||
for (Instruction instruction : instructions) {
|
||||
monitor.checkCancelled();
|
||||
if (!instruction.getFlowType().isCall() || !instruction.getFlowType().isTerminal()) {
|
||||
continue;
|
||||
}
|
||||
Address[] destinations = instruction.getFlows();
|
||||
StringBuilder limitation = new StringBuilder();
|
||||
limitation.append("Ghidra Listing reports a terminal call at ")
|
||||
.append(canonicalAddress(instruction.getAddress()));
|
||||
if (destinations.length == 0) {
|
||||
limitation.append(" with no resolved callee; its no-return basis is unknown.");
|
||||
}
|
||||
else {
|
||||
limitation.append(" to ");
|
||||
for (int index = 0; index < destinations.length; index++) {
|
||||
if (index > 0) limitation.append(", ");
|
||||
Address destination = destinations[index];
|
||||
limitation.append(canonicalAddress(destination));
|
||||
Function target = currentProgram.getFunctionManager().getFunctionAt(destination);
|
||||
if (target == null) {
|
||||
limitation.append(" (callee identity unknown)");
|
||||
}
|
||||
else {
|
||||
limitation.append(" ").append(procedureName(target));
|
||||
limitation.append(" (external=").append(target.isExternal());
|
||||
limitation.append(", thunk=").append(target.isThunk());
|
||||
limitation.append(", hasNoReturn=").append(target.hasNoReturn()).append(")");
|
||||
}
|
||||
}
|
||||
limitation.append("; fallthrough is excluded by Ghidra's flow model.");
|
||||
Function target = destinations.length == 1
|
||||
? currentProgram.getFunctionManager().getFunctionAt(destinations[0])
|
||||
: null;
|
||||
if (target != null && target.hasNoReturn()) {
|
||||
limitation.append(" This records Ghidra's FunctionManager flag, not an independent verification that the callee cannot return.");
|
||||
}
|
||||
else {
|
||||
limitation.append(" The reason for the terminal flow is not established by this observation.");
|
||||
}
|
||||
}
|
||||
limitations.add(limitation.toString());
|
||||
}
|
||||
return limitations;
|
||||
}
|
||||
|
||||
private static String renderAssembly(List<Instruction> instructions) {
|
||||
return String.join("\n", renderAssemblyLines(instructions));
|
||||
}
|
||||
@@ -3486,7 +3792,9 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
.thenComparing(Reference::isPrimary);
|
||||
|
||||
private record InventoryItem(Address address, String value, JsonObject facts) {}
|
||||
private record AddressNameRequest(String inputAddress, Address address, String name) {}
|
||||
private record FunctionEntry(Function function, InventoryItem item) {}
|
||||
private record FunctionAnalysisObservation(JsonObject dossier, List<String> limitations) {}
|
||||
private record InstructionScan(List<Instruction> instructions, boolean truncated) {}
|
||||
private record SessionDescriptor(
|
||||
String transport,
|
||||
|
||||
@@ -3,20 +3,25 @@ import ghidra.app.script.GhidraScript;
|
||||
import ghidra.program.model.address.Address;
|
||||
import ghidra.program.model.listing.*;
|
||||
import ghidra.program.model.symbol.FlowType;
|
||||
import ghidra.program.model.symbol.Reference;
|
||||
import ghidra.program.model.pcode.PcodeOp;
|
||||
import java.util.ArrayList;
|
||||
import java.util.ArrayDeque;
|
||||
import java.util.Deque;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
import java.util.TreeSet;
|
||||
|
||||
/**
|
||||
* Undo wrong no-return flags from auto-analysis. A function flagged no-return that contains a
|
||||
* real RET is an ordinary function (stream readers, cxa_guard_acquire, ...). Callers treated the
|
||||
* call as a terminator and the decompiler truncated them. Clear the flag, then re-disassemble only
|
||||
* real RET is an ordinary function (stream readers, cxa_guard_acquire, ...). Known returning
|
||||
* libc/TLS imports have no body to inspect, so clear their flags by exact external ABI name.
|
||||
* Callers treated the call as a terminator and the decompiler truncated them. Clear the flag, then re-disassemble only
|
||||
* the call sites of the functions that were cleared and refit their owning functions in place.
|
||||
*/
|
||||
public final class ReaGhidraNoReturnFix extends GhidraScript {
|
||||
private static final Set<String> RETURNING_IMPORTS = Set.of(
|
||||
"malloc", "__tls_get_addr", "memcpy", "memset", "strlen", "__errno_location");
|
||||
|
||||
@Override
|
||||
public void run() throws Exception {
|
||||
// Windows P0 does not admit analysis-database mutation.
|
||||
@@ -24,66 +29,95 @@ public final class ReaGhidraNoReturnFix extends GhidraScript {
|
||||
FunctionManager fm = currentProgram.getFunctionManager();
|
||||
Listing li = currentProgram.getListing();
|
||||
Set<Address> cleared = new HashSet<>();
|
||||
for (Function f : fm.getFunctions(true)) {
|
||||
Deque<Address> pending = new ArrayDeque<>();
|
||||
for (Function f : fm.getExternalFunctions()) {
|
||||
monitor.checkCancelled();
|
||||
if (!f.hasNoReturn() || f.isThunk()) continue;
|
||||
for (Instruction ins : li.getInstructions(f.getBody(), true)) {
|
||||
FlowType ft = ins.getFlowType();
|
||||
boolean decodedReturn = false;
|
||||
for (PcodeOp operation : ins.getPcode()) {
|
||||
if (operation.getOpcode() == PcodeOp.RETURN) decodedReturn = true;
|
||||
}
|
||||
if (decodedReturn && !ft.isJump() && !ft.isCall()) {
|
||||
f.setNoReturn(false);
|
||||
cleared.add(f.getEntryPoint());
|
||||
break;
|
||||
}
|
||||
}
|
||||
// Do not apply import-name assumptions to local functions or to arbitrary externals.
|
||||
if (!f.hasNoReturn() || !RETURNING_IMPORTS.contains(externalAbiName(f))) continue;
|
||||
f.setNoReturn(false);
|
||||
enqueueReturning(f, cleared, pending);
|
||||
}
|
||||
// A thunk inherits the flag of the function it forwards to.
|
||||
for (Function f : fm.getFunctions(true)) {
|
||||
monitor.checkCancelled();
|
||||
if (!f.isThunk()) continue;
|
||||
Function target = f.getThunkedFunction(true);
|
||||
if (target != null && cleared.contains(target.getEntryPoint())) {
|
||||
cleared.add(f.getEntryPoint());
|
||||
if (clearDecodedReturn(f, li)) {
|
||||
enqueueReturning(f, cleared, pending);
|
||||
}
|
||||
}
|
||||
int redone = 0;
|
||||
TreeSet<Address> owners = new TreeSet<>();
|
||||
if (!cleared.isEmpty()) {
|
||||
List<Address> sites = new ArrayList<>();
|
||||
for (Instruction ins : li.getInstructions(true)) {
|
||||
while (!pending.isEmpty()) {
|
||||
Address target = pending.removeFirst();
|
||||
TreeSet<Address> sites = new TreeSet<>();
|
||||
// References include computed import calls after Ghidra resolves their destination.
|
||||
// Collect before redisassembly mutates the reference table.
|
||||
for (Reference reference : currentProgram.getReferenceManager().getReferencesTo(target)) {
|
||||
monitor.checkCancelled();
|
||||
FlowType ft = ins.getFlowType();
|
||||
if (!(ft.isCall() && ft.isTerminal())) continue;
|
||||
Address[] flows = ins.getFlows();
|
||||
if (flows.length == 0 || !cleared.contains(flows[0])) continue;
|
||||
// Do not redisassemble into an independently identified function.
|
||||
Address next = ins.getMaxAddress().add(1);
|
||||
if (fm.getFunctionAt(next) != null || !currentProgram.getMemory().contains(next)) continue;
|
||||
sites.add(ins.getAddress());
|
||||
sites.add(reference.getFromAddress());
|
||||
}
|
||||
for (Address s : sites) {
|
||||
monitor.checkCancelled();
|
||||
Instruction ins = li.getInstructionAt(s);
|
||||
if (ins == null) continue;
|
||||
FlowType ft = ins.getFlowType();
|
||||
if (!(ft.isCall() && ft.isTerminal())) continue;
|
||||
Address[] flows = ins.getFlows();
|
||||
boolean callsTarget = false;
|
||||
for (Address flow : flows) {
|
||||
if (flow.equals(target)) callsTarget = true;
|
||||
}
|
||||
if (!callsTarget) continue;
|
||||
// Do not redisassemble into an independently identified function.
|
||||
Address next = ins.getMaxAddress().add(1);
|
||||
if (fm.getFunctionAt(next) != null || !currentProgram.getMemory().contains(next)) continue;
|
||||
Function owner = fm.getFunctionContaining(s);
|
||||
if (owner != null && !owner.isThunk()) owners.add(owner.getEntryPoint());
|
||||
clearListing(ins.getMinAddress(), ins.getMaxAddress());
|
||||
disassemble(s);
|
||||
redone++;
|
||||
}
|
||||
for (Address e : owners) {
|
||||
monitor.checkCancelled();
|
||||
Function f = fm.getFunctionAt(e);
|
||||
// Refit the body in place so the name, signature and calling convention survive.
|
||||
if (f != null) CreateFunctionCmd.fixupFunctionBody(currentProgram, f, monitor);
|
||||
if (owner != null && !owner.isThunk()) {
|
||||
CreateFunctionCmd.fixupFunctionBody(currentProgram, owner, monitor);
|
||||
// Auto-analysis may have propagated the false flag to this truncated caller.
|
||||
// Its RET becomes visible only now; repair its callers in the same pass.
|
||||
if (clearDecodedReturn(owner, li)) enqueueReturning(owner, cleared, pending);
|
||||
}
|
||||
}
|
||||
// No analyzeChanges here: on libpl2.so it adds about 70 s, which pushes startup past the 330 s deadline.
|
||||
}
|
||||
// No analyzeChanges here: on libpl2.so it adds about 70 s, which pushes startup past the 330 s deadline.
|
||||
println(
|
||||
"REA no-return fix: cleared=" + cleared.size() + " sites=" + redone + " refit=" + owners.size());
|
||||
}
|
||||
|
||||
private void enqueueReturning(Function function, Set<Address> cleared, Deque<Address> pending) {
|
||||
if (cleared.add(function.getEntryPoint())) pending.addLast(function.getEntryPoint());
|
||||
// A thunk inherits the flag of the function it forwards to, including through other thunks.
|
||||
Address[] thunks = function.getFunctionThunkAddresses(true);
|
||||
if (thunks == null) return;
|
||||
for (Address thunk : thunks) {
|
||||
if (cleared.add(thunk)) pending.addLast(thunk);
|
||||
}
|
||||
}
|
||||
|
||||
private String externalAbiName(Function function) {
|
||||
String name = function.getName();
|
||||
// Mach-O stores C-linkage symbols with one object-format underscore prefix.
|
||||
if (currentProgram.getExecutableFormat().contains("Mach-O") && name.startsWith("_"))
|
||||
return name.substring(1);
|
||||
return name;
|
||||
}
|
||||
|
||||
private boolean clearDecodedReturn(Function function, Listing listing) throws Exception {
|
||||
if (!function.hasNoReturn() || function.isThunk()) return false;
|
||||
for (Instruction instruction : listing.getInstructions(function.getBody(), true)) {
|
||||
monitor.checkCancelled();
|
||||
FlowType flow = instruction.getFlowType();
|
||||
if (flow.isJump() || flow.isCall()) continue;
|
||||
for (PcodeOp operation : instruction.getPcode()) {
|
||||
if (operation.getOpcode() != PcodeOp.RETURN) continue;
|
||||
function.setNoReturn(false);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,6 +24,29 @@ REA can attach to a user-owned Chrome-family browser through the Chrome DevTools
|
||||
|
||||
Target discovery returns the complete in-scope target array in one result.
|
||||
|
||||
## Screenshot comparison memory budget
|
||||
|
||||
`compare_web_screenshots` admits both PNGs against a 256 MiB estimated
|
||||
working-memory budget before inflating either image. This is a processing
|
||||
buffer budget for this operation, not a PNG dimension limit, CDP transport
|
||||
limit, or process RSS guarantee. Header-valid dimensions that exceed the
|
||||
budget return `resource_constraint` with `details.resource: "memory"` and the
|
||||
estimated and maximum working-set bytes. Invalid PNG structure remains
|
||||
malformed input; encodings outside supported pixel-comparison coverage remain
|
||||
unsupported-target errors.
|
||||
|
||||
The conservative estimate includes the two decoder-owned decoded input
|
||||
buffers, the current image's concatenated IDAT buffer, Node zlib's default
|
||||
16 KiB output chunk backing rounded up for retained chunks, the concatenated
|
||||
inflate result while its output chunks are retained, one possible extra output
|
||||
chunk before zlib checks `maxOutputLength`, unfiltered scanline storage,
|
||||
RGB-to-RGBA expansion, and the first retained RGBA image while the second is
|
||||
decoded. RGBA output aliases its unfiltered buffer and is counted once.
|
||||
Admission uses checked integer arithmetic before image-sized allocation.
|
||||
Caller-owned JSON/base64 strings, input parsing and validation, transport
|
||||
buffers, garbage-collector behavior, and other runtime memory are outside this
|
||||
estimate; the budget does not promise a bound on total process memory.
|
||||
|
||||
Electron `file://` pages use a separate provider and target boundary; see [electron-observation.md](electron-observation.md).
|
||||
|
||||
## Provider authorities
|
||||
@@ -53,8 +76,19 @@ google-chrome \
|
||||
|
||||
REA does not launch, own, or terminate this browser. Use a dedicated profile and stop it yourself when the investigation is complete. Do not expose the debugging port on a non-loopback interface.
|
||||
|
||||
On Linux, a reachable but locked desktop keyring can stall Chrome before its
|
||||
first navigation. For a dedicated investigation profile, add
|
||||
`--password-store=basic` to bypass that keyring. Empty or placeholder frame URLs
|
||||
then describe an unloaded page; REA continues to require an allowed, committed
|
||||
URL. See [the startup investigation](https://github.com/morluto/rea/issues/1384#issuecomment-6093781686).
|
||||
|
||||
## Request boundary
|
||||
|
||||
CDP connections enforce the shared 64 MiB limit per WebSocket message. A larger
|
||||
reply, including a valid reply from Chrome, fails with `payload_limit`. The limit
|
||||
counts message bytes before JSON parsing; it does not bound the memory used by
|
||||
decoded strings or JSON objects.
|
||||
|
||||
Discovery supplies a literal loopback CDP endpoint and returns all eligible
|
||||
page targets unless an exact HTTP(S) origin filter narrows the result. Follow-up
|
||||
inspection requests select one target; their default scope is its current origin. `localhost`, private-LAN
|
||||
|
||||
@@ -156,6 +156,18 @@ Storage cookies are scoped to the page URL at each capture. Unsupported action
|
||||
tags, unknown fields, duplicate step IDs, and undeclared secret references fail
|
||||
validation.
|
||||
|
||||
Cookies are installed before the initial navigation. Local storage entries are
|
||||
seeded once per browser storage partition; session storage entries are seeded
|
||||
once per origin in each top-level browsing context. Seeding occurs before the
|
||||
first document scripts for that area. An origin first reached by a later action
|
||||
receives its seed on that first visit. Application updates, deletions, and
|
||||
empty values then persist across later navigations in the same area; a new
|
||||
scenario starts with its own seeds. Partitioned frames can receive separate
|
||||
local storage seeds even when they share an origin. The pre-script mechanism
|
||||
uses CDP debugger instrumentation. The observer resumes only pauses whose top
|
||||
frame belongs to its private seed script; page breakpoints and `debugger`
|
||||
statements remain under the attached debugger client's control.
|
||||
|
||||
The result is Evidence with an initial state followed by one record per
|
||||
declared action. Each step reports action status, elapsed time, sanitized URLs,
|
||||
event bounds, and independently typed capture states for screenshot, DOM,
|
||||
|
||||
+64
-35
@@ -1,16 +1,25 @@
|
||||
# Optional NativeAOT metadata recovery
|
||||
|
||||
REA integrates [Washi1337/ghidra-nativeaot](https://github.com/Washi1337/ghidra-nativeaot)
|
||||
through an optional headless adapter. The original source is an opt-in submodule,
|
||||
pinned to `effeb734fc570c32650f88b159608979dc7b423e`; its MIT license and existing
|
||||
source notices are retained. No upstream source or algorithms are rewritten.
|
||||
REA has a built-in read-only snapshot parser for x64 PE targets with .NET 8 RTR
|
||||
9.1 metadata. It parses the exact admitted Ghidra session snapshot and uses an
|
||||
in-memory overlay for rehydrated metadata; it does not define Ghidra DataTypes,
|
||||
labels, functions, or memory bytes. The hydration semantics are adapted from
|
||||
[Washi1337/ghidra-nativeaot](https://github.com/Washi1337/ghidra-nativeaot) at
|
||||
`effeb734fc570c32650f88b159608979dc7b423e` (MIT), cross-checked against
|
||||
.NET's `MethodTable.h` at runtime commit
|
||||
`a3b2d40328be0be3f8dd43f0e5cc925b8827b044` (MIT). The required notice ships in
|
||||
[`licenses/nativeaot-parser-MIT.txt`](../licenses/nativeaot-parser-MIT.txt).
|
||||
|
||||
The initial tested layout is .NET runtime **8.0.22**, NativeAOT RTR **9.1**,
|
||||
x86-64 ELF and native Windows PE **targets analyzed on Linux x64**, using
|
||||
Ghidra **12.1.4** and JDK **21**. The build accepts any Ghidra 12.1.x release and
|
||||
javac 21 or newer, and it compiles with `--release 21`. Real verification of
|
||||
this layout remains the Ghidra 12.1.4 and JDK 21 pair. Other NativeAOT layouts,
|
||||
architectures, and hosts are unsupported.
|
||||
The optional headless Ghidra adapter remains available for database annotations
|
||||
and broader recovery. It uses the same pinned upstream commit as an opt-in
|
||||
submodule and is a separate, more capable workflow.
|
||||
|
||||
The parser currently handles x64 PE32+ and RTR 9.1, including bounded
|
||||
DEHYDRATED_DATA processing, MethodTable relationships, virtual slots, and
|
||||
frozen strings. Other RTR versions and PE architectures are reported as
|
||||
unsupported or partial. The matching .NET 8.0.22 fixture is source-built and
|
||||
the parser's Windows-host Ghidra workflow has not yet had a real Windows-host
|
||||
run. This does not establish Windows Ghidra import or decompiler behavior.
|
||||
Windows Ghidra P0 has no database mutation authority. A PE/CLI or ReadyToRun
|
||||
assembly should use `inspect_managed_artifact`; it is not NativeAOT.
|
||||
|
||||
@@ -43,42 +52,62 @@ arbitrary caller-supplied bytes. The bounded loader accepts regular JARs up to
|
||||
|
||||
## Use existing tools
|
||||
|
||||
Open the executable with provider `ghidra`. `inspect_native_load_image` returns
|
||||
optional `observations.metadata_recovery` inline, including the detected header,
|
||||
layout, recovery status, type category paths/addresses, diagnostics, annotation
|
||||
coverage and derived-memory SHA. The enclosing `unsupported` load-image status
|
||||
for PE/ELF concerns independent DOS loader verification; metadata recovery has
|
||||
its own status. It is not a failed import.
|
||||
Open the executable with provider `ghidra`. For an x64 PE, `inspect_native_load_image`
|
||||
returns `observations.metadata_recovery` inline from the captured source snapshot,
|
||||
including the RTR header, MethodTable addresses, frozen strings, derived overlay
|
||||
identity, and examined/recovered counts. Its enclosing `unsupported` load-image
|
||||
status means independent PE load-image verification was not performed; it does
|
||||
not mean the metadata parser failed or the executable failed to load.
|
||||
|
||||
Pass one returned type path or its typed address to `inspect_native_data_type`.
|
||||
Its optional `metadata_recovery` includes related/base type, implemented
|
||||
interfaces and virtual slots with targets and known procedure names. Follow
|
||||
these with `analyze_function`, string searches and ordinary cross-references.
|
||||
For CLI use:
|
||||
Pass a returned MethodTable address to `inspect_native_data_type`. A direct PE
|
||||
query also derives the snapshot metadata from the authenticated Ghidra session
|
||||
handshake, so it does not require a preceding load-image call. In a persistent
|
||||
MCP session, load-image followed by type inspection reuses that same derivation.
|
||||
The database result remains `unavailable` when Ghidra has no DataType at that
|
||||
address; its `metadata_recovery.source` is `read-only-derived-overlay` and
|
||||
carries related type, interface, slot, and target observations. Generated
|
||||
address identities do not claim original managed names. The parser does not
|
||||
merge frozen literals into Ghidra's database string inventory. Follow slot
|
||||
targets with `analyze_function` and ordinary cross-references.
|
||||
|
||||
The built-in CLI path uses a MethodTable address returned by the parser or
|
||||
another observation:
|
||||
|
||||
```sh
|
||||
rea inspect-native-load-image ./NativeAotFixture --provider ghidra --json
|
||||
rea inspect-native-data-type ./NativeAotFixture --type /NativeAOT/MethodTables/Class_ADDRESS_MT --provider ghidra --json
|
||||
rea inspect-native-data-type ./NativeAotFixture --address 0xMETHODTABLE_ADDRESS --provider ghidra --json
|
||||
rea function ./NativeAotFixture 0xADDRESS --provider ghidra --json
|
||||
```
|
||||
|
||||
No new MCP workflow or approval flag is required. Configuration opts into the
|
||||
adapter; without it ordinary Ghidra profiles/results remain unchanged.
|
||||
The generated `/NativeAOT/...` category path belongs to the optional headless
|
||||
adapter. Use `--type /NativeAOT/MethodTables/Class_ADDRESS_MT` only when that
|
||||
adapter is configured and has created the corresponding Ghidra DataType.
|
||||
Built-in PE analysis includes its parser contract and resource policy in the
|
||||
Ghidra analysis profile; non-PE profiles do not include that contract.
|
||||
|
||||
## Evidence boundaries
|
||||
|
||||
Discovery prefers an imported RTR symbol, otherwise the upstream initialized,
|
||||
non-executable-memory signature heuristic. REA rejects ambiguous candidates,
|
||||
unknown layout versions, malformed directory rows, unsupported flags and
|
||||
unmapped ranges before recovery. It requires the verified dehydrated layout.
|
||||
The built-in parser uses an initialized non-executable PE signature scan and
|
||||
requires a unique RTR 9.1 header and exactly one DEHYDRATED_DATA row. Rehydrated
|
||||
bytes live only in a parser-owned overlay, identified by virtual address, size,
|
||||
and SHA-256; the caller's source Buffer remains unchanged. Limits derive from
|
||||
captured section extents and format field bounds. Separate resource policies
|
||||
admit up to 128 MiB of captured PE bytes, 64 MiB-equivalent parser work, 64 MiB
|
||||
of exact parser-owned overlay/index Buffers, and 4 MiB of serialized NativeAOT
|
||||
report JSON. These are local parser limits, independent of MCP delivery; the
|
||||
buffer counter does not claim to cap V8 heap or total process memory. Rows that
|
||||
do not fit the report budget are omitted with explicit partial status and
|
||||
coverage. Coverage counts describe the pointer scan, recovered MethodTables,
|
||||
and frozen-string candidates examined, not every possible runtime type or
|
||||
object. Cancellation is honored during snapshot I/O and checked immediately
|
||||
before and after parsing; the synchronous parser call itself is bounded but
|
||||
cannot be interrupted mid-call. Parser output never claims that a Ghidra
|
||||
database type was defined.
|
||||
|
||||
The adapter performs rehydration, method-table recovery and frozen-object
|
||||
annotation in one ephemeral Program transaction. Fatal failure/cancellation
|
||||
rolls it back. Recovery may change analysis-memory bytes, never the original
|
||||
executable file. Derived ranges have a digest and **no original file offset**;
|
||||
they are not captured runtime state. Candidate/committed instance annotation
|
||||
counts distinguish partial frozen-object coverage. Coverage concerns the
|
||||
recovered candidate set, not every possible object or every runtime type.
|
||||
The optional adapter performs its annotations in an ephemeral Program
|
||||
transaction. That workflow may change analysis-memory bytes, never the original
|
||||
executable file. It is unavailable on Windows P0, which has no database
|
||||
mutation authority.
|
||||
|
||||
REA preserves pre-recovery calling conventions and uses the loaded compiler
|
||||
specification default for newly created methods. The upstream universal
|
||||
|
||||
+34
-4
@@ -8,9 +8,12 @@ Executable bytes remain unchanged.
|
||||
Use [installation](installation.md#ghidra) for supported hosts, prerequisites,
|
||||
configuration, startup deadlines, resource controls, and cleanup recovery.
|
||||
[Windows P0](windows-ghidra-p0.md) has its own admission and lifecycle boundary;
|
||||
[NativeAOT recovery](ghidra-nativeaot.md) is an optional, separately verified
|
||||
extension. Check the [release boundary](installation.md#released-package-and-main)
|
||||
when using npm.
|
||||
[NativeAOT observations](ghidra-nativeaot.md) include a built-in read-only PE
|
||||
snapshot parser and a separate optional annotation extension. Check the
|
||||
[release boundary](installation.md#released-package-and-main) when using npm.
|
||||
|
||||
See [import language and analysis seeds](installation.md#ghidra-import-language-and-analysis-seeds)
|
||||
to override the header-detected language or seed analysis.
|
||||
|
||||
Provider selection and profile identity follow
|
||||
[MCP contracts](mcp-contracts.md#identity-and-discovery). One deep
|
||||
@@ -29,9 +32,15 @@ not equivalence with another engine's pseudocode or analysis database.
|
||||
| Strings | Only Ghidra-defined string `Data` is observed. Items report charset, byte length, and whether a required null terminator is missing. The API cannot distinguish a present terminator from fixed/Pascal layouts when no terminator is missing, so that state is named `present_or_not_required`. |
|
||||
| Memory | Memory-block end addresses are exclusive. Read/write/execute, initialization, overlay, address space, and image base are direct Ghidra observations. |
|
||||
| Inventory | Procedure, symbol, and string listings plus searches return the complete matching collection in one response; callers do not provide offsets or result-count limits. |
|
||||
| Search | Literal and Java-regex searches scan the complete immutable inventory and return all matching entries inline. Search waits for a reply or caller cancellation; there are no caller-supplied offsets or result-count limits. |
|
||||
| Search | Literal and Java-regex searches return every match in the immutable provider-defined inventory; string search does not cover every text sequence in artifact bytes. Search waits for a reply or caller cancellation; there are no caller-supplied offsets or result-count limits. |
|
||||
| Analysis state | The socket is exposed only after default auto-analysis completes. Established operations wait for their reply or caller cancellation; there is no fixed response-size ceiling. |
|
||||
|
||||
String-search completeness applies only to Ghidra-defined string `Data`. Packed
|
||||
or undefined literals, including Rust strings, may be absent; an empty result
|
||||
does not prove text is absent. Both list and search Evidence carry this
|
||||
limitation. When an address is known, use `read_bytes` and `xrefs` to inspect
|
||||
bytes and references.
|
||||
|
||||
## Admitted function-analysis semantics
|
||||
|
||||
| Concern | Ghidra contract |
|
||||
@@ -46,6 +55,27 @@ not equivalence with another engine's pseudocode or analysis database.
|
||||
| P-code value flow | Function dossiers include up to 3,000 high-p-code operations, at most 64 inputs per operation, and 12,000 def-use edges. Results identify memory reads/writes, branches, and calls by p-code opcode, and mark decompiler-dead operations. Large results report truncation, an omitted-operation lower bound, and known omitted-input/edge counts. The dossier is an intra-function graph; `trace_native_values` composes derived parameter/argument and return/output dependencies across resolved calls. It does not resolve memory aliasing, call side effects, runtime behavior, persistent state, or business meaning. |
|
||||
| Result extent | Function instruction scans and native API boundary observations remain complete. P-code flow is the explicit bounded exception; its count fields distinguish exact retained counts from lower-bound or known-omitted counts. Caller cancellation and provider failures remain distinct. |
|
||||
|
||||
Function extents and instruction completeness refer to Ghidra's analyzed body.
|
||||
An inferred no-return call can terminate that body before subsequent bytes.
|
||||
Pseudocode, assembly, procedure info, instruction windows, and dossiers report
|
||||
observed terminal call sites in Evidence limitations, including resolved callee
|
||||
names, addresses, and FunctionManager flags when available. Instruction windows
|
||||
and dossiers also retain these warnings in their result limitations. An
|
||||
unresolved callee or unestablished terminal-flow basis remains explicit. These
|
||||
observations qualify the analysis; they do not prove that a no-return flag is
|
||||
incorrect or recover excluded code. Analysis profiles version this observation
|
||||
contract so older snapshots cannot replay unqualified results.
|
||||
|
||||
On non-Windows hosts, the ephemeral import repair clears no-return flags on the
|
||||
known returning external ABI names `malloc`, `__tls_get_addr`, `memcpy`, `memset`,
|
||||
`strlen`, and `__errno_location`. This relies on their expected ABI behavior;
|
||||
external implementations are not inspected. The repair re-disassembles affected
|
||||
terminal calls and refits their owning functions, preserving independent function
|
||||
entries. If a recovered caller reveals a decoded return, its propagated no-return
|
||||
flag and affected callers are repaired in the same pass. Unknown imports and
|
||||
genuine no-return imports keep their flags. Analysis profiles version this repair
|
||||
so snapshots produced before it are not reused. Windows P0 does not run it.
|
||||
|
||||
`npm run verify:ghidra` compiles the versioned C oracle into debug and stripped
|
||||
host-native targets (x86-64/AArch64 ELF on Linux or Mach-O on macOS), plus a native
|
||||
DWARF 4 type-layout object. It proves the admitted operations, external functions, resolved thunks, exports, stripped-name
|
||||
|
||||
+33
-24
@@ -68,21 +68,15 @@ selects the npm release, and persistent MCP registrations are pinned to the
|
||||
version that performed setup. Installing newer instructions does not update a
|
||||
running server or its registration.
|
||||
|
||||
The npm release checked on **2026-10-10** was **6.3.0**, published from
|
||||
[`74cf9e14`](https://github.com/morluto/rea/commit/74cf9e1401fcd89482fe39c27cc898f5858e9b72),
|
||||
with bundled skill version 34. It does not include automatic Pi or Hermes
|
||||
registration: `setup --client pi` and `setup --client hermes` reject those IDs
|
||||
before writing configuration. Their setup workflows below require a source build
|
||||
or a later release containing those integrations; `@latest` alone cannot select
|
||||
unpublished changes.
|
||||
|
||||
The earlier release checked on **2026-10-07** was **5.0.0** (133 MCP tools), published
|
||||
from the fixed checkpoint
|
||||
[`b33236ec`](https://github.com/morluto/rea/releases/tag/rea-agents-5.0.0).
|
||||
The public CLI, MCP catalog and target-free session, and isolated update from
|
||||
4.1.0 to 5.0.0 were verified through npm. The artifact includes Windows native
|
||||
controls, Android/JADX and firmware tools, Ghidra function annotations, and
|
||||
retained application-Evidence references.
|
||||
The npm release checked on **2026-10-11** was **6.4.0**, published from
|
||||
[`a56f7bb6`](https://github.com/morluto/rea/commit/a56f7bb668ab23fec8d33f7af353ac7ed2fe1895).
|
||||
The actual registry package accepted dry-run setup plans for all 17 client IDs
|
||||
listed below, including Qwen Code, Pi and Hermes. OpenCode's plan selected
|
||||
`OPENCODE_CONFIG_DIR` even when that directory did not yet exist. These checks
|
||||
used an isolated home on macOS with Node 24; they verify planning and argument
|
||||
admission, without writing client configuration or establishing live integration
|
||||
with every client. This resolves the npm 6.3.0 setup gaps tracked in
|
||||
[#1650](https://github.com/morluto/rea/issues/1650).
|
||||
|
||||
Main's catalog describes the current code. A source build or a subsequent
|
||||
release containing changes after this checkpoint is required for newer
|
||||
@@ -170,9 +164,6 @@ schema constraints; REA still validates complete canonical inputs. Full-profile
|
||||
Windows remain unverified; see [native client verification](testing.md).
|
||||
|
||||
For [Pi](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/mcp.md),
|
||||
automatic setup requires a build containing the integration; npm 6.3.0 does not
|
||||
include it. See [released package and main](#released-package-and-main).
|
||||
In a supported build,
|
||||
setup writes a `type: "stdio"` entry under `mcpServers.rea` in the user-level
|
||||
`~/.pi/agent/mcp.json` and installs the bundled skill under `~/.agents/skills`.
|
||||
Use `rea setup --client pi --dry-run --json` to inspect the plan, then
|
||||
@@ -231,10 +222,7 @@ registration. Doctor treats an `enabled: false` entry as active when
|
||||
`enabledServers` lists `rea`, as OMP does, unless `disabledServers` also lists
|
||||
it. Run setup under each profile that should load REA.
|
||||
|
||||
For Hermes, automatic setup also requires a build containing the integration;
|
||||
npm 6.3.0 does not include it. See
|
||||
[released package and main](#released-package-and-main).
|
||||
In a supported build, setup writes a `mcp_servers.rea` entry to `config.yaml`, preserving
|
||||
For Hermes, setup writes a `mcp_servers.rea` entry to `config.yaml`, preserving
|
||||
comments and unrelated settings. It resolves that file from `HERMES_HOME`, else
|
||||
from the platform default Hermes itself uses — `%LOCALAPPDATA%\hermes` on
|
||||
Windows, `~/.hermes` elsewhere — including any `HERMES_DATA_DIR_SUFFIX`. Hermes
|
||||
@@ -455,7 +443,7 @@ vendor-defined limits, and a paid license is optional. REA reuses any detected
|
||||
installation and preserves Hopper during uninstall.
|
||||
|
||||
The supported native host baseline is macOS 12+, Ubuntu 24.04+, Fedora 41+,
|
||||
Nobara 44+, 64-bit Arch Linux, or CachyOS. Ghidra and IDA have their own provider-specific
|
||||
Nobara 44+, 64-bit Arch Linux, CachyOS, or Omarchy. Ghidra and IDA have their own provider-specific
|
||||
host requirements; Windows Ghidra uses the [experimental P0 boundary](windows-ghidra-p0.md).
|
||||
|
||||
On macOS, approved setup downloads the official DMG, checks its published size
|
||||
@@ -725,6 +713,27 @@ is no fixed per-operation or response-size ceiling. Unresolved computed calls
|
||||
remain unknown, reference-kind provenance is preserved, and provider-specific
|
||||
pseudocode is never treated as original source or Hopper-equivalent text.
|
||||
|
||||
### Ghidra import language and analysis seeds
|
||||
|
||||
Ghidra detects the language from the executable header. To set it (for example, Borland C++ PE images):
|
||||
|
||||
```bash
|
||||
export REA_GHIDRA_LANGUAGE_ID="x86:LE:32:default"
|
||||
export REA_GHIDRA_COMPILER_SPEC_ID="borlandcpp" # optional; requires a language
|
||||
```
|
||||
|
||||
REA passes `-processor`/`-cspec`, checks both against the installation's `.ldefs`, and records them in the analysis profile. DOS imports reject an override.
|
||||
|
||||
To seed code that auto-analysis cannot reach (computed jumps, pointers in data):
|
||||
|
||||
```bash
|
||||
export REA_GHIDRA_SEED_FILE=/absolute/path/to/seeds.tsv
|
||||
```
|
||||
|
||||
Each line is `address<TAB>kind[<TAB>name]`, where kind is `function`, `code` or `label` (a label needs a name). Addresses are `0x` hex in the default space; `#` lines are comments. A pre-script applies seeds before auto-analysis: functions from the highest address down, then code, then labels. The seed digest is part of the analysis profile, and the session fails if the applied seeds differ. Result limitations report the outcome counts; a seed fails when it overlaps an instruction Ghidra already decoded.
|
||||
|
||||
Label seeds can name mapped uninitialized storage such as `.bss`; function and code seeds require initialized bytes.
|
||||
|
||||
### Ghidra heap and CPU controls
|
||||
|
||||
Set resource controls in the environment that launches the CLI or MCP server,
|
||||
@@ -851,7 +860,7 @@ For a client that requires manual configuration, use:
|
||||
"mcpServers": {
|
||||
"rea": {
|
||||
"command": "npx",
|
||||
"args": ["-y", "rea-agents@6.3.0", "mcp"]
|
||||
"args": ["-y", "rea-agents@6.4.0", "mcp"]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -60,6 +60,12 @@ manifest and graph commitments, JavaScript Application Graph, static
|
||||
Electron summary, reconstruction statistics, and explicit limitations. It does
|
||||
not require a live Hopper, Ghidra, browser, or Electron process.
|
||||
|
||||
The application graph selects the shallowest `package.json` outside
|
||||
`node_modules` as its package root, breaking equal-depth ties by Unicode
|
||||
code-point order. Dependency manifests remain package observations but do not
|
||||
declare Electron main or renderer entries. If only dependency manifests are
|
||||
present, the inventory artifact remains the graph root.
|
||||
|
||||
## ASAR integrity
|
||||
|
||||
ASAR inventory checks Electron integrity metadata for embedded archive entries
|
||||
@@ -89,9 +95,9 @@ queries use per-file range indexes, while call sites use exact-range lookups.
|
||||
Existing node budgets and coverage reporting remain in effect.
|
||||
|
||||
Progress identifies the source file being parsed and projected. Cancellation is
|
||||
checked between files, with event-loop yields that allow notifications and
|
||||
completed file-local allocations to be released. A single file's synchronous
|
||||
parser and semantic extraction still need to finish before that boundary.
|
||||
checked between files and throughout result transfer. The owned analyzer can
|
||||
be stopped while its synchronous parser or semantic extractor is active;
|
||||
the CLI/MCP process remains available for control requests.
|
||||
|
||||
Graph and Evidence identifiers hash canonical JSON incrementally, without
|
||||
assembling a single string for the whole graph. The canonical bytes and existing
|
||||
@@ -113,6 +119,46 @@ smaller view, for example `--format json --filter-output
|
||||
evidence_id,normalized_result.statistics`. Streaming output does not bound the
|
||||
memory needed to construct the analysis graph itself.
|
||||
|
||||
JavaScript source parsing, static analysis and semantic projection run in one
|
||||
owned analysis process, reused sequentially across files. Target code is never
|
||||
executed. The analyzer defaults to 1024 MiB of V8 old-space and a 300000 ms
|
||||
deadline per source, including startup and fact transfer. Select
|
||||
`max_heap_mb` and `analysis_timeout_ms` through MCP, or `--max-heap-mb` and
|
||||
`--analysis-timeout-ms` on the dedicated CLI command. These choices control the
|
||||
analyzer; the caller's `NODE_OPTIONS` still controls the CLI/MCP process heap.
|
||||
V8 old-space limits do not impose a process RSS limit.
|
||||
|
||||
A source that exhausts the analyzer heap does not terminate the CLI/MCP
|
||||
process. Completed static facts are retained, other files can still be analyzed,
|
||||
and unexamined semantic relationships remain explicit unknowns. The parent
|
||||
also checks transfer expansion and accumulated results against its remaining
|
||||
heap before accepting semantic facts. A resource-limited result has partial
|
||||
coverage, the observed limits and an `analyze-javascript-source` unknown scope
|
||||
with the specific failure, including captured diagnostics when available.
|
||||
Resource failures do not increment syntax parse failures. There is no fixed
|
||||
source-length cutoff.
|
||||
|
||||
Static observations also need memory when expanded into nodes, relationships
|
||||
and Evidence. If that expansion does not fit, the file node retains
|
||||
`static_analysis` and any admitted `semantic_module_analysis` observations,
|
||||
including their exact values and source locations. Use the public modules item
|
||||
view to read them. The graph marks those unexpanded relationships as unknown.
|
||||
If cumulative result capacity is exhausted, analysis stops; inventoried but
|
||||
unexamined files remain present with explicit unknown scopes.
|
||||
|
||||
A timeout stops the remaining analysis and returns typed failure details with
|
||||
completed partial Evidence. MCP failure details return
|
||||
`partial_observation: { kind: "retained-evidence", evidence_id: "..." }`
|
||||
after recording succeeds, so `inspect_analysis_view` can open that analysis
|
||||
directly without serializing the entire partial graph into the error response.
|
||||
CLI failure output retains the complete partial Evidence.
|
||||
Cancellation after source analysis begins likewise
|
||||
retains completed facts; an SDK client may reject its cancelled request before
|
||||
receiving that reply. On the same MCP connection, inspect the Evidence bundle
|
||||
and use the retained analysis ID with `inspect_analysis_view`. Owned analyzer
|
||||
cleanup is verified before its private files are released. Cleanup uncertainty
|
||||
is reported separately and retains ownership for retry.
|
||||
|
||||
CLI workflows parse larger JSON input files incrementally from one verified
|
||||
regular-file handle, without constructing a document-sized string. Smaller
|
||||
files retain native JSON parsing. Both paths require strict JSON and valid
|
||||
@@ -207,6 +253,46 @@ while locally shadowed receivers can still contribute network endpoint
|
||||
candidates. A template recovered with a parser error and no cooked value stays
|
||||
dynamic; its raw spelling is not treated as a valid JavaScript string.
|
||||
|
||||
RPC client syntax also contributes network endpoint candidates, in both
|
||||
`analyze_javascript_application` and `analyze_web_bundle`, distinguished by
|
||||
`mechanism`:
|
||||
|
||||
- `trpc:procedure:<method>` records the static member path between the root
|
||||
expression and the method, such as `invoice.list` in
|
||||
`root.invoice.list.useQuery(...)`. tRPC proxies keep these property names
|
||||
through identifier minification. React hooks and TanStack option factories
|
||||
(`useQuery`, `useMutation`, `useInfiniteQuery`, `useSubscription`, suspense
|
||||
and prefetch variants, `queryOptions`, `infiniteQueryOptions`,
|
||||
`mutationOptions`, `subscriptionOptions`) are recognized directly. The
|
||||
vanilla `query` and `mutate` methods are recognized on a root bound to
|
||||
`createTRPCClient` or `createTRPCProxyClient` in the same lexical binding, or on a
|
||||
path of at least two segments in a source containing `TRPCClientError`,
|
||||
`trpc-accept`, or `/trpc`. Factory bindings must have one unconditional
|
||||
initializer and no binding assignments; shadowed locals do not inherit client
|
||||
identity. `subscribe` additionally requires a tRPC observer
|
||||
argument (`onData`, `onError`, `onStarted`, `onStopped`, `onComplete`, or
|
||||
`onConnectionStateChange`). `this` roots, browser globals, and any path
|
||||
containing an RTK Query `endpoints` segment are never procedures; cache-key
|
||||
helpers such as `queryKey` issue no request and are excluded.
|
||||
- `trpc:link:<link>` records the literal `url` option of `httpLink`,
|
||||
`httpBatchLink`, `httpBatchStreamLink`, and `httpSubscriptionLink` when the
|
||||
link's callee name survives bundling, including `(0, module.httpBatchLink)`
|
||||
import-call wrappers.
|
||||
- `graphql:query`, `graphql:mutation`, and `graphql:subscription` record a
|
||||
named operation found at the start of a string or cooked template literal —
|
||||
the name must be followed by variable definitions (`($`), a directive, or a
|
||||
selection set starting with a field name — or in a precompiled
|
||||
`OperationDefinition` document object. A graphql-tag `loc.source.body` copy is
|
||||
not reported twice. Anonymous operations are not reported.
|
||||
|
||||
A procedure path is the server-side procedure name, not a resolved URL; the
|
||||
transport URL depends on the link configuration and batching. When the proxy is
|
||||
reached through a member expression, such as a webpack module binding
|
||||
(`l.S.invoice.list`) or a property holding the proxy (`o.api.post.byId`), the
|
||||
recorded path keeps those leading carrier segments; the procedure path is its
|
||||
suffix. Utility calls such as `useUtils().invoice.list.fetch()`, `useQueries`,
|
||||
and links whose callee was renamed by a minifier are not recognized.
|
||||
|
||||
Each recovered bundle module retains the exact factory-source digest. A complete
|
||||
bounded AST also receives a `babel-ast-v1` structural fingerprint that ignores
|
||||
ordinary identifier names while retaining syntax, literals, operators, object
|
||||
@@ -230,6 +316,14 @@ substitute version-dependent Electron defaults for omitted `webPreferences`.
|
||||
Dynamic option objects, bridge keys, API objects, and IPC channel expressions
|
||||
remain unknown and make coverage partial.
|
||||
|
||||
Electron API identity follows proven lexical module aliases. Reassigned bindings,
|
||||
dynamic selections, and visibly overwritten namespace members are unresolved.
|
||||
Direct and destructuring assignment, loop, update, and delete targets are checked
|
||||
through namespace aliases; unrelated member writes do not discard observed API
|
||||
identity. This static check does
|
||||
not evaluate arbitrary call side effects or runtime registration. Missing findings
|
||||
do not establish that an application has no Electron boundary usage.
|
||||
|
||||
IPC pairing is an inference, not an observation. A renderer `invoke` pairs only
|
||||
with one unique `ipcMain.handle`/`handleOnce` candidate on the same exact literal
|
||||
channel; a renderer send pairs only with one unique `ipcMain.on`/`once`
|
||||
|
||||
@@ -167,6 +167,9 @@ Native call tracing and process capture retain available observations in
|
||||
`details.partial_observation` on failure, including when cleanup succeeds.
|
||||
The observation reports its partial coverage; cleanup details describe host
|
||||
state separately from the execution failure.
|
||||
A process capture that fails during a finalization interval keeps every signal
|
||||
attempt and its delivery result in `details.partial_observation`, with
|
||||
`elapsed_ms` and `exit_code` null when no exit was observed.
|
||||
Derived comparisons and reconstruction verification yield before computation
|
||||
and before publication, so cancellation cannot race with successful Evidence.
|
||||
|
||||
@@ -305,6 +308,23 @@ that reference for a summary, one section/module, or a stable page, or call
|
||||
bundle exports stream canonical JSON into an atomically published file. A broad
|
||||
follow-up or `get_evidence_bundle` can also exceed the response budget; exporting
|
||||
preserves the complete session without sending it through a single MCP frame.
|
||||
To discover existing records without their large payloads, call
|
||||
`get_evidence_bundle` with `detail: "summary"`. Its `result.records` contain
|
||||
Evidence IDs, artifact/provider identities, analysis-profile digests and native
|
||||
procedure metadata. Optional `filters` match exact `evidence_id`, `operation`,
|
||||
`target_sha256`, `analysis_profile_digest` or `procedure_address`. Use a discovered
|
||||
ID with `inspect_analysis_view`; neither discovery nor retained selection starts
|
||||
a provider. `retention: "complete-record"` describes stored Evidence, while
|
||||
`native_dossier.value_flow` independently reports truncation, unavailable or
|
||||
unknown metadata. Missing omission counts remain null. Summary output carries
|
||||
every matching record; actual transport admission still applies. Omitted
|
||||
`detail` preserves complete canonical bundle delivery; filters require summary.
|
||||
|
||||
For a portable exported bundle, the CLI shares this projection:
|
||||
`rea inspect-evidence-bundle '{"path":"/tmp/evidence.json","detail":"summary","filters":{"operation":"analyze_function"}}' --format json`.
|
||||
The CLI validates and materializes that local bundle under its existing memory
|
||||
budget; summary delivery does not make arbitrary large files cheap to import.
|
||||
|
||||
Cancelling an export stops further serialization and removes its staging file.
|
||||
The destination changes only when a complete export is atomically published.
|
||||
|
||||
|
||||
@@ -400,11 +400,8 @@ GUI authority, and REA never falls back automatically to Hopper.
|
||||
`set_address_name` and `set_addresses_names` give Ghidra sessions the same
|
||||
compact naming results as Hopper: a local function entry renames that function,
|
||||
and any other mapped address, such as a global, receives a new or renamed primary
|
||||
label. Names owned by another address in the same namespace are rejected;
|
||||
the same leaf name remains valid in a different namespace. A batch applies each
|
||||
address separately and reports one success boolean per address, preserving the
|
||||
other entries when one fails. Ghidra batches do not move or swap names. Use them
|
||||
to name many globals and helpers cheaply; use
|
||||
label. A batch applies each address separately and reports one success boolean
|
||||
per address. Use them to name many globals and helpers cheaply; use
|
||||
`annotate_native_function` when you also want comments and a refreshed dossier.
|
||||
Ghidra name edits preserve the existing namespace. Supply either a leaf name
|
||||
such as `renamed` or a fully qualified name in that namespace, such as
|
||||
|
||||
@@ -181,6 +181,100 @@ run the same owned-process cleanup path. Settlement reports whether the
|
||||
sampled process group quiesced or whether cleanup was needed or unverifiable;
|
||||
sampling cannot prove that every short-lived or detached descendant was seen.
|
||||
|
||||
### Finalization interval
|
||||
|
||||
By default a deadline sends `SIGKILL` at once. Set `finalization_ms` to give the
|
||||
target time to finish: when `timeout_ms` or `idle_timeout_ms` fires, REA sends
|
||||
`SIGTERM`, keeps capturing output and selected files, and sends `SIGKILL` once
|
||||
if the interval ends first. Cancellation is not delayed: it sends `SIGKILL`
|
||||
immediately, also during finalization, and ends the run as cancelled.
|
||||
|
||||
```json
|
||||
{
|
||||
"executable": "node",
|
||||
"arguments": ["./job.mjs"],
|
||||
"filesystem_observation_paths": ["./reports"],
|
||||
"timeout_ms": 30000,
|
||||
"finalization_ms": 1500
|
||||
}
|
||||
```
|
||||
|
||||
REA binds the captured root's first start identity to its ownership token,
|
||||
checking the identity before and after reading that token. A missing or
|
||||
mismatched token, or an identity change during validation, leaves the identity
|
||||
unverified rather than adopting a replacement process. Both finalization
|
||||
signals recheck that retained identity immediately before the call; the check
|
||||
and signal remain separate operating-system operations. REA does not fall back
|
||||
to a bare PID kill for either finalization signal; failed identity or ownership
|
||||
validation is recorded as `unverified` with its reason. Cancellation before any
|
||||
deadline keeps the existing terminal kill. Descendants are not signalled during
|
||||
finalization; owned-process cleanup handles them afterwards.
|
||||
|
||||
`exit.reason` keeps the initiating deadline (`timeout` or `idle_timeout`).
|
||||
`exit.code` is `null` for every deadline exit; the code of an exit observed
|
||||
during finalization is `exit.finalization.exit_code`, and the observed signal
|
||||
stays in `exit.signal`. `exit.finalization` records what was observed and is
|
||||
absent when no finalization was attempted:
|
||||
|
||||
```json
|
||||
{
|
||||
"code": null,
|
||||
"signal": 0,
|
||||
"reason": "timeout",
|
||||
"finalization": {
|
||||
"requested_ms": 1500,
|
||||
"elapsed_ms": 12,
|
||||
"exit_code": 0,
|
||||
"signals": [
|
||||
{ "signal": "SIGTERM", "sent_at_ms": 0, "delivery": "signaled" }
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `signals` lists each attempt in order: `SIGTERM` first, at most one `SIGKILL`.
|
||||
`sent_at_ms` is the attempt time on a monotonic clock, relative to the start of
|
||||
finalization, also for an attempt that could not be delivered.
|
||||
- `delivery` is `signaled` (the identity matched and the signal call returned),
|
||||
`gone` (no live process matched), `identity-changed` (the PID now belongs to a
|
||||
different process, which was not signalled) or `unverified` (the identity could
|
||||
not be established or the call failed). None of them proves the process
|
||||
exited.
|
||||
An `unverified` attempt may carry a `reason` string naming why: an unavailable
|
||||
identity and its reason, an inspection or signal error, or delivery that did
|
||||
not settle within the bound (1000 ms). An attempt recorded `unverified` after
|
||||
that bound can still complete later; it stays identity-checked, and its late
|
||||
result is not added to the record.
|
||||
- `elapsed_ms` is when REA observed the exit, relative to the start of
|
||||
finalization, or `null` when no exit was observed. The PTY layer can deliver the
|
||||
exit slightly after the operating system ended the process.
|
||||
- `exit_code` is the code the PTY reported when the exit was observed during
|
||||
finalization and was not caused by a signal (`exit.signal` is `0`); a
|
||||
finalizer that exits with `3` records `3`, one that exits cleanly records `0`.
|
||||
It is `null` when no exit was observed or the target ended on a signal, and it
|
||||
is never inferred. A target that exited on its own near the deadline is
|
||||
recorded the same way.
|
||||
- The observed signal stays in `exit.signal`; `exit.code` is `null` for every
|
||||
deadline exit. The record does not attribute an exit to a particular signal,
|
||||
and a `SIGKILL` from a scenario event next to the escalation cannot be told
|
||||
apart from it.
|
||||
- If the `SIGKILL` cannot be delivered, REA stops waiting for an exit, keeps every
|
||||
observation collected so far with the delivery results, and releases the run
|
||||
through owned-process cleanup.
|
||||
- The `SIGKILL` is not sequenced after the `SIGTERM` delivery result: the two
|
||||
deliveries are independent, because cancellation must not wait for a slow
|
||||
identity check.
|
||||
|
||||
`timeout_ms + finalization_ms + settle_ms` must stay at or below
|
||||
`Number.MAX_SAFE_INTEGER` (9007199254740991), the largest total exactly
|
||||
representable as a JavaScript number; the scenario is rejected otherwise. This is
|
||||
a representability limit, not a timer limit. The nominal lifecycle budget is that
|
||||
sum; polling, snapshotting and cleanup lie outside it. Scenario `events` keep
|
||||
being dispatched during finalization and are recorded as usual. `finalization_ms`
|
||||
of `0`, the default, keeps today's behaviour and the committed identity of every
|
||||
existing scenario. `elapsed_ms` and `sent_at_ms` are timing data and take part in
|
||||
exit comparison like `settlement.elapsed_ms`.
|
||||
|
||||
Every capture requires `truncation_details`, with separate accounting for:
|
||||
|
||||
- `raw_terminal`: original UTF-8 PTY chunk bytes and observed/retained chunk
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -15,6 +15,52 @@ the original run after a partial publication.
|
||||
For a release that needs an independently frozen application checkpoint, use
|
||||
the optional manual path below. It retains explicit prepare and publish phases.
|
||||
|
||||
## Contributor-aware release notes
|
||||
|
||||
Use [the rea-changelog-update skill](https://github.com/morluto/rea/blob/main/.agents/skills/rea-changelog-update/SKILL.md)
|
||||
to turn the bot draft into grouped Highlights, Changes, and Fixes with inline
|
||||
`Thanks @...` credit. `npm run release:notes -- inventory` collects the full
|
||||
selected Git range and paginated GitHub PR/author/closing-issue metadata;
|
||||
`record --summary` produces a compact provenance link for public notes, while
|
||||
`record` produces the full record for review. Credit a contribution once even
|
||||
when it appears in highlights, details and migrations; omit maintainer
|
||||
`@morluto` self-thanks. Keep complete authorship metadata in the inventory
|
||||
instead of duplicating a credit ledger in the release body. `check` validates credits
|
||||
and optional PR-body equality, and `render` writes the verified release section.
|
||||
Run a subcommand without its required options to see its usage. GitHub reads
|
||||
require an authenticated `gh`; record/check/render work offline from the saved
|
||||
inventory and complete local Git history.
|
||||
|
||||
Save a finalized inventory on the release branch at
|
||||
`docs/releases/VERSION.contributions.json`. Review unknown identities and
|
||||
reverted/backported work explicitly; GitHub metadata does not establish the
|
||||
quality of a user-facing claim. Preserve the existing breaking-change section
|
||||
and checkpoint checks. The contribution inventory is a reviewed snapshot, not
|
||||
a signed attestation; the offline checker verifies local range coverage and
|
||||
credit consistency, not GitHub's authorship independently.
|
||||
|
||||
Before editing the live PR, add `rea:release-notes-finalized` to prevent ordinary
|
||||
main-push proposals from overwriting the editorial notes. Wait for any already
|
||||
running proposal to finish. The guard applies to main-push automation; do not
|
||||
run manual checkpoint preparation over finalized notes. Remove the label to
|
||||
resume bot regeneration. The label does not freeze the base branch: new base
|
||||
commits require re-inventorying and re-reviewing the notes.
|
||||
|
||||
Update both the release branch's `CHANGELOG.md` and the PR body, preserving
|
||||
Release Please's surrounding metadata. Before creating a release, the workflow
|
||||
checks an inventory when present, requires its target to equal the reviewed
|
||||
merge's first parent, and compares the freshly fetched PR body to the changelog.
|
||||
It rejects changes outside the release metadata paths and stale credit records
|
||||
before any tag or publication. Review the diffs within allowed metadata paths
|
||||
as well, especially package dependencies and scripts. Existing candidates without inventories retain their current
|
||||
checks. Merging still starts publication; a notes edit does not authorize a
|
||||
merge or release by itself.
|
||||
|
||||
The checker supports inline Release Please PR bodies up to GitHub's 65,536
|
||||
character limit, including the surrounding header/footer. Overflow-link PR
|
||||
bodies need a separate integration and fail explicitly; credits are never
|
||||
silently truncated.
|
||||
|
||||
## 1. Select a source for a manual checkpoint
|
||||
|
||||
Use the next minor version proposed by Release Please, including releases with
|
||||
|
||||
+111
-22
@@ -23,6 +23,11 @@ cases. Share setup without hiding inputs or expected evidence. For successful
|
||||
`.ok` assertion adds no coverage. Protocol fixtures must reject unmodeled
|
||||
commands and preserve actual producer reply shapes.
|
||||
|
||||
Stateless source-analysis matrices may share a file-owned stdio MCP connection
|
||||
and retain representative compiled CLI parity checks. Every distinct source
|
||||
scenario still runs through the public workflow; session-lifecycle scenarios
|
||||
need their own sessions and cleanup assertions.
|
||||
|
||||
Report unavailable authority as a named skip. Await owned cleanup before a
|
||||
verifier reports success. Native process inspectors are released at Vitest
|
||||
worker teardown; fork termination does not run Node exit hooks, and per-file
|
||||
@@ -391,6 +396,12 @@ regressions until a real fixture establishes equivalent coverage.
|
||||
|
||||
`verify:browser` exercises source-map failure isolation and expanded-output limits through the compiled CLI and stdio MCP with real Chrome. It also submits five 2 MiB source-map annotations, checks retained script identities and explicit map omissions, and closes its owned fixture target. Replacement and document-reset behavior has separate producer-boundary coverage.
|
||||
|
||||
The raw Chrome fixture launchers use `--password-store=basic` so Linux startup
|
||||
does not wait for an unlock prompt from a reachable but locked desktop keyring.
|
||||
Such a stall leaves the initial page unloaded with empty or placeholder CDP URLs;
|
||||
it does not establish permission to capture that page. See
|
||||
[the keyring diagnosis](https://github.com/morluto/rea/issues/1384#issuecomment-6093781686).
|
||||
|
||||
`verify:browser` also captures a source-owned noise canvas as a real PNG above
|
||||
8 MiB through the CLI and stdio MCP, with complete byte/digest parity and real PNG
|
||||
decoding. Its SDK client explicitly permits the larger inline JSON response;
|
||||
@@ -444,14 +455,15 @@ installation (`application.java.min` through `application.java.max`). Current
|
||||
12.1 releases require JDK 21 or newer and set no maximum. The lanes below still
|
||||
prove behavior on the verified Ghidra 12.1.4 and JDK 21 build.
|
||||
|
||||
| Ghidra lane | Supported runner/target | Additional local tools |
|
||||
| ------------------------------------------ | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
|
||||
| `npm run verify:ghidra` | Linux x64/arm64 ELF or macOS x64/arm64 Mach-O | Host C compiler, Ghidra 12.1.4, and full JDK 21 |
|
||||
| `npm run verify:ghidra:swift` | macOS x64/arm64 Mach-O | Host Swift compiler, Ghidra 12.1.4, and full JDK 21 |
|
||||
| `npm run verify:ghidra:switch` | Linux x64 ELF; GCC/Clang optimized and stripped switch fixtures | GCC, Clang, GNU nm/objdump/strip, Ghidra 12.1.4, and full JDK 21 |
|
||||
| `npm run verify:ghidra:aarch64-jump-table` | Any supported Ghidra host; AArch64 ELF; byte/halfword tables; host ARM64 Mach-O | Clang with AArch64 target support, Ghidra 12.1.4, and full JDK 21 |
|
||||
| `npm run verify:ghidra:cross-format` | Any supported Ghidra host; also analyzes AArch64 ELF, x86-64 PE, and x86-64 Mach-O | `clang`, LLD, and `lld-link` in addition to host-lane prerequisites |
|
||||
| `npm run verify:ghidra:windows` | Controlled Windows x64 with native x86-64 PE; `-- --x86` selects native x86 PE | Ghidra 12.1.4, full JDK 21, and the matching native artifact |
|
||||
| Ghidra lane | Supported runner/target | Additional local tools |
|
||||
| ------------------------------------------ | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
|
||||
| `npm run verify:ghidra` | Linux x64/arm64 ELF or macOS x64/arm64 Mach-O | Host C compiler, Ghidra 12.1.4, and full JDK 21 |
|
||||
| `npm run verify:ghidra:seeds` | POSIX host; source-owned 32-bit x86 raw fixture; function order and BSS labels | Ghidra 12.1.4 and full JDK 21 |
|
||||
| `npm run verify:ghidra:swift` | macOS x64/arm64 Mach-O | Host Swift compiler, Ghidra 12.1.4, and full JDK 21 |
|
||||
| `npm run verify:ghidra:switch` | Linux x64 ELF; GCC/Clang optimized and stripped switch fixtures | GCC, Clang, GNU nm/objdump/strip, Ghidra 12.1.4, and full JDK 21 |
|
||||
| `npm run verify:ghidra:aarch64-jump-table` | Any supported Ghidra host; AArch64 ELF; signed/unsigned byte/halfword tables; host ARM64 Mach-O | Clang with AArch64 target support, Ghidra 12.1.4, and full JDK 21 |
|
||||
| `npm run verify:ghidra:cross-format` | Any supported Ghidra host; also analyzes AArch64 ELF, x86-64 PE, and x86-64 Mach-O | `clang`, LLD, and `lld-link` in addition to host-lane prerequisites |
|
||||
| `npm run verify:ghidra:windows` | Controlled Windows x64 with native x86-64 PE; `-- --x86` selects native x86 PE | Ghidra 12.1.4, full JDK 21, and the matching native artifact |
|
||||
|
||||
The optional Swift lane compiles a source-owned fixture and checks modern and
|
||||
provider-demangled procedure names through real stdio MCP, with and without the
|
||||
@@ -665,6 +677,13 @@ boundaries and budgets.
|
||||
|
||||
## Optional NativeAOT Ghidra analysis
|
||||
|
||||
The source parser lane is independent of the optional annotation JAR. Its
|
||||
synthetic RTR 9.1 PE golden checks captured-byte digest admission, rehydration
|
||||
into an immutable parser overlay, MethodTable/slot relationships, and frozen
|
||||
literal extraction; provider tests exercise `inspect_native_load_image` and
|
||||
address-based `inspect_native_data_type` with the database result still
|
||||
unavailable. These tests do not establish Windows Ghidra host behavior.
|
||||
|
||||
This lane is separate from the default native lane. `build:fixtures:nativeaot`
|
||||
requires an existing .NET SDK 8.0.416, the platform NativeAOT compiler/linker and
|
||||
runtime pack 8.0.22. It builds benign sources into ignored `_reference/` and
|
||||
@@ -765,12 +784,37 @@ Pull requests that change only root `README*.md` files, `docs/`, `AGENTS.md`,
|
||||
or `CONTRIBUTING.md` run formatting and generated-document validation without
|
||||
the source-test shards or native package lanes. Classification compares the PR
|
||||
head with its merge base, so later base-branch changes do not expand that scope.
|
||||
The required static and coverage aggregate jobs remain present and fail if
|
||||
classification fails. Mixed changes, unknown paths and CI configuration changes
|
||||
retain full checks. Main pushes retain the full baseline, including after
|
||||
documentation merges; newer commits cancel superseded CI for the same PR or
|
||||
main branch. Release publication has its own concurrency group and retains its
|
||||
package verification and public-registry canary.
|
||||
Static and coverage aggregate jobs remain present and fail if classification
|
||||
fails. Every code change retains the complete four-shard Linux suite; individual
|
||||
test files are not selected by imports or filenames.
|
||||
|
||||
`scripts/ci/plan.mjs` selects lanes using the ownership table in
|
||||
`scripts/ci/scopes.mjs`. Provider implementation changes run their real-provider
|
||||
verifiers and the Linux x64 installed-package/Inspector baseline. Test-only
|
||||
changes run the Linux suite without native package verification. Website changes
|
||||
run website checks. Provider workflow changes validate the workflow and run that provider's verifier.
|
||||
Manual-provider/Android workflow changes run workflow validation only.
|
||||
Shared process/contract/CLI boundaries, dependencies, packaging inputs, unknown
|
||||
paths, and changes to the planner select the full baseline. Package script-only
|
||||
edits select their owning verifier when its entrypoint is identifiable; unknown
|
||||
or shared script changes select everything. Android build-script edits retain
|
||||
the Linux suite and workflow validation; real Android verification stays manual.
|
||||
Renames account for both paths.
|
||||
|
||||
Main pushes and manual dispatch retain the full baseline, including all portable
|
||||
provider lanes and the native matrix below. A `ci:full` PR label selects the full
|
||||
baseline on the next PR run; manual dispatch can request it immediately. Each
|
||||
classification job records its selection and broad-fallback reasons in the run
|
||||
summary. Representative Git-diff cases live in
|
||||
`tests/boundary/filesystem/ciChangeScope.test.ts`.
|
||||
|
||||
`CI required` checks classification and every selected lane, rejecting failed,
|
||||
cancelled, or unexpectedly skipped work. Existing required check names are
|
||||
preserved; changing branch protection to require the new aggregate is a separate
|
||||
repository-settings operation. Newer commits cancel superseded CI for the same
|
||||
PR or main branch. Release publication has its own concurrency group and retains
|
||||
its package verification and public-registry canary. Licensed/self-hosted
|
||||
provider workflows and the Android smoke test retain their explicit triggers.
|
||||
|
||||
CI exercises the pinned Node.js runtime on native hosted runners:
|
||||
|
||||
@@ -784,17 +828,59 @@ CI exercises the pinned Node.js runtime on native hosted runners:
|
||||
|
||||
Package lanes assert the actual Node platform/architecture before verification
|
||||
and record those values with the Node version. The four-host package matrix
|
||||
runs at most two jobs concurrently with explicit timeouts and Node heap/thread
|
||||
limits; Windows retains its curated capability lane. Package checks cover
|
||||
installation, CLI/MCP discovery, target-free
|
||||
analysis, configuration backups/recovery, Evidence and owned lifecycle; Inspector
|
||||
checks execute source-owned loopback targets and special filename cases.
|
||||
runs up to four jobs concurrently on independent hosted runners with explicit
|
||||
timeouts and per-runner Node heap/thread limits; Windows retains its curated
|
||||
capability lane. Package checks cover installation, CLI/MCP discovery,
|
||||
target-free analysis, configuration backups/recovery, Evidence and owned
|
||||
lifecycle; Inspector checks execute source-owned loopback targets and special
|
||||
filename cases.
|
||||
|
||||
The Apple artifact CLI/MCP lane uses one `macos-15` job to avoid adding demand
|
||||
for scarce macOS runner capacity. It runs the artifact-format and metadata
|
||||
verifiers, followed by one consolidated focused invocation containing all
|
||||
existing native and process-boundary regression paths.
|
||||
|
||||
The Linux source-test shards download a shared runtime and generated test
|
||||
metadata snapshot produced by the build job with `npm run test:prepare`. The
|
||||
snapshot includes `dist/`, generated skills, the MCP tool catalog, product
|
||||
catalog, and managed-conformance metadata. Each shard still installs the locked
|
||||
dependencies and runs its assigned coverage-enabled tests; it no longer
|
||||
regenerates the same metadata independently.
|
||||
|
||||
These native baseline checks complement the Linux source-test shards and the
|
||||
separate Apple-artifact and real-provider lanes. Actual Hopper, Ghidra, IDA,
|
||||
browser and managed-tool claims require their corresponding verification lanes.
|
||||
Runner labels follow the [GitHub hosted-runner reference](https://docs.github.com/en/actions/reference/runners/github-hosted-runners).
|
||||
|
||||
### Termux Android browser smoke test
|
||||
|
||||
The **Real Termux browser verification** workflow is manual-only while its
|
||||
emulator setup is being validated. After the workflow reaches the default
|
||||
branch, select a revision under **Actions → Real Termux browser verification →
|
||||
Run workflow**. It boots an Android 11 (API 30) x86_64 emulator, installs the
|
||||
checksum-verified Termux 0.118.3 APK, and lets the app initialize its bootstrap.
|
||||
The host stages the selected revision using `git archive HEAD`; no workstation
|
||||
build outputs or dependencies are copied into Android.
|
||||
|
||||
`scripts/verify/termux/emulator.sh` sends a `RUN_COMMAND` intent to Termux's app
|
||||
service. Root access is limited to staging files, sending the intent, and reading
|
||||
diagnostics; installation, compilation, Node, and Chromium run as Termux's app
|
||||
UID. The in-app script installs Termux's Android-linked Node distribution,
|
||||
checks its version against `.nvmrc`, pins npm from `packageManager`, runs
|
||||
`npm ci`, and builds with `build:termux`. Termux repository packages, including
|
||||
Chromium, are resolved at run time; their installed versions are logged.
|
||||
|
||||
The verifier leaves Playwright's cache environment overrides unset, requires
|
||||
a healthy public MCP doctor result, and captures URL plus DOM from a loopback
|
||||
HTTP fixture through `capture-browser-scenario`. It checks completed steps,
|
||||
the exact URL, a DOM marker, and reported browser cleanup. This lane covers one
|
||||
Android emulator and Termux combination, not physical ARM devices, Electron,
|
||||
or the complete desktop browser suite. A workflow definition alone does not
|
||||
establish passing Android coverage; inspect its execution receipt.
|
||||
|
||||
The `termux-browser-diagnostics` artifact retains the Termux command log, exit
|
||||
status, and Android logcat on success or failure. The emulator is disposable.
|
||||
|
||||
## Developer commands
|
||||
|
||||
Use source feedback while editing, explicit boundary checks for the changed
|
||||
@@ -862,9 +948,12 @@ provider evidence.
|
||||
Vitest projects use up to two workers, bounded by host parallelism.
|
||||
`process-boundary` runs later with serial files because process-tree sampling
|
||||
shares host resources. Acceptance and process-global cases use isolated forks
|
||||
without serial scheduling. Pure domain/service and MCP boundary projects share
|
||||
immutable module graphs; each MCP session still owns its resources. Other
|
||||
projects retain per-file isolation. See `vitest.config.ts` for current settings.
|
||||
without serial scheduling. Pure domain/service, composition, conformance,
|
||||
evaluation, and MCP boundary projects share module graphs; their tests avoid
|
||||
process-global state and keep resource cleanup test-scoped. Each MCP session
|
||||
still owns its resources. Adapter, non-MCP boundary, acceptance, and
|
||||
process-global projects retain per-file isolation. See `vitest.config.ts` for
|
||||
current settings.
|
||||
|
||||
Build/documentation writers hold checkout-local output locks. Tests do not
|
||||
hold a broad command lock; `check:pr` finishes tests before document validation.
|
||||
|
||||
@@ -21,9 +21,12 @@ additional permission flag or degraded mode is required.
|
||||
- the 25 read-only Ghidra inventory, memory, and function-analysis operations.
|
||||
|
||||
Loaded memory reads and file offsets preserve Ghidra's observed source mappings.
|
||||
PE load-image inspection returns measurements with an explicit unsupported
|
||||
attestation status; independent format-specific verification currently supports
|
||||
DOS MZ only.
|
||||
PE load-image inspection returns an explicit unsupported attestation status;
|
||||
independent load-image verification currently supports DOS MZ only. For x64 PE
|
||||
NativeAOT RTR 9.1, the read-only snapshot parser can report metadata through
|
||||
`inspect_native_load_image` and `inspect_native_data_type` without a JAR or
|
||||
database mutation. A real Windows-host NativeAOT Ghidra run is still required
|
||||
to establish the complete host workflow.
|
||||
|
||||
This boundary does not establish general Windows feature parity. Ghidra GUI
|
||||
state, annotations, other target architectures and formats, and Hopper-only
|
||||
|
||||
@@ -3,9 +3,12 @@
|
||||
"entry": [
|
||||
"src/main.ts",
|
||||
"src/cli.ts",
|
||||
"src/composition/javascriptAnalysisWorkerProcess.ts",
|
||||
"src/contracts/errorSchemas.ts",
|
||||
"src/evaluation/AgentEvaluationFixtures.ts",
|
||||
"src/application/RealHopperAssertions.ts",
|
||||
"scripts/verify/termux/browser.mjs",
|
||||
"scripts/verify-real-ghidra-noreturn.mjs",
|
||||
"tests/processOwnershipRunner.ts"
|
||||
],
|
||||
"project": ["src/**/*.ts", "tests/**/*.ts", "scripts/**/*.mjs"],
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
The MIT License (MIT)
|
||||
=====================
|
||||
|
||||
Copyright © 2023-2025 Washi
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the “Software”), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
||||
The read-only NativeAOT parser adapts format and hydration semantics from
|
||||
Washi1337/ghidra-nativeaot commit
|
||||
effeb734fc570c32650f88b159608979dc7b423e (MIT). MethodTable field layout was
|
||||
cross-checked against .NET runtime src/coreclr/nativeaot/Runtime/inc/MethodTable.h
|
||||
at dotnet/runtime commit a3b2d40328be0be3f8dd43f0e5cc925b8827b044 (MIT).
|
||||
@@ -1,5 +1,7 @@
|
||||
#include "authority.hpp"
|
||||
|
||||
#include <delayimp.h>
|
||||
|
||||
namespace rea {
|
||||
|
||||
static const napi_type_tag RESOURCE_TAG = {0x45fcbf0b49e84fe3ULL, 0x9e3e5074130de843ULL};
|
||||
@@ -121,3 +123,13 @@ static napi_value initialize(napi_env env, napi_value exports) {
|
||||
} // namespace rea
|
||||
|
||||
NAPI_MODULE(NODE_GYP_MODULE_NAME, rea::initialize)
|
||||
|
||||
// Node-API symbols are delay-imported from "node.exe". Resolve them against the
|
||||
// host executable so Node.js, Bun, and other Node-API hosts load the addon
|
||||
// regardless of their executable file name.
|
||||
static FARPROC WINAPI reaHostHook(unsigned event, PDelayLoadInfo info) {
|
||||
if (event != dliNotePreLoadLibrary || _stricmp(info->szDll, "node.exe") != 0) return nullptr;
|
||||
return reinterpret_cast<FARPROC>(GetModuleHandleW(nullptr));
|
||||
}
|
||||
|
||||
decltype(__pfnDliNotifyHook2) __pfnDliNotifyHook2 = reaHostHook;
|
||||
|
||||
@@ -219,9 +219,10 @@ std::unique_ptr<Runtime> createRuntime(const std::wstring& parentPath, const std
|
||||
require(BCryptGenRandom(nullptr, random.data(), random.size(), BCRYPT_USE_SYSTEM_PREFERRED_RNG) == 0,
|
||||
"Generate runtime identity failed", parentPath, ERROR_GEN_FAILURE);
|
||||
auto result = std::make_unique<Runtime>();
|
||||
result->path = parent->path + L"\\" + prefix + hex(random.data(), random.size());
|
||||
const auto name = prefix + hex(random.data(), random.size());
|
||||
result->path = parent->path + (parent->path.back() == L'\\' ? L"" : L"\\") + name;
|
||||
Security security;
|
||||
result->directory = createRelative(parent->get(), prefix + hex(random.data(), random.size()),
|
||||
result->directory = createRelative(parent->get(), name,
|
||||
DELETE | FILE_LIST_DIRECTORY | FILE_READ_ATTRIBUTES | READ_CONTROL,
|
||||
true, security, result->path);
|
||||
try {
|
||||
|
||||
Generated
+20
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "rea-agents",
|
||||
"version": "6.3.0",
|
||||
"version": "6.4.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "rea-agents",
|
||||
"version": "6.3.0",
|
||||
"version": "6.4.0",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@babel/parser": "8.0.4",
|
||||
@@ -17,6 +17,7 @@
|
||||
"@lydell/node-pty": "1.1.0",
|
||||
"@modelcontextprotocol/client": "2.3.1",
|
||||
"@modelcontextprotocol/server": "2.3.1",
|
||||
"@xmldom/xmldom": "0.9.12",
|
||||
"@xterm/addon-serialize": "0.14.0",
|
||||
"@xterm/headless": "6.0.0",
|
||||
"@zip.js/zip.js": "2.15.0",
|
||||
@@ -38,6 +39,7 @@
|
||||
"plist": "5.0.0",
|
||||
"semver": "7.8.5",
|
||||
"smol-toml": "1.9.0",
|
||||
"stream-chain": "4.2.6",
|
||||
"stream-json": "3.7.0",
|
||||
"write-file-atomic": "7.0.0",
|
||||
"ws": "8.21.0",
|
||||
@@ -64,6 +66,7 @@
|
||||
"oxlint": "1.87.0",
|
||||
"turbo": "2.11.7",
|
||||
"typescript": "7.0.2",
|
||||
"typescript-termux": "npm:typescript@5.9.3",
|
||||
"vitepress": "1.6.4",
|
||||
"vitest": "5.0.3"
|
||||
},
|
||||
@@ -7367,6 +7370,21 @@
|
||||
"@typescript/typescript-win32-x64": "7.0.2"
|
||||
}
|
||||
},
|
||||
"node_modules/typescript-termux": {
|
||||
"name": "typescript",
|
||||
"version": "5.9.3",
|
||||
"resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
|
||||
"integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"bin": {
|
||||
"tsc": "bin/tsc",
|
||||
"tsserver": "bin/tsserver"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=14.17"
|
||||
}
|
||||
},
|
||||
"node_modules/unbash": {
|
||||
"version": "4.0.12",
|
||||
"resolved": "https://registry.npmjs.org/unbash/-/unbash-4.0.12.tgz",
|
||||
|
||||
+15
-2
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "rea-agents",
|
||||
"mcpName": "io.github.morluto/rea",
|
||||
"version": "6.3.0",
|
||||
"version": "6.4.0",
|
||||
"description": "Reverse engineer anything from your terminal or agent with one CLI and MCP server.",
|
||||
"license": "MIT",
|
||||
"repository": {
|
||||
@@ -30,6 +30,9 @@
|
||||
"access": "public"
|
||||
},
|
||||
"type": "module",
|
||||
"imports": {
|
||||
"#javascript-analysis-worker": "./dist/composition/javascriptAnalysisWorkerProcess.js"
|
||||
},
|
||||
"bin": {
|
||||
"rea": "scripts/rea.mjs",
|
||||
"rea-agents": "scripts/rea.mjs"
|
||||
@@ -50,6 +53,7 @@
|
||||
"bridge/ghidra/ReaGhidraBridge.java",
|
||||
"bridge/ghidra/ReaGhidraExtensions.java",
|
||||
"bridge/ghidra/ReaGhidraPrepareCom.java",
|
||||
"bridge/ghidra/ReaGhidraApplySeeds.java",
|
||||
"bridge/ghidra/ReaGhidraNoReturnFix.java",
|
||||
"bridge/android/ReaJadxBridge.java",
|
||||
"bridge/mitmproxy/capture.py",
|
||||
@@ -77,9 +81,11 @@
|
||||
"bridge/pwndbg/launch_impl.py",
|
||||
"bridge/pwndbg/core_context.py",
|
||||
"third_party/pwndbg/README.md",
|
||||
"third_party/pwndbg/LICENSE.md"
|
||||
"third_party/pwndbg/LICENSE.md",
|
||||
"licenses/nativeaot-parser-MIT.txt"
|
||||
],
|
||||
"scripts": {
|
||||
"release:notes": "node scripts/release-notes.mjs",
|
||||
"deps:check": "node scripts/check-dependency-install.mjs",
|
||||
"metadata:generate": "node scripts/run-exclusive.mjs artifacts node scripts/generate-package-metadata.mjs",
|
||||
"metadata:check": "node scripts/generate-package-metadata.mjs --check",
|
||||
@@ -101,6 +107,8 @@
|
||||
"precheck:ci": "npm run deps:check",
|
||||
"check:ci": "node scripts/run-exclusive.mjs artifacts turbo run typecheck lint format:check knip metadata:check jscpd scan:todos && npm run verify:architecture-guards",
|
||||
"build:cached": "node scripts/check-dependency-install.mjs && node scripts/run-exclusive.mjs artifacts turbo run artifacts:skills",
|
||||
"build:termux": "node scripts/run-exclusive.mjs artifacts npm run build:termux:run",
|
||||
"build:termux:run": "npm run compile:termux && npm run artifacts:skills",
|
||||
"format": "oxfmt .",
|
||||
"format:check": "oxfmt --check .",
|
||||
"lint": "oxlint && npm run verify:module-boundaries",
|
||||
@@ -150,6 +158,7 @@
|
||||
"verify:apktool": "npm run build:cached && node scripts/verify-apktool.mjs",
|
||||
"verify:adb": "npm run build:cached && node scripts/verify-adb.mjs",
|
||||
"verify:ghidra": "npm run build:cached && node scripts/verify-real-ghidra.mjs",
|
||||
"verify:ghidra:seeds": "npm run build:cached && node scripts/verify/ghidra/seeds.mjs",
|
||||
"verify:ghidra:mips": "npm run build:cached && node scripts/verify-real-ghidra-mips.mjs",
|
||||
"verify:ghidra:swift": "npm run build:cached && node scripts/verify-real-ghidra-swift.mjs",
|
||||
"verify:ida": "npm run build:cached && node scripts/verify-real-ida.mjs",
|
||||
@@ -236,6 +245,7 @@
|
||||
"verify:recorded:crash": "npm run build:cached && node scripts/verify-recorded-crash.mjs",
|
||||
"mcp-catalog:check:built": "node scripts/generate-mcp-tool-catalog.mjs --check",
|
||||
"compile": "node scripts/clean-build-output.mjs && tsc -p tsconfig.build.json",
|
||||
"compile:termux": "node scripts/clean-build-output.mjs && node --max-old-space-size=3072 node_modules/typescript-termux/lib/tsc.js -p tsconfig.build.json",
|
||||
"artifacts:mcp-catalog": "node scripts/generate-mcp-tool-catalog.mjs",
|
||||
"artifacts:skills": "node scripts/generate-skill-metadata.mjs",
|
||||
"artifacts:product-catalog": "node scripts/generate-product-catalog.mjs",
|
||||
@@ -261,6 +271,7 @@
|
||||
"@lydell/node-pty": "1.1.0",
|
||||
"@modelcontextprotocol/client": "2.3.1",
|
||||
"@modelcontextprotocol/server": "2.3.1",
|
||||
"@xmldom/xmldom": "0.9.12",
|
||||
"@xterm/addon-serialize": "0.14.0",
|
||||
"@xterm/headless": "6.0.0",
|
||||
"@zip.js/zip.js": "2.15.0",
|
||||
@@ -281,6 +292,7 @@
|
||||
"plist": "5.0.0",
|
||||
"semver": "7.8.5",
|
||||
"smol-toml": "1.9.0",
|
||||
"stream-chain": "4.2.6",
|
||||
"stream-json": "3.7.0",
|
||||
"write-file-atomic": "7.0.0",
|
||||
"ws": "8.21.0",
|
||||
@@ -309,6 +321,7 @@
|
||||
"oxlint": "1.87.0",
|
||||
"turbo": "2.11.7",
|
||||
"typescript": "7.0.2",
|
||||
"typescript-termux": "npm:typescript@5.9.3",
|
||||
"vitepress": "1.6.4",
|
||||
"vitest": "5.0.3"
|
||||
}
|
||||
|
||||
@@ -78,7 +78,7 @@ const classes = await mkdtemp(join(output, "classes-"));
|
||||
try {
|
||||
const sources = [
|
||||
...(await files(join(source, "src/main/java"), ".java")),
|
||||
...(await files(join(root, "bridge/ghidra/extensions/nativeaot"), ".java")),
|
||||
...(await files(join(root, "bridge/nativeaot"), ".java")),
|
||||
];
|
||||
const jars = await files(join(install, "Ghidra"), ".jar");
|
||||
const args = [
|
||||
|
||||
@@ -84,7 +84,8 @@ if (!(await exists(join(headers, "node_api.h")))) {
|
||||
}
|
||||
const artifact = "rea-windows-x64.node";
|
||||
// MSVC short-import libraries do not produce a correct Node import table with
|
||||
// GNU ld. Generate a GNU import library for the exact stable Node-API symbols.
|
||||
// GNU ld. Generate a GNU delay-import library for the exact stable Node-API
|
||||
// symbols; addon.cc binds it to the host executable, whatever its file name.
|
||||
const sources = ["addon.cc", "filesystem.cc", "process.cc"];
|
||||
const sourceText = (
|
||||
await Promise.all(
|
||||
@@ -106,7 +107,7 @@ await writeFile(
|
||||
definition,
|
||||
`LIBRARY node.exe\nEXPORTS\n${symbols.join("\n")}\n`,
|
||||
);
|
||||
run("x86_64-w64-mingw32-dlltool", ["-d", definition, "-l", library]);
|
||||
run("x86_64-w64-mingw32-dlltool", ["-d", definition, "-y", library]);
|
||||
run(compiler, [
|
||||
"-std=c++17",
|
||||
"-O2",
|
||||
|
||||
@@ -0,0 +1,198 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { appendFileSync } from "node:fs";
|
||||
import { matchesGlob } from "node:path";
|
||||
import { SCOPES } from "./scopes.mjs";
|
||||
|
||||
const flags = [
|
||||
"code",
|
||||
"runtime",
|
||||
"package",
|
||||
"apple",
|
||||
"fixtures",
|
||||
"readiness",
|
||||
"docs",
|
||||
"workflows",
|
||||
...Object.keys(SCOPES),
|
||||
];
|
||||
// Actionlint validates workflow syntax only. The semantic suite asserts these
|
||||
// workflows' release, publication and self-hosted runner safeguards.
|
||||
const GUARDED_WORKFLOWS = new Set([
|
||||
".github/workflows/release.yml",
|
||||
".github/workflows/real-ghidra-windows.yml",
|
||||
]);
|
||||
// Sources compiled by build-conformance-fixtures.mjs. The readiness journey
|
||||
// analyzes the built C fixture as well as its own applications.
|
||||
const FIXTURE_SOURCES = [
|
||||
"tests/conformance/c/**",
|
||||
"tests/conformance/napi/**",
|
||||
"tests/conformance/objc/**",
|
||||
"tests/conformance/swift/**",
|
||||
"tests/conformance/versions/**",
|
||||
];
|
||||
const READINESS_SOURCES = ["tests/conformance/readiness/**"];
|
||||
const matchesAny = (file, patterns) =>
|
||||
patterns.some((pattern) => matchesGlob(file, pattern));
|
||||
const selected = Object.fromEntries(flags.map((flag) => [flag, false]));
|
||||
const reasons = [];
|
||||
const full = (reason) => {
|
||||
for (const flag of flags) selected[flag] = true;
|
||||
reasons.push(reason);
|
||||
};
|
||||
const git = (...args) =>
|
||||
execFileSync("git", args, { encoding: "utf8" }).trimEnd();
|
||||
function classifyPackage(base, head) {
|
||||
const before = JSON.parse(git("show", `${base}:package.json`));
|
||||
const after = JSON.parse(git("show", `${head}:package.json`));
|
||||
const { scripts: beforeScripts, ...beforeRest } = before;
|
||||
const { scripts: afterScripts, ...afterRest } = after;
|
||||
if (JSON.stringify(beforeRest) !== JSON.stringify(afterRest)) {
|
||||
full("Package metadata or dependencies changed");
|
||||
return;
|
||||
}
|
||||
selected.code = true;
|
||||
const changed = [
|
||||
...new Set([
|
||||
...Object.keys(beforeScripts ?? {}),
|
||||
...Object.keys(afterScripts ?? {}),
|
||||
]),
|
||||
].filter((key) => beforeScripts?.[key] !== afterScripts?.[key]);
|
||||
for (const script of changed) {
|
||||
if (
|
||||
/^(test(?::|$)|check(?::|$)|lint(?::|$)|format(?::|$)|knip$|jscpd$)/.test(
|
||||
script,
|
||||
)
|
||||
)
|
||||
continue;
|
||||
if (
|
||||
["compile:termux", "build:termux", "build:termux:run"].includes(script)
|
||||
) {
|
||||
selected.workflows = true;
|
||||
reasons.push(
|
||||
`Android build script changed; real Android verification remains manual: ${script}`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
// Verify scripts own provider setup as well as execution. Reuse their
|
||||
// existing source-file ownership rather than maintaining a second map.
|
||||
const entrypoint = (command) =>
|
||||
/^(?:npm run build:cached && )?node (scripts\/[\w/-]+\.mjs)(?: [\w./=-]+)*$/.exec(
|
||||
command ?? "",
|
||||
)?.[1];
|
||||
const beforeEntry = entrypoint(beforeScripts?.[script]);
|
||||
const afterEntry = entrypoint(afterScripts?.[script]);
|
||||
const entries = [beforeEntry, afterEntry].filter(Boolean);
|
||||
const simple =
|
||||
(!beforeScripts?.[script] || beforeEntry) &&
|
||||
(!afterScripts?.[script] || afterEntry);
|
||||
const scriptOwners =
|
||||
simple &&
|
||||
entries.every((entry) =>
|
||||
Object.values(SCOPES).some((patterns) => patterns.includes(entry)),
|
||||
)
|
||||
? Object.entries(SCOPES)
|
||||
.filter(([, patterns]) =>
|
||||
entries.some((entry) => patterns.includes(entry)),
|
||||
)
|
||||
.map(([owner]) => owner)
|
||||
: [];
|
||||
if (scriptOwners.length)
|
||||
for (const owner of scriptOwners) selected[owner] = true;
|
||||
else full(`Shared or unknown package script: ${script}`);
|
||||
}
|
||||
}
|
||||
|
||||
const event = process.env.GITHUB_EVENT_NAME;
|
||||
let files = [];
|
||||
if (
|
||||
event === "push" ||
|
||||
event === "schedule" ||
|
||||
event === "workflow_dispatch" ||
|
||||
process.env.CI_FULL === "true"
|
||||
) {
|
||||
full("Full baseline requested by event or ci:full label");
|
||||
} else if (event === "pull_request") {
|
||||
const base = git("merge-base", process.env.BASE_SHA, process.env.HEAD_SHA);
|
||||
const head = process.env.HEAD_SHA;
|
||||
// Disable rename detection so both the removed and added paths participate.
|
||||
files = git("diff", "--name-only", "--no-renames", "-z", base, head)
|
||||
.split("\0")
|
||||
.filter(Boolean);
|
||||
for (const file of files) {
|
||||
if (
|
||||
/^(README(?:_[^/]+)?\.md|AGENTS\.md|CONTRIBUTING\.md)$/.test(file) ||
|
||||
file.startsWith("docs/")
|
||||
) {
|
||||
selected.docs = true;
|
||||
continue;
|
||||
}
|
||||
if (/^(src|tests)\/.*\.(?:test|spec)\.[cm]?[jt]sx?$/.test(file)) {
|
||||
selected.code = true;
|
||||
continue;
|
||||
}
|
||||
const owners = Object.entries(SCOPES)
|
||||
.filter(([, patterns]) =>
|
||||
patterns.some((pattern) => matchesGlob(file, pattern)),
|
||||
)
|
||||
.map(([owner]) => owner);
|
||||
for (const owner of owners) selected[owner] = true;
|
||||
if (
|
||||
file.startsWith(".github/workflows/") ||
|
||||
file === ".github/actionlint.yaml"
|
||||
) {
|
||||
selected.workflows = true;
|
||||
if (GUARDED_WORKFLOWS.has(file)) selected.code = true;
|
||||
if (file !== ".github/workflows/ci.yml") continue;
|
||||
}
|
||||
if (owners.length === 1 && owners[0] === "website") continue;
|
||||
if (/^src\/(contracts|domain)\//.test(file)) selected.docs = true;
|
||||
if (file === "package.json") {
|
||||
classifyPackage(base, head);
|
||||
continue;
|
||||
}
|
||||
selected.code = true;
|
||||
if (matchesAny(file, FIXTURE_SOURCES)) {
|
||||
selected.fixtures = true;
|
||||
selected.readiness = true;
|
||||
}
|
||||
if (matchesAny(file, READINESS_SOURCES)) selected.readiness = true;
|
||||
if (file.startsWith("tests/")) continue;
|
||||
if (
|
||||
/^(src\/process\/|src\/filesystem\/|src\/contracts\/tool|src\/server\/createServer|src\/cli\.ts|scripts\/ci\/)/.test(
|
||||
file,
|
||||
)
|
||||
) {
|
||||
full(`Shared boundary: ${file}`);
|
||||
} else if (owners.length) {
|
||||
if (file.startsWith("src/") || file.startsWith("bridge/"))
|
||||
selected.runtime = true;
|
||||
} else {
|
||||
full(`Unclassified input: ${file}`);
|
||||
}
|
||||
}
|
||||
} else throw new Error(`Unsupported CI event: ${event}`);
|
||||
selected.runtime ||= selected.package;
|
||||
const linux = { os: "ubuntu-latest", platform: "linux", architecture: "x64" };
|
||||
const matrix = {
|
||||
include: selected.package
|
||||
? [
|
||||
linux,
|
||||
{ os: "ubuntu-24.04-arm", platform: "linux", architecture: "arm64" },
|
||||
{ os: "macos-15", platform: "darwin", architecture: "arm64" },
|
||||
{ os: "macos-15-intel", platform: "darwin", architecture: "x64" },
|
||||
]
|
||||
: [linux],
|
||||
};
|
||||
const plan = { ...selected, matrix, files, reasons };
|
||||
if (process.env.GITHUB_OUTPUT)
|
||||
appendFileSync(
|
||||
process.env.GITHUB_OUTPUT,
|
||||
Object.entries({ ...selected, matrix: JSON.stringify(matrix) })
|
||||
.map(([key, value]) => `${key}=${value}\n`)
|
||||
.join(""),
|
||||
);
|
||||
if (process.env.GITHUB_STEP_SUMMARY)
|
||||
appendFileSync(
|
||||
process.env.GITHUB_STEP_SUMMARY,
|
||||
`### Selected CI lanes\n\n${flags.filter((flag) => selected[flag]).join(", ") || "Formatting only"}\n\n${reasons.join("\n\n")}\n`,
|
||||
);
|
||||
console.log(JSON.stringify(plan));
|
||||
@@ -0,0 +1,25 @@
|
||||
const jobs = JSON.parse(process.env.CI_JOBS);
|
||||
const outputs = jobs.changes.outputs;
|
||||
const required = ["changes", "static", "test"];
|
||||
const lanes = {
|
||||
code: ["build", "test-shard"],
|
||||
runtime: ["windows-native-build", "package-e2e"],
|
||||
package: ["windows-curated"],
|
||||
apple: ["apple-artifacts"],
|
||||
fixtures: ["conformance-fixtures"],
|
||||
readiness: ["reconstruction-readiness"],
|
||||
docs: ["docs"],
|
||||
};
|
||||
for (const [scope, value] of Object.entries(outputs)) {
|
||||
if (value === "true") required.push(...(lanes[scope] ?? [scope]));
|
||||
}
|
||||
const failed = [...new Set(required)].filter(
|
||||
(name) => jobs[name]?.result !== "success",
|
||||
);
|
||||
for (const [name, job] of Object.entries(jobs)) {
|
||||
if (["failure", "cancelled"].includes(job.result) && !failed.includes(name))
|
||||
failed.push(name);
|
||||
}
|
||||
if (failed.length)
|
||||
throw new Error(`CI lanes did not pass: ${failed.join(", ")}`);
|
||||
console.log("Every selected CI lane passed");
|
||||
@@ -0,0 +1,228 @@
|
||||
// Ownership of the real-provider and website verification lanes. Shared build
|
||||
// inputs are classified once in plan.mjs rather than repeated in every workflow.
|
||||
export const SCOPES = {
|
||||
browser: [
|
||||
".github/workflows/real-browser.yml",
|
||||
"scripts/verify-real-browser.mjs",
|
||||
"scripts/verify-browser-network.mjs",
|
||||
"scripts/verify-browser-scripts.mjs",
|
||||
"scripts/verify-browser-modules.mjs",
|
||||
"scripts/verify-browser-dom-destinations.mjs",
|
||||
"scripts/lib/browser-*.mjs",
|
||||
"scripts/fixtures/browser-network-site.mjs",
|
||||
"scripts/fixtures/browser-script-site.mjs",
|
||||
"scripts/fixtures/browser-module-site.mjs",
|
||||
"scripts/fixtures/browser-verifier-site.mjs",
|
||||
"scripts/fixtures/page-cdp-proxy.mjs",
|
||||
"src/browser/**",
|
||||
"src/domain/browser*.ts",
|
||||
"src/domain/webScript*.ts",
|
||||
"src/domain/webModule*.ts",
|
||||
"src/application/WebModule*.ts",
|
||||
"src/composition/webModules.ts",
|
||||
"src/artifacts/readStableArtifact.ts",
|
||||
"src/artifacts/ArtifactPaths.ts",
|
||||
"src/application/WebScriptExportService.ts",
|
||||
"src/contracts/webScriptToolContracts.ts",
|
||||
"src/server/registerWebScriptTool.ts",
|
||||
"src/server/registerWebModuleTool.ts",
|
||||
"src/server/createServer.ts",
|
||||
"src/cli/webModuleCommands.ts",
|
||||
"src/cli.ts",
|
||||
"src/cliCommandNames.ts",
|
||||
"src/contracts/toolEffects.ts",
|
||||
"src/cli/webScriptCommands.ts",
|
||||
],
|
||||
binary_layout: [
|
||||
".github/workflows/real-binary-layout.yml",
|
||||
"bridge/pwntools/**",
|
||||
"src/native/pwntools/**",
|
||||
"src/domain/native/binaryLayout*",
|
||||
"src/application/binaryDiagnostics/**",
|
||||
"src/contracts/native/binaryDiagnosticsToolContracts.ts",
|
||||
"src/composition/binaryDiagnostics.ts",
|
||||
"src/server/registerBinaryDiagnosticsTools.ts",
|
||||
"src/server/createServer.ts",
|
||||
"src/cli/binaryDiagnosticsCommands.ts",
|
||||
"src/cli.ts",
|
||||
"src/cliCommandNames.ts",
|
||||
"src/contracts/toolContracts.ts",
|
||||
"src/contracts/toolEffects.ts",
|
||||
"src/process/OwnedCommand.ts",
|
||||
"src/process/ProviderProcess.ts",
|
||||
"src/process/ProviderDeadline.ts",
|
||||
"src/process/ProcessOwnership*.ts",
|
||||
"src/process/PrivateRuntimeRoot.ts",
|
||||
"src/artifacts/readStableArtifact*.ts",
|
||||
"src/artifacts/ArtifactReader.ts",
|
||||
"src/domain/analysisError*.ts",
|
||||
"scripts/verify-binary-layout.mjs",
|
||||
"scripts/lib/elf-section-name-fixtures.mjs",
|
||||
"scripts/lib/large-result-mcp.mjs",
|
||||
"scripts/fixtures/binary-layout*",
|
||||
],
|
||||
evm: [
|
||||
".github/workflows/real-evm-interface.yml",
|
||||
"src/evm/**",
|
||||
"src/domain/evm/**",
|
||||
"src/application/evm/**",
|
||||
"src/contracts/evm/**",
|
||||
"src/composition/evm.ts",
|
||||
"src/cli/evmCommands.ts",
|
||||
"src/server/registerEvmTools.ts",
|
||||
"src/server/createServer.ts",
|
||||
"src/cli.ts",
|
||||
"src/cliCommandNames.ts",
|
||||
"src/contracts/toolContracts.ts",
|
||||
"src/contracts/toolEffects.ts",
|
||||
"src/process/OwnedCommand.ts",
|
||||
"src/process/ProviderProcess.ts",
|
||||
"src/process/ProcessOwnership*.ts",
|
||||
"src/process/ProviderDeadline.ts",
|
||||
"src/process/PrivateRuntimeRoot.ts",
|
||||
"src/artifacts/readStableArtifact*.ts",
|
||||
"src/artifacts/ArtifactReader.ts",
|
||||
"scripts/verify-evm-interface.mjs",
|
||||
"scripts/fixtures/evm-interface.sol",
|
||||
"third_party/evmole/**",
|
||||
],
|
||||
javascript_recovery: [
|
||||
".github/workflows/real-javascript-recovery.yml",
|
||||
"src/javascript/recovery/**",
|
||||
"src/domain/javascript/**",
|
||||
"src/application/javascript/JavaScriptRecovery*.ts",
|
||||
"src/composition/javascriptRecovery.ts",
|
||||
"src/contracts/javascript/javascriptRecoveryToolContracts.ts",
|
||||
"src/cli/javascriptRecoveryCommands.ts",
|
||||
"src/cli/javascriptApplicationAnalysis.ts",
|
||||
"src/server/registerJavaScriptRecoveryTool.ts",
|
||||
"src/server/createServer.ts",
|
||||
"src/server/sessionAvailabilityPolicy.ts",
|
||||
"src/application/CapabilityInventory.ts",
|
||||
"src/cli.ts",
|
||||
"src/cliCommandNames.ts",
|
||||
"src/contracts/toolContracts.ts",
|
||||
"src/contracts/toolEffects.ts",
|
||||
"scripts/rea.mjs",
|
||||
"src/process/**",
|
||||
"src/artifacts/SafeOutputTree.ts",
|
||||
"scripts/verify/javascript/recovery.mjs",
|
||||
"scripts/fixtures/javascript-recovery/**",
|
||||
"third_party/wakaru",
|
||||
],
|
||||
recorded_crash: [
|
||||
".github/workflows/real-recorded-crash.yml",
|
||||
"bridge/pwntools/**",
|
||||
"bridge/pwndbg/**",
|
||||
"src/native/pwntools/**",
|
||||
"src/native/pwndbg/**",
|
||||
"src/domain/native/recordedCrash*",
|
||||
"src/application/binaryDiagnostics/**",
|
||||
"src/contracts/native/binaryDiagnosticsToolContracts.ts",
|
||||
"src/contracts/toolEffects.ts",
|
||||
"src/contracts/toolOutputSchemaGroups.ts",
|
||||
"src/composition/binaryDiagnostics.ts",
|
||||
"src/server/registerRecordedCrashTools.ts",
|
||||
"src/server/createServer.ts",
|
||||
"src/server/sessionAvailabilityPolicy.ts",
|
||||
"src/cli/binaryDiagnosticsCommands.ts",
|
||||
"src/cliCommandNames.ts",
|
||||
"src/process/**",
|
||||
"src/artifacts/readStableArtifact*.ts",
|
||||
"src/domain/analysisError*.ts",
|
||||
"scripts/verify-recorded-crash.mjs",
|
||||
"scripts/verify/recorded-crash/**",
|
||||
"scripts/lib/recorded-core-fixtures.py",
|
||||
"scripts/fixtures/recorded-crash.c",
|
||||
"third_party/pwndbg/**",
|
||||
],
|
||||
web_network: [
|
||||
".github/workflows/real-web-network-captures.yml",
|
||||
"bridge/mitmproxy/**",
|
||||
"src/browser/history/**",
|
||||
"src/domain/webNetworkCapture*.ts",
|
||||
"src/domain/analysisError*.ts",
|
||||
"src/contracts/errorSchemas.ts",
|
||||
"src/domain/explicitSensitive*.ts",
|
||||
"src/application/WebNetworkCapture*",
|
||||
"src/composition/webNetworkCaptures.ts",
|
||||
"src/cli/webNetworkCaptureCommands.ts",
|
||||
"src/server/registerWebNetworkCaptureTool.ts",
|
||||
"src/server/createServer.ts",
|
||||
"src/server/toolRegistrationOptions.ts",
|
||||
"src/cli.ts",
|
||||
"src/cliCommandNames.ts",
|
||||
"src/contracts/toolContracts.ts",
|
||||
"src/contracts/toolEffects.ts",
|
||||
"src/process/OwnedCommand.ts",
|
||||
"src/process/ProviderProcess.ts",
|
||||
"src/process/ProviderDeadline.ts",
|
||||
"src/process/ProcessOwnership*.ts",
|
||||
"src/process/PrivateRuntimeRoot.ts",
|
||||
"src/process/capture/ProcessCaptureLifecycle.ts",
|
||||
"src/artifacts/readStableArtifact*.ts",
|
||||
"src/artifacts/ArtifactReader.ts",
|
||||
"src/contracts/webNetworkCaptureToolContracts.ts",
|
||||
"scripts/verify-web-network-captures.mjs",
|
||||
"scripts/fixtures/generate-mitmproxy-capture.py",
|
||||
],
|
||||
web_runtime: [
|
||||
".github/workflows/real-web-runtime.yml",
|
||||
"scripts/verify-browser-runtime.mjs",
|
||||
"scripts/lib/browser-runtime-*.mjs",
|
||||
"src/process/ProviderProcess.ts",
|
||||
"src/process/ProcessOwnership*.ts",
|
||||
"scripts/fixtures/browser-runtime-site.mjs",
|
||||
"src/browser/execution/**",
|
||||
"src/browser/CdpConnection.ts",
|
||||
"src/browser/CdpEndpoint.ts",
|
||||
"src/browser/CdpTarget*.ts",
|
||||
"src/browser/CdpCaptureValues.ts",
|
||||
"src/browser/CdpAuthorized*.ts",
|
||||
"src/domain/webRuntime.ts",
|
||||
"src/domain/webExecution.ts",
|
||||
"src/domain/webEventListeners.ts",
|
||||
"src/domain/browserObservationError*.ts",
|
||||
"src/application/WebRuntime*.ts",
|
||||
"src/composition/webRuntime.ts",
|
||||
"src/contracts/webRuntimeToolContracts.ts",
|
||||
"src/contracts/toolEffects.ts",
|
||||
"src/server/registerWebRuntimeTools.ts",
|
||||
"src/server/createServer.ts",
|
||||
"src/cli/webRuntimeCommands.ts",
|
||||
"src/cli/commandCancellation.ts",
|
||||
"src/cli.ts",
|
||||
],
|
||||
web_source_map: [
|
||||
".github/workflows/real-web-source-map.yml",
|
||||
"scripts/verify-browser-source-maps.mjs",
|
||||
"scripts/lib/browser-source-map-e2e.mjs",
|
||||
"scripts/fixtures/browser-source-map-site.mjs",
|
||||
"src/javascript/sourceMaps/**",
|
||||
"src/browser/assets/**",
|
||||
"src/domain/webSource*.ts",
|
||||
"src/domain/webScript*.ts",
|
||||
"src/application/WebSourceLocation*.ts",
|
||||
"src/application/ValidateSelectedWebScript.ts",
|
||||
"src/composition/webSourceLocations.ts",
|
||||
"src/server/registerWebSourceLocationTool.ts",
|
||||
"src/server/createServer.ts",
|
||||
"src/contracts/webScriptToolContracts.ts",
|
||||
"src/process/**",
|
||||
"src/cli/webSourceLocationCommands.ts",
|
||||
"src/cli.ts",
|
||||
"src/artifacts/readStableArtifact.ts",
|
||||
],
|
||||
nativeaot: [
|
||||
"bridge/nativeaot/**",
|
||||
"scripts/generate-nativeaot-fixtures.mjs",
|
||||
"tests/fixtures/nativeaot/**",
|
||||
".github/workflows/nativeaot-fixtures.yml",
|
||||
],
|
||||
website: [
|
||||
"website/**",
|
||||
"scripts/*website*.py",
|
||||
"scripts/test-website-scripts.mjs",
|
||||
".github/workflows/website-check.yml",
|
||||
],
|
||||
};
|
||||
@@ -0,0 +1,61 @@
|
||||
/* Keep the configured launcher identity alive while its resource-limited
|
||||
* Python child runs. Both remain in REA's owned process group. */
|
||||
#define _POSIX_C_SOURCE 200809L
|
||||
#include <errno.h>
|
||||
#include <signal.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <sys/resource.h>
|
||||
#include <sys/types.h>
|
||||
#include <sys/wait.h>
|
||||
#include <unistd.h>
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
char *python = getenv("REA_VERIFY_LIMITED_PYTHON");
|
||||
char *limit = getenv("REA_VERIFY_LIMIT_OPTION");
|
||||
if (python == NULL || python[0] != '/' || limit == NULL) {
|
||||
fputs("limited Python fixture requires an absolute Python and limit\n", stderr);
|
||||
return 127;
|
||||
}
|
||||
struct rlimit core = {0, 0};
|
||||
if (setrlimit(RLIMIT_CORE, &core) != 0) {
|
||||
perror("setrlimit");
|
||||
return 127;
|
||||
}
|
||||
char **arguments = calloc((size_t)argc + 4, sizeof(char *));
|
||||
if (arguments == NULL) return 127;
|
||||
arguments[0] = "/usr/bin/prlimit";
|
||||
arguments[1] = limit;
|
||||
arguments[2] = "--";
|
||||
arguments[3] = python;
|
||||
for (int index = 1; index < argc; index++) arguments[index + 3] = argv[index];
|
||||
pid_t child = fork();
|
||||
if (child == 0) {
|
||||
execv(arguments[0], arguments);
|
||||
perror("execv");
|
||||
_exit(127);
|
||||
}
|
||||
free(arguments);
|
||||
if (child < 0) {
|
||||
perror("fork");
|
||||
return 127;
|
||||
}
|
||||
int status;
|
||||
while (waitpid(child, &status, 0) < 0) {
|
||||
if (errno == EINTR) continue;
|
||||
perror("waitpid");
|
||||
return 127;
|
||||
}
|
||||
if (WIFEXITED(status)) return WEXITSTATUS(status);
|
||||
if (WIFSIGNALED(status)) {
|
||||
int signum = WTERMSIG(status);
|
||||
signal(signum, SIG_DFL);
|
||||
sigset_t signals;
|
||||
sigemptyset(&signals);
|
||||
sigaddset(&signals, signum);
|
||||
sigprocmask(SIG_UNBLOCK, &signals, NULL);
|
||||
raise(signum);
|
||||
return 128 + signum;
|
||||
}
|
||||
return 127;
|
||||
}
|
||||
@@ -28,6 +28,11 @@ import {
|
||||
isolatedFixtureUrl,
|
||||
} from "./browser-runtime-world-e2e.mjs";
|
||||
|
||||
// Arming, progress delivery, isolated-world proof and the page action cross
|
||||
// separate transports. Leave time for their round trips on shared CI runners;
|
||||
// coverage can include a late action even after request collection has ended.
|
||||
const executionObservationMs = 2_000;
|
||||
|
||||
/** Exercise native browser runtime attribution through public CLI and stdio MCP, including arming. */
|
||||
export async function verifyBrowserRuntime(
|
||||
executable,
|
||||
@@ -210,7 +215,7 @@ async function executionMcp(
|
||||
const response = await client.callTool(
|
||||
{
|
||||
name: "observe_web_execution",
|
||||
arguments: { ...input, observation_ms: 300 },
|
||||
arguments: { ...input, observation_ms: executionObservationMs },
|
||||
},
|
||||
{
|
||||
timeout: 40_000,
|
||||
@@ -259,7 +264,7 @@ async function executionCli(entrypoint, input, env, action, isolatedContext) {
|
||||
input.cdp_endpoint,
|
||||
input.target_id,
|
||||
"--observation-ms",
|
||||
"300",
|
||||
String(executionObservationMs),
|
||||
"--json",
|
||||
],
|
||||
{ env, stdio: ["ignore", "pipe", "pipe"] },
|
||||
@@ -382,7 +387,14 @@ function assertExecution(evidence, site, requireBlock = true) {
|
||||
const request = result.requests.find((item) =>
|
||||
item.url.endsWith("/evidence?marker=chosen"),
|
||||
);
|
||||
assert.ok(request, "The selected request initiator must be retained");
|
||||
assert.ok(
|
||||
request,
|
||||
`The selected request initiator must be retained: ${JSON.stringify({
|
||||
window: result.window,
|
||||
requests: result.requests.map((item) => item.url),
|
||||
excluded_requests: result.excluded_requests,
|
||||
})}`,
|
||||
);
|
||||
assert.ok(
|
||||
request.callsites.some(
|
||||
(site) =>
|
||||
|
||||
@@ -18,6 +18,7 @@ export async function startRuntimeFixtureBrowser(executable, origin) {
|
||||
command: executable,
|
||||
arguments: [
|
||||
"--headless=new",
|
||||
"--password-store=basic",
|
||||
...(process.env.REA_BROWSER_NO_SANDBOX === "true"
|
||||
? ["--no-sandbox"]
|
||||
: []),
|
||||
|
||||
@@ -0,0 +1,683 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { createServer } from "node:http";
|
||||
import { once } from "node:events";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
|
||||
import { CdpConnection } from "../../dist/browser/CdpConnection.js";
|
||||
import { createBrowserScenarioProvider } from "../../dist/composition/browserScenario.js";
|
||||
import { browserScenarioSchema } from "../../dist/domain/browserScenario.js";
|
||||
import {
|
||||
runScenarioCli,
|
||||
scenarioProfiles,
|
||||
} from "./browser-scenario-verifier.mjs";
|
||||
|
||||
const scenarioFor = (browser, origin) =>
|
||||
browserScenarioSchema.parse({
|
||||
browser,
|
||||
start_url: { url: `${origin}/storage-main` },
|
||||
actions: [
|
||||
{
|
||||
step_id: "frame-ready",
|
||||
action: "wait_for",
|
||||
locator: { kind: "css", selector: "#frame-report[data-ready='true']" },
|
||||
state: "attached",
|
||||
timeout_ms: 10_000,
|
||||
},
|
||||
{
|
||||
step_id: "top-level-partition",
|
||||
action: "goto",
|
||||
destination: {
|
||||
url: `${origin.replace("127.0.0.1", "localhost")}/storage-top`,
|
||||
},
|
||||
wait_until: "load",
|
||||
},
|
||||
{
|
||||
step_id: "return-to-origin",
|
||||
action: "goto",
|
||||
destination: { url: `${origin}/storage-main` },
|
||||
wait_until: "load",
|
||||
},
|
||||
],
|
||||
storage: {
|
||||
local_storage: [
|
||||
{
|
||||
origin,
|
||||
entries: [
|
||||
{
|
||||
name: "rea_storage_probe",
|
||||
value: { source: "literal", value: "main-initial" },
|
||||
},
|
||||
{
|
||||
name: "rea_storage_remove",
|
||||
value: { source: "literal", value: "main-remove-seed" },
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
origin: origin.replace("127.0.0.1", "localhost"),
|
||||
entries: [
|
||||
{
|
||||
name: "rea_storage_probe",
|
||||
value: { source: "literal", value: "frame-initial" },
|
||||
},
|
||||
{
|
||||
name: "rea_storage_remove",
|
||||
value: { source: "literal", value: "frame-remove-seed" },
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
session_storage: [
|
||||
{
|
||||
origin,
|
||||
entries: [
|
||||
{
|
||||
name: "rea_storage_probe",
|
||||
value: { source: "literal", value: "main-session-initial" },
|
||||
},
|
||||
{
|
||||
name: "rea_storage_empty",
|
||||
value: { source: "literal", value: "main-empty-seed" },
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
origin: origin.replace("127.0.0.1", "localhost"),
|
||||
entries: [
|
||||
{
|
||||
name: "rea_storage_probe",
|
||||
value: { source: "literal", value: "frame-session-initial" },
|
||||
},
|
||||
{
|
||||
name: "rea_storage_empty",
|
||||
value: { source: "literal", value: "frame-empty-seed" },
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
capture: { after_each_step: ["dom"], at_end: ["dom"] },
|
||||
});
|
||||
|
||||
const raceWithTimeout = (operation, timeoutMs, onTimeout = () => undefined) => {
|
||||
const controller = new AbortController();
|
||||
return Promise.race([
|
||||
operation,
|
||||
delay(timeoutMs, undefined, { signal: controller.signal }).then(onTimeout),
|
||||
]).finally(() => controller.abort());
|
||||
};
|
||||
|
||||
const storagePage = (kind, port) => {
|
||||
if (kind === "main")
|
||||
return `<!doctype html><html><body><pre id="main-report"></pre><pre id="frame-report"></pre><iframe src="http://localhost:${port}/storage-frame"></iframe><script>
|
||||
const state = { local: localStorage.getItem("rea_storage_probe"), session: sessionStorage.getItem("rea_storage_probe"), removed: localStorage.getItem("rea_storage_remove"), empty: sessionStorage.getItem("rea_storage_empty") };
|
||||
document.querySelector("#main-report").textContent = JSON.stringify(state);
|
||||
localStorage.setItem("rea_storage_probe", "main-updated");
|
||||
sessionStorage.setItem("rea_storage_probe", "main-session-updated");
|
||||
localStorage.removeItem("rea_storage_remove");
|
||||
sessionStorage.setItem("rea_storage_empty", "");
|
||||
addEventListener("message", (event) => {
|
||||
if (event.data?.kind !== "rea-storage-frame") return;
|
||||
const report = document.querySelector("#frame-report");
|
||||
report.textContent = JSON.stringify(event.data.state);
|
||||
report.dataset.ready = "true";
|
||||
});
|
||||
</script></body></html>`;
|
||||
if (kind === "frame")
|
||||
return `<!doctype html><html><body><pre id="frame-state"></pre><script>
|
||||
const state = { local: localStorage.getItem("rea_storage_probe"), session: sessionStorage.getItem("rea_storage_probe"), removed: localStorage.getItem("rea_storage_remove"), empty: sessionStorage.getItem("rea_storage_empty") };
|
||||
document.querySelector("#frame-state").textContent = JSON.stringify(state);
|
||||
localStorage.setItem("rea_storage_probe", "frame-updated");
|
||||
sessionStorage.setItem("rea_storage_probe", "frame-session-updated");
|
||||
localStorage.removeItem("rea_storage_remove");
|
||||
sessionStorage.setItem("rea_storage_empty", "");
|
||||
parent.postMessage({ kind: "rea-storage-frame", state }, "*");
|
||||
</script></body></html>`;
|
||||
if (kind === "debugger")
|
||||
return `<!doctype html><html><body><pre id="debugger-state"></pre><script>
|
||||
window.reaDebuggerBefore = true;
|
||||
debugger;
|
||||
window.reaDebuggerAfter = true;
|
||||
document.querySelector("#debugger-state").textContent = String(window.reaDebuggerAfter);
|
||||
</script></body></html>`;
|
||||
return `<!doctype html><html><body><pre id="top-report"></pre><script>
|
||||
const state = { local: localStorage.getItem("rea_storage_probe"), session: sessionStorage.getItem("rea_storage_probe"), removed: localStorage.getItem("rea_storage_remove"), empty: sessionStorage.getItem("rea_storage_empty") };
|
||||
document.querySelector("#top-report").textContent = JSON.stringify(state);
|
||||
localStorage.setItem("rea_storage_probe", "top-updated");
|
||||
sessionStorage.setItem("rea_storage_probe", "top-session-updated");
|
||||
</script></body></html>`;
|
||||
};
|
||||
|
||||
const startStorageSite = async () => {
|
||||
let port = 0;
|
||||
const server = createServer((request, response) => {
|
||||
const path = new URL(request.url ?? "/", "http://127.0.0.1").pathname;
|
||||
const kind =
|
||||
path === "/storage-main"
|
||||
? "main"
|
||||
: path === "/storage-frame"
|
||||
? "frame"
|
||||
: path === "/storage-debugger"
|
||||
? "debugger"
|
||||
: path === "/storage-top"
|
||||
? "top"
|
||||
: undefined;
|
||||
if (kind === undefined) {
|
||||
response.writeHead(404).end();
|
||||
return;
|
||||
}
|
||||
response.setHeader("content-type", "text/html; charset=utf-8");
|
||||
response.end(storagePage(kind, port));
|
||||
});
|
||||
server.listen(0, "::");
|
||||
await once(server, "listening");
|
||||
port = server.address().port;
|
||||
return {
|
||||
origin: `http://127.0.0.1:${port}`,
|
||||
close: async () => {
|
||||
server.close();
|
||||
// Chrome can leave speculative TCP connections without an HTTP request;
|
||||
// server.close() alone waits indefinitely for those owned sockets.
|
||||
server.closeAllConnections();
|
||||
await once(server, "close");
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
const domText = (step) => {
|
||||
assert.equal(step.artifacts.dom.state, "captured");
|
||||
return step.artifacts.dom.value.text;
|
||||
};
|
||||
|
||||
const assertContainsState = (text, id, expected) => {
|
||||
const match = new RegExp(`<pre id="${id}"[^>]*>(.*?)</pre>`, "su").exec(text);
|
||||
assert.ok(match, `Captured DOM omitted #${id}: ${text}`);
|
||||
assert.deepEqual(JSON.parse(match[1]), expected);
|
||||
};
|
||||
|
||||
/** Verify one-time local/session storage initialization on a real Chromium. */
|
||||
export async function verifyScenarioStorage({
|
||||
executable,
|
||||
endpoint,
|
||||
targetId,
|
||||
}) {
|
||||
const site = await startStorageSite();
|
||||
try {
|
||||
await verifyLaunchStorage(executable, site.origin);
|
||||
await verifyConnectedStorage(endpoint, targetId, site.origin);
|
||||
return {
|
||||
mocked: false,
|
||||
launch: true,
|
||||
connect: true,
|
||||
oopifInitialization: true,
|
||||
firstInlineScriptSeeded: true,
|
||||
updatesAndDeletesPersistAcrossNavigation: true,
|
||||
newScenarioGetsFreshSeed: true,
|
||||
externalTargetSurvives: true,
|
||||
cancellationCleanup: true,
|
||||
};
|
||||
} finally {
|
||||
await site.close();
|
||||
}
|
||||
}
|
||||
|
||||
const verifyLaunchStorage = async (executable, origin) => {
|
||||
const scenario = scenarioFor(
|
||||
{ mode: "launch", executable_path: executable },
|
||||
origin,
|
||||
);
|
||||
const profilesBefore = await scenarioProfiles();
|
||||
const result = await runScenarioCli(scenario);
|
||||
const capture = result.normalized_result;
|
||||
assert.ok(capture, "Launch-mode storage scenario returned no result");
|
||||
assertLaunchStorageStates(capture);
|
||||
assert.equal(capture.browser.cleanup, "terminated-owned-process");
|
||||
const profilesAfter = await scenarioProfiles();
|
||||
assert.ok([...profilesAfter].every((profile) => profilesBefore.has(profile)));
|
||||
|
||||
const freshScenario = browserScenarioSchema.parse({
|
||||
...scenario,
|
||||
actions: [scenario.actions[0]],
|
||||
});
|
||||
const fresh = await createBrowserScenarioProvider(
|
||||
process.env,
|
||||
).captureScenario(freshScenario);
|
||||
assert.equal(fresh.ok, true);
|
||||
if (!fresh.ok) throw fresh.error;
|
||||
assertContainsState(domText(fresh.value.steps[0]), "main-report", {
|
||||
local: "main-initial",
|
||||
session: "main-session-initial",
|
||||
removed: "main-remove-seed",
|
||||
empty: "main-empty-seed",
|
||||
});
|
||||
};
|
||||
|
||||
const assertLaunchStorageStates = (capture) => {
|
||||
assertContainsState(domText(capture.steps[0]), "main-report", {
|
||||
local: "main-initial",
|
||||
session: "main-session-initial",
|
||||
removed: "main-remove-seed",
|
||||
empty: "main-empty-seed",
|
||||
});
|
||||
assertContainsState(domText(capture.steps[1]), "frame-report", {
|
||||
local: "frame-initial",
|
||||
session: "frame-session-initial",
|
||||
removed: "frame-remove-seed",
|
||||
empty: "frame-empty-seed",
|
||||
});
|
||||
assertContainsState(domText(capture.steps[2]), "top-report", {
|
||||
local: "frame-updated",
|
||||
session: "frame-session-updated",
|
||||
removed: null,
|
||||
empty: "",
|
||||
});
|
||||
assertContainsState(domText(capture.steps[3]), "main-report", {
|
||||
local: "main-updated",
|
||||
session: "main-session-updated",
|
||||
removed: null,
|
||||
empty: "",
|
||||
});
|
||||
assertContainsState(domText(capture.steps[3]), "frame-report", {
|
||||
local: "top-updated",
|
||||
session: "top-session-updated",
|
||||
removed: null,
|
||||
empty: "",
|
||||
});
|
||||
};
|
||||
|
||||
const verifyConnectedStorage = async (endpoint, targetId, origin) => {
|
||||
const scenario = scenarioFor(
|
||||
{ mode: "connect", cdp_endpoint: endpoint, target_id: targetId },
|
||||
origin,
|
||||
);
|
||||
const targets = await (
|
||||
await fetch(`${endpoint}/json/list`, { signal: AbortSignal.timeout(2_000) })
|
||||
).json();
|
||||
const target = targets.find(({ id }) => id === targetId);
|
||||
assert.ok(target, "External browser target is missing");
|
||||
const observer = await CdpConnection.connect(
|
||||
target.webSocketDebuggerUrl,
|
||||
"capture_browser_scenario",
|
||||
);
|
||||
try {
|
||||
await assertStorageKeyPartition(observer, origin);
|
||||
const result = await runScenarioCli(scenario);
|
||||
const capture = result.normalized_result;
|
||||
assert.ok(capture, "Connect-mode storage scenario returned no result");
|
||||
assertContainsState(domText(capture.steps[0]), "main-report", {
|
||||
local: "main-initial",
|
||||
session: "main-session-initial",
|
||||
removed: "main-remove-seed",
|
||||
empty: "main-empty-seed",
|
||||
});
|
||||
assert.equal(capture.browser.cleanup, "disconnected-external");
|
||||
await verifyCallerDebuggerPause(observer, origin, scenario);
|
||||
await verifyCancelledStorage(observer, endpoint, targetId, scenario);
|
||||
} finally {
|
||||
await raceWithTimeout(restoreTarget(observer, target.url), 1_000).catch(
|
||||
() => undefined,
|
||||
);
|
||||
await raceWithTimeout(observer.close(), 1_000).catch(() => undefined);
|
||||
}
|
||||
};
|
||||
|
||||
const assertStorageKeyPartition = async (connection, origin) => {
|
||||
const attachedSessions = new Set();
|
||||
const unsubscribe = connection.onEvent((event) => {
|
||||
if (
|
||||
event.method === "Target.attachedToTarget" &&
|
||||
typeof event.params === "object" &&
|
||||
event.params !== null &&
|
||||
"sessionId" in event.params &&
|
||||
typeof event.params.sessionId === "string" &&
|
||||
"targetInfo" in event.params &&
|
||||
typeof event.params.targetInfo === "object" &&
|
||||
event.params.targetInfo !== null &&
|
||||
"type" in event.params.targetInfo &&
|
||||
event.params.targetInfo.type === "iframe"
|
||||
)
|
||||
attachedSessions.add(event.params.sessionId);
|
||||
});
|
||||
await connection.send("Page.enable");
|
||||
await connection.send("Target.setAutoAttach", {
|
||||
autoAttach: true,
|
||||
waitForDebuggerOnStart: false,
|
||||
flatten: true,
|
||||
});
|
||||
try {
|
||||
const start = await connection.send("Runtime.evaluate", {
|
||||
expression: "performance.timeOrigin",
|
||||
returnByValue: true,
|
||||
});
|
||||
await connection.send("Page.navigate", {
|
||||
url: `${origin}/storage-main`,
|
||||
});
|
||||
await waitForPagePath(connection, "/storage-main", start.result.value);
|
||||
|
||||
let childSession;
|
||||
let childFrameId;
|
||||
const deadline = Date.now() + 10_000;
|
||||
while (Date.now() < deadline && childFrameId === undefined) {
|
||||
for (const sessionId of attachedSessions) {
|
||||
try {
|
||||
const tree = await connection.send(
|
||||
"Page.getFrameTree",
|
||||
{},
|
||||
sessionId,
|
||||
);
|
||||
const frame = tree.frameTree?.frame;
|
||||
if (
|
||||
typeof frame?.id === "string" &&
|
||||
frame.url ===
|
||||
`${origin.replace("127.0.0.1", "localhost")}/storage-frame`
|
||||
) {
|
||||
childSession = sessionId;
|
||||
childFrameId = frame.id;
|
||||
break;
|
||||
}
|
||||
} catch {
|
||||
// The iframe may detach between target discovery and frame inspection.
|
||||
}
|
||||
}
|
||||
if (childFrameId === undefined) await delay(25);
|
||||
}
|
||||
assert.ok(
|
||||
childSession && childFrameId,
|
||||
"Storage fixture OOPIF was not attached",
|
||||
);
|
||||
const child = await connection.send(
|
||||
"Storage.getStorageKey",
|
||||
{ frameId: childFrameId },
|
||||
childSession,
|
||||
);
|
||||
const rootTree = await connection.send("Page.getFrameTree");
|
||||
const beforeRoot = await connection.send("Storage.getStorageKey", {
|
||||
frameId: rootTree.frameTree.frame.id,
|
||||
});
|
||||
await connection.send("Page.navigate", {
|
||||
url: `${origin.replace("127.0.0.1", "localhost")}/storage-top`,
|
||||
});
|
||||
await waitForPagePath(connection, "/storage-top", start.result.value);
|
||||
const afterTree = await connection.send("Page.getFrameTree");
|
||||
const topLevel = await connection.send("Storage.getStorageKey", {
|
||||
frameId: afterTree.frameTree.frame.id,
|
||||
});
|
||||
assert.notEqual(
|
||||
child.storageKey,
|
||||
topLevel.storageKey,
|
||||
`Chrome did not expose distinct partition keys: child=${JSON.stringify(child)} top=${JSON.stringify(topLevel)} rootBefore=${JSON.stringify(beforeRoot)}`,
|
||||
);
|
||||
} finally {
|
||||
unsubscribe();
|
||||
await connection.send("Target.setAutoAttach", {
|
||||
autoAttach: false,
|
||||
waitForDebuggerOnStart: false,
|
||||
flatten: true,
|
||||
});
|
||||
}
|
||||
};
|
||||
|
||||
const waitForPagePath = async (connection, path, priorTimeOrigin) => {
|
||||
const deadline = Date.now() + 10_000;
|
||||
while (Date.now() < deadline) {
|
||||
const result = await connection.send("Runtime.evaluate", {
|
||||
expression: `JSON.stringify({ path: location.pathname, time: performance.timeOrigin })`,
|
||||
returnByValue: true,
|
||||
});
|
||||
const current = JSON.parse(result.result.value);
|
||||
if (current.path === path && current.time !== priorTimeOrigin) return;
|
||||
await delay(25);
|
||||
}
|
||||
throw new Error(`External browser did not navigate to ${path}`);
|
||||
};
|
||||
|
||||
const verifyCallerDebuggerPause = async (caller, origin, scenario) => {
|
||||
await caller.send("Runtime.enable");
|
||||
await caller.send("Debugger.enable");
|
||||
const controller = new AbortController();
|
||||
const debuggerUrl = `${origin}/storage-debugger`;
|
||||
const scripts = new Map();
|
||||
const protocolTrace = [];
|
||||
let pauseObserved = false;
|
||||
let resolvePaused;
|
||||
const paused = new Promise((resolve) => {
|
||||
resolvePaused = (callFrameId) => {
|
||||
pauseObserved = true;
|
||||
resolve(callFrameId);
|
||||
};
|
||||
});
|
||||
const unsubscribe = caller.onEvent((event) => {
|
||||
if (event.method === "Debugger.scriptParsed") {
|
||||
const params = event.params;
|
||||
if (
|
||||
typeof params === "object" &&
|
||||
params !== null &&
|
||||
"scriptId" in params &&
|
||||
typeof params.scriptId === "string" &&
|
||||
"url" in params &&
|
||||
typeof params.url === "string"
|
||||
)
|
||||
scripts.set(params.scriptId, params.url);
|
||||
if (
|
||||
typeof params === "object" &&
|
||||
params !== null &&
|
||||
"url" in params &&
|
||||
typeof params.url === "string" &&
|
||||
params.url.includes("storage")
|
||||
)
|
||||
protocolTrace.push({ method: event.method, url: params.url });
|
||||
} else if (event.method === "Debugger.paused") {
|
||||
const params = event.params;
|
||||
const callFrames =
|
||||
typeof params === "object" && params !== null && "callFrames" in params
|
||||
? params.callFrames
|
||||
: undefined;
|
||||
const frame = Array.isArray(callFrames) ? callFrames[0] : undefined;
|
||||
const scriptId =
|
||||
typeof frame === "object" && frame !== null && "location" in frame
|
||||
? frame.location?.scriptId
|
||||
: undefined;
|
||||
protocolTrace.push({
|
||||
method: event.method,
|
||||
url: typeof scriptId === "string" ? scripts.get(scriptId) : undefined,
|
||||
reason:
|
||||
typeof params === "object" && params !== null && "reason" in params
|
||||
? params.reason
|
||||
: undefined,
|
||||
});
|
||||
if (
|
||||
typeof frame === "object" &&
|
||||
frame !== null &&
|
||||
typeof frame.callFrameId === "string" &&
|
||||
typeof scriptId === "string" &&
|
||||
scripts.get(scriptId) === debuggerUrl
|
||||
)
|
||||
resolvePaused(frame.callFrameId);
|
||||
}
|
||||
});
|
||||
const debuggerScenario = browserScenarioSchema.parse({
|
||||
...scenario,
|
||||
start_url: { url: debuggerUrl },
|
||||
actions: [
|
||||
{ step_id: "after-debugger", action: "wait_for_timeout", duration_ms: 1 },
|
||||
],
|
||||
});
|
||||
let settled = false;
|
||||
const pending = createBrowserScenarioProvider(process.env)
|
||||
.captureScenario(debuggerScenario, { signal: controller.signal })
|
||||
.finally(() => {
|
||||
settled = true;
|
||||
unsubscribe();
|
||||
});
|
||||
try {
|
||||
const callFrameId = await raceWithTimeout(paused, 10_000, () => {
|
||||
throw new Error(
|
||||
`Caller debugger pause was not observed at ${debuggerUrl}; settled=${String(settled)} events=${JSON.stringify(protocolTrace)}`,
|
||||
);
|
||||
});
|
||||
const pausedState = await caller.send("Debugger.evaluateOnCallFrame", {
|
||||
callFrameId,
|
||||
expression: "window.reaDebuggerAfter === true",
|
||||
returnByValue: true,
|
||||
});
|
||||
assert.equal(
|
||||
pausedState.result.value,
|
||||
false,
|
||||
"Caller-owned debugger frame resumed before the caller inspected it",
|
||||
);
|
||||
await delay(100);
|
||||
assert.equal(
|
||||
settled,
|
||||
false,
|
||||
"Storage observer resumed a caller-owned debugger pause",
|
||||
);
|
||||
await caller.send("Debugger.resume");
|
||||
pauseObserved = false;
|
||||
const result = await pending;
|
||||
if (!result.ok) throw result.error;
|
||||
const debuggerState = await caller.send("Runtime.evaluate", {
|
||||
expression: "window.reaDebuggerAfter === true",
|
||||
returnByValue: true,
|
||||
});
|
||||
assert.equal(debuggerState.result.value, true);
|
||||
} catch (cause) {
|
||||
if (!pauseObserved) controller.abort(cause);
|
||||
else await caller.send("Debugger.resume").catch(() => undefined);
|
||||
const completion = await raceWithTimeout(
|
||||
pending.catch((pendingCause) => ({ ok: false, error: pendingCause })),
|
||||
5_000,
|
||||
);
|
||||
if (completion === undefined)
|
||||
throw new Error("Capture remained pending after cancellation", { cause });
|
||||
const completionError =
|
||||
typeof completion === "object" &&
|
||||
completion !== null &&
|
||||
"error" in completion
|
||||
? completion.error
|
||||
: typeof completion === "object" &&
|
||||
completion !== null &&
|
||||
"ok" in completion &&
|
||||
completion.ok === false
|
||||
? completion.error
|
||||
: undefined;
|
||||
const completionDetail =
|
||||
completionError === undefined
|
||||
? "capture did not report an error"
|
||||
: (() => {
|
||||
const causes = [];
|
||||
let current = completionError;
|
||||
for (
|
||||
let depth = 0;
|
||||
depth < 4 && current !== undefined;
|
||||
depth += 1
|
||||
) {
|
||||
causes.push({
|
||||
name: current.name,
|
||||
tag: current._tag,
|
||||
reason: current.reason,
|
||||
message: current.message,
|
||||
});
|
||||
current = current.cause;
|
||||
}
|
||||
return JSON.stringify(causes);
|
||||
})();
|
||||
throw new Error(`${String(cause)}; capture=${completionDetail}`, {
|
||||
cause,
|
||||
});
|
||||
} finally {
|
||||
unsubscribe();
|
||||
await raceWithTimeout(caller.send("Debugger.disable"), 1_000).catch(
|
||||
() => undefined,
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
const verifyCancelledStorage = async (
|
||||
observer,
|
||||
endpoint,
|
||||
targetId,
|
||||
scenario,
|
||||
) => {
|
||||
const controller = new AbortController();
|
||||
const cancelledScenario = browserScenarioSchema.parse({
|
||||
...scenario,
|
||||
actions: [
|
||||
{
|
||||
step_id: "cancelled-wait",
|
||||
action: "wait_for",
|
||||
locator: { kind: "css", selector: "#never-storage-ready" },
|
||||
state: "visible",
|
||||
timeout_ms: 30_000,
|
||||
},
|
||||
],
|
||||
});
|
||||
const priorDocument = await observer.send("Runtime.evaluate", {
|
||||
expression: "performance.timeOrigin",
|
||||
returnByValue: true,
|
||||
});
|
||||
const pending = createBrowserScenarioProvider(process.env).captureScenario(
|
||||
cancelledScenario,
|
||||
{ signal: controller.signal },
|
||||
);
|
||||
await waitForNewDocument(
|
||||
observer,
|
||||
"/storage-main",
|
||||
priorDocument.result.value,
|
||||
);
|
||||
const state = await observer.send("Runtime.evaluate", {
|
||||
expression:
|
||||
"JSON.parse(document.querySelector('#main-report').textContent)",
|
||||
returnByValue: true,
|
||||
});
|
||||
assert.deepEqual(state.result.value, {
|
||||
local: "main-initial",
|
||||
session: "main-session-initial",
|
||||
removed: "main-remove-seed",
|
||||
empty: "main-empty-seed",
|
||||
});
|
||||
controller.abort(new Error("cancel storage seed scenario"));
|
||||
const result = await pending;
|
||||
assert.ok(
|
||||
!result.ok ||
|
||||
result.value.steps.some(({ status }) => status === "cancelled"),
|
||||
"Cancellation unexpectedly completed every scenario step",
|
||||
);
|
||||
const targets = await (await fetch(`${endpoint}/json/list`)).json();
|
||||
assert.ok(
|
||||
targets.some(({ id }) => id === targetId),
|
||||
"Cancelled connect-mode storage cleanup closed the external target",
|
||||
);
|
||||
};
|
||||
|
||||
const waitForNewDocument = async (connection, path, priorTimeOrigin) => {
|
||||
const deadline = Date.now() + 10_000;
|
||||
while (Date.now() < deadline) {
|
||||
const result = await connection.send("Runtime.evaluate", {
|
||||
expression: `JSON.stringify({ path: location.pathname, time: performance.timeOrigin, state: document.querySelector('#main-report')?.textContent })`,
|
||||
returnByValue: true,
|
||||
});
|
||||
const current = JSON.parse(result.result.value);
|
||||
if (
|
||||
current.path === path &&
|
||||
current.time !== priorTimeOrigin &&
|
||||
current.state !== undefined
|
||||
)
|
||||
return;
|
||||
await new Promise((resolve) => setTimeout(resolve, 25));
|
||||
}
|
||||
throw new Error(`External browser did not navigate to ${path}`);
|
||||
};
|
||||
|
||||
const restoreTarget = async (connection, url) => {
|
||||
await connection.send("Page.navigate", { url });
|
||||
const deadline = Date.now() + 10_000;
|
||||
while (Date.now() < deadline) {
|
||||
const result = await connection.send("Runtime.evaluate", {
|
||||
expression: "location.href",
|
||||
returnByValue: true,
|
||||
});
|
||||
if (result.result.value === url) return;
|
||||
await new Promise((resolve) => setTimeout(resolve, 25));
|
||||
}
|
||||
throw new Error("External browser did not return to its original URL");
|
||||
};
|
||||
@@ -65,7 +65,11 @@ function assertObservationEvents(result) {
|
||||
throw new Error(
|
||||
"Real Chrome attach-window console observation was missing",
|
||||
);
|
||||
if (result.network.websocket_events.length < 1)
|
||||
if (
|
||||
!result.network.websocket_connections.some(
|
||||
(connection) => connection.events.length > 0,
|
||||
)
|
||||
)
|
||||
throw new Error("Real Chrome WebSocket metadata was missing");
|
||||
}
|
||||
|
||||
@@ -77,8 +81,8 @@ function assertObservationPrivacy(result) {
|
||||
result.network.requests.some(
|
||||
(request) => request.body_shapes.status !== "not_approved",
|
||||
) ||
|
||||
result.network.websocket_events.some(
|
||||
(event) => event.payload_shape !== null,
|
||||
result.network.websocket_connections.some((connection) =>
|
||||
connection.events.some((event) => event.payload_shape !== null),
|
||||
)
|
||||
)
|
||||
throw new Error(
|
||||
@@ -246,15 +250,17 @@ export function assertSensitiveShapes(result) {
|
||||
)
|
||||
throw new Error("Real Chrome selected console text was not preserved");
|
||||
if (
|
||||
!result.network.websocket_events.some(
|
||||
(event) =>
|
||||
event.payload_shape?.format === "json" &&
|
||||
event.payload_shape.json_shape?.properties.some(
|
||||
({ path }) =>
|
||||
path.length === 1 &&
|
||||
path[0].kind === "property" &&
|
||||
path[0].name === "token",
|
||||
),
|
||||
!result.network.websocket_connections.some((connection) =>
|
||||
connection.events.some(
|
||||
(event) =>
|
||||
event.payload_shape?.format === "json" &&
|
||||
event.payload_shape.json_shape?.properties.some(
|
||||
({ path }) =>
|
||||
path.length === 1 &&
|
||||
path[0].kind === "property" &&
|
||||
path[0].name === "token",
|
||||
),
|
||||
),
|
||||
)
|
||||
)
|
||||
throw new Error("Real Chrome WebSocket JSON shape was missing");
|
||||
|
||||
@@ -62,6 +62,36 @@ export async function verifyGhidraAddressBoundaries(client, entry, cli) {
|
||||
await query("read_bytes", { address: padded, length: 1 }),
|
||||
await query("read_bytes", { address: entry, length: 1 }),
|
||||
);
|
||||
const aliasKeys = [entry, padded];
|
||||
assert.deepEqual(
|
||||
await query("set_addresses_names", {
|
||||
names: {
|
||||
[entry]: "rea_same_address_alias",
|
||||
[padded]: "rea_same_address_alias",
|
||||
},
|
||||
}),
|
||||
Object.fromEntries(aliasKeys.map((address) => [address, true])),
|
||||
"Equivalent address spellings with one requested name share one edit",
|
||||
);
|
||||
const nameBeforeAliasedBatch = await query("address_name", {
|
||||
address: entry,
|
||||
});
|
||||
assert.equal(nameBeforeAliasedBatch, "rea_same_address_alias");
|
||||
assert.deepEqual(
|
||||
await query("set_addresses_names", {
|
||||
names: {
|
||||
[entry]: "rea_first_address_alias",
|
||||
[padded]: "rea_second_address_alias",
|
||||
},
|
||||
}),
|
||||
Object.fromEntries(aliasKeys.map((address) => [address, false])),
|
||||
"Conflicting spellings of one Ghidra address must be rejected together",
|
||||
);
|
||||
assert.equal(
|
||||
await query("address_name", { address: entry }),
|
||||
nameBeforeAliasedBatch,
|
||||
"A rejected alias collision must leave the preexisting target name unchanged",
|
||||
);
|
||||
for (const [command, flags, operation] of [
|
||||
["read-bytes", ["--length", "1"], "read_bytes"],
|
||||
[
|
||||
|
||||
@@ -58,6 +58,15 @@ export async function verifyGhidraBoundaries(
|
||||
`${name}: ${JSON.stringify(validate.errors)}`,
|
||||
);
|
||||
assert.deepEqual(reply.structuredContent, JSON.parse(mcpTextValue(reply)));
|
||||
if (name === "list_strings" || name === "search_strings")
|
||||
assert.ok(
|
||||
reply.structuredContent.limitations.some((limitation) =>
|
||||
/Only Ghidra-defined string Data.*empty search does not establish/u.test(
|
||||
limitation,
|
||||
),
|
||||
),
|
||||
`${name} omitted defined-string coverage`,
|
||||
);
|
||||
successfulCalls++;
|
||||
return result;
|
||||
};
|
||||
@@ -378,6 +387,12 @@ export async function verifyGhidraBoundaries(
|
||||
).some((item) => item.address === marker.address),
|
||||
false,
|
||||
);
|
||||
assert.deepEqual(
|
||||
await call("search_strings", {
|
||||
pattern: `REA_GHIDRA_ABSENT_${randomUUID()}`,
|
||||
}),
|
||||
[],
|
||||
);
|
||||
assert.equal(
|
||||
(await call("inspect_native_instruction", { address: strings[0].address }))
|
||||
.status,
|
||||
|
||||
@@ -183,7 +183,9 @@ export async function verifyGhidraLargeResults({
|
||||
assert.equal(dossier.procedure.name, "rea_frame_probe");
|
||||
assert.ok(dossier.comments.some(({ text }) => text === comment));
|
||||
assert.ok(
|
||||
dossierRecord.raw_result.comments.some(({ text }) => text === comment),
|
||||
dossierRecord.raw_result.value.comments.some(
|
||||
({ text }) => text === comment,
|
||||
),
|
||||
);
|
||||
const { stdout } = await exec(
|
||||
process.execPath,
|
||||
|
||||
@@ -62,7 +62,7 @@ export async function verifyGhidraNamespaceAnnotations({
|
||||
assert.ok(symbol, `Missing demangled fixture global ${namespace}::value`);
|
||||
return { namespace, address: symbol.address };
|
||||
});
|
||||
await verifyAddressNameCollisions(call, procedures, names);
|
||||
await verifyDuplicateAddressNames(call, procedures, names);
|
||||
const ambiguous = await reject("procedure_address", { procedure: "same" });
|
||||
assert.equal(ambiguous.code, "invalid_request");
|
||||
for (const { address } of entries)
|
||||
@@ -134,11 +134,8 @@ export async function verifyGhidraNamespaceAnnotations({
|
||||
});
|
||||
assert.deepEqual(unchanged.annotations, qualified.annotations);
|
||||
}
|
||||
await verifyAddressNames(call, [
|
||||
{ ...entries[0], kind: "function" },
|
||||
{ ...entries[2], kind: "function" },
|
||||
...globals.map((symbol) => ({ ...symbol, kind: "label" })),
|
||||
]);
|
||||
await verifyAddressNames(call, [entries[0], entries[2]], "function");
|
||||
await verifyAddressNames(call, globals, "label");
|
||||
for (const { namespace, address } of entries) {
|
||||
assert.equal(
|
||||
await call("set_address_name", {
|
||||
@@ -182,133 +179,85 @@ export async function verifyGhidraNamespaceAnnotations({
|
||||
}
|
||||
}
|
||||
|
||||
async function verifyAddressNameCollisions(call, procedures, names) {
|
||||
const find = (items, name) => {
|
||||
const symbol = items.find(
|
||||
(item) => item.value === name || item.value === `_${name}`,
|
||||
async function verifyDuplicateAddressNames(call, procedures, names) {
|
||||
const functionEntries = [
|
||||
"rea_duplicate_function_a",
|
||||
"rea_duplicate_function_b",
|
||||
].map((name) => {
|
||||
const procedure = procedures.find(
|
||||
({ value }) => value === name || value === `_${name}`,
|
||||
);
|
||||
assert.ok(symbol, `Missing naming collision fixture ${name}`);
|
||||
return symbol;
|
||||
};
|
||||
const first = find(procedures, "rea_name_first");
|
||||
const second = find(procedures, "rea_name_second");
|
||||
const firstLabel = find(names, "rea_name_data_first");
|
||||
const secondLabel = find(names, "rea_name_data_second");
|
||||
for (const [owner, other, name] of [
|
||||
[first, second, "owned_function"],
|
||||
[firstLabel, secondLabel, "owned_label"],
|
||||
]) {
|
||||
assert.equal(
|
||||
await call("set_address_name", { address: owner.address, name }),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
await call("set_address_name", { address: other.address, name }),
|
||||
false,
|
||||
"A different address must not acquire the existing name",
|
||||
);
|
||||
assert.equal(
|
||||
await call("address_name", { address: other.address }),
|
||||
other.value,
|
||||
);
|
||||
assert.equal(
|
||||
await call("set_address_name", { address: owner.address, name }),
|
||||
true,
|
||||
"The owning address may repeat the same name",
|
||||
);
|
||||
}
|
||||
for (const [address, name, original] of [
|
||||
[first.address, "owned_label", "owned_function"],
|
||||
[firstLabel.address, "owned_function", "owned_label"],
|
||||
]) {
|
||||
assert.equal(await call("set_address_name", { address, name }), false);
|
||||
assert.equal(await call("address_name", { address }), original);
|
||||
}
|
||||
const alphaFunction = find(procedures, "alpha::same");
|
||||
const alphaLabel = find(names, "alpha::value");
|
||||
for (const name of ["value", "alpha::value"]) {
|
||||
assert.equal(
|
||||
await call("set_address_name", { address: alphaFunction.address, name }),
|
||||
false,
|
||||
);
|
||||
assert.equal(
|
||||
await call("address_name", { address: alphaFunction.address }),
|
||||
alphaFunction.value,
|
||||
);
|
||||
}
|
||||
assert.equal(
|
||||
await call("set_address_name", {
|
||||
address: alphaLabel.address,
|
||||
name: "same",
|
||||
}),
|
||||
false,
|
||||
);
|
||||
assert.equal(
|
||||
await call("address_name", { address: alphaLabel.address }),
|
||||
alphaLabel.value,
|
||||
);
|
||||
assert.deepEqual(
|
||||
await call("set_addresses_names", {
|
||||
names: {
|
||||
[secondLabel.address]: "batch_available",
|
||||
[second.address]: "owned_function",
|
||||
[firstLabel.address]: "owned_label",
|
||||
},
|
||||
}),
|
||||
{
|
||||
[secondLabel.address]: true,
|
||||
[second.address]: false,
|
||||
[firstLabel.address]: true,
|
||||
assert.ok(procedure, `Missing duplicate-name fixture function ${name}`);
|
||||
return { address: procedure.address, original: procedure.value };
|
||||
});
|
||||
const labelEntries = ["rea_duplicate_label_a", "rea_duplicate_label_b"].map(
|
||||
(name) => {
|
||||
const symbol = names.find(
|
||||
(item) =>
|
||||
(item.value === name || item.value === `_${name}`) &&
|
||||
item.symbol.primary,
|
||||
);
|
||||
assert.ok(symbol, `Missing duplicate-name fixture label ${name}`);
|
||||
return { address: symbol.address, original: symbol.value };
|
||||
},
|
||||
);
|
||||
assert.equal(
|
||||
await call("address_name", { address: second.address }),
|
||||
second.value,
|
||||
);
|
||||
assert.equal(
|
||||
await call("address_name", { address: first.address }),
|
||||
"owned_function",
|
||||
);
|
||||
assert.equal(
|
||||
await call("address_name", { address: secondLabel.address }),
|
||||
"batch_available",
|
||||
);
|
||||
const before = await call("annotate_native_function", {
|
||||
procedure: second.address,
|
||||
comment: "Keep this comment",
|
||||
inline_comment: "Keep this inline comment",
|
||||
});
|
||||
assert.equal(
|
||||
await call("set_address_name", {
|
||||
address: second.address,
|
||||
name: "owned_function",
|
||||
}),
|
||||
false,
|
||||
);
|
||||
const unchanged = await call("annotate_native_function", {
|
||||
procedure: second.address,
|
||||
name: second.value,
|
||||
});
|
||||
assert.deepEqual(unchanged.annotations, before.annotations);
|
||||
assert.equal(
|
||||
await call("set_address_name", {
|
||||
address: first.address,
|
||||
name: "released_function",
|
||||
}),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
await call("set_address_name", {
|
||||
address: second.address,
|
||||
name: "owned_function",
|
||||
}),
|
||||
true,
|
||||
"A name becomes available after its old owner releases it",
|
||||
);
|
||||
|
||||
for (const [kind, entries] of [
|
||||
["function", functionEntries],
|
||||
["label", labelEntries],
|
||||
]) {
|
||||
const [owner, duplicate] = entries;
|
||||
const name = `rea_duplicate_${kind}`;
|
||||
assert.equal(
|
||||
await call("set_address_name", { address: owner.address, name }),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
await call("set_address_name", { address: owner.address, name }),
|
||||
true,
|
||||
"Renaming the current owner to its existing name remains idempotent",
|
||||
);
|
||||
assert.equal(
|
||||
await call("set_address_name", { address: duplicate.address, name }),
|
||||
false,
|
||||
);
|
||||
assert.equal(
|
||||
await call("address_name", { address: duplicate.address }),
|
||||
duplicate.original,
|
||||
"A rejected duplicate leaves the other address unchanged",
|
||||
);
|
||||
|
||||
const batchName = `rea_batch_duplicate_${kind}`;
|
||||
assert.deepEqual(
|
||||
await call("set_addresses_names", {
|
||||
names: {
|
||||
[owner.address]: batchName,
|
||||
[duplicate.address]: batchName,
|
||||
},
|
||||
}),
|
||||
{ [owner.address]: true, [duplicate.address]: false },
|
||||
);
|
||||
assert.equal(
|
||||
await call("address_name", { address: owner.address }),
|
||||
batchName,
|
||||
);
|
||||
assert.equal(
|
||||
await call("address_name", { address: duplicate.address }),
|
||||
duplicate.original,
|
||||
"A failed batch member rolls back only its own address",
|
||||
);
|
||||
const owners = (await call("list_names")).filter(
|
||||
(item) => item.value === batchName && item.symbol.primary,
|
||||
);
|
||||
assert.deepEqual(
|
||||
owners.map(({ address }) => address),
|
||||
[owner.address],
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function verifyAddressNames(call, symbols) {
|
||||
for (const { namespace, address, kind } of symbols) {
|
||||
async function verifyAddressNames(call, symbols, kind) {
|
||||
for (const { namespace, address } of symbols) {
|
||||
assert.equal(
|
||||
await call("set_address_name", {
|
||||
address,
|
||||
@@ -323,11 +272,14 @@ async function verifyAddressNames(call, symbols) {
|
||||
);
|
||||
assert.equal(await call("address_name", { address }), qualified);
|
||||
assert.equal(
|
||||
await call("set_address_name", { address, name: "other::literal" }),
|
||||
await call("set_address_name", {
|
||||
address,
|
||||
name: `other::literal_${kind}`,
|
||||
}),
|
||||
true,
|
||||
);
|
||||
const literal = await call("address_name", { address });
|
||||
assert.equal(literal, `${namespace}::other::literal`);
|
||||
assert.equal(literal, `${namespace}::other::literal_${kind}`);
|
||||
assert.equal(
|
||||
await call("set_address_name", { address, name: literal }),
|
||||
true,
|
||||
@@ -368,7 +320,7 @@ async function verifyAddressNames(call, symbols) {
|
||||
assert.deepEqual(
|
||||
await call("set_addresses_names", {
|
||||
names: {
|
||||
[valid.address]: `${valid.namespace}::batch_renamed`,
|
||||
[valid.address]: `${valid.namespace}::batch_renamed_${kind}`,
|
||||
[invalid.address]: `${invalid.namespace}::`,
|
||||
},
|
||||
}),
|
||||
@@ -376,17 +328,17 @@ async function verifyAddressNames(call, symbols) {
|
||||
);
|
||||
assert.equal(
|
||||
await call("address_name", { address: valid.address }),
|
||||
`${valid.namespace}::batch_renamed`,
|
||||
`${valid.namespace}::batch_renamed_${kind}`,
|
||||
);
|
||||
assert.equal(
|
||||
await call("address_name", { address: invalid.address }),
|
||||
`${invalid.namespace}::address_qualified_${invalid.kind}`,
|
||||
`${invalid.namespace}::address_qualified_${kind}`,
|
||||
);
|
||||
const names = await call("list_names");
|
||||
for (const { namespace, address, kind } of symbols) {
|
||||
for (const { namespace, address } of symbols) {
|
||||
const expected =
|
||||
address === valid.address
|
||||
? `${namespace}::batch_renamed`
|
||||
? `${namespace}::batch_renamed_${kind}`
|
||||
: `${namespace}::address_qualified_${kind}`;
|
||||
assert.ok(
|
||||
names.some(
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
import { z } from "zod";
|
||||
import {
|
||||
releaseAuthorSchema as author,
|
||||
releaseIssueSchema as issue,
|
||||
releasePullRequestSchema,
|
||||
} from "./release-notes.mjs";
|
||||
|
||||
const pageInfo = z.object({
|
||||
hasNextPage: z.boolean(),
|
||||
endCursor: z.string().nullable(),
|
||||
});
|
||||
const connection = (node) => z.object({ nodes: z.array(node), pageInfo });
|
||||
const associated = z.object({
|
||||
number: z.number().int().positive(),
|
||||
mergeCommit: z.object({ oid: z.string() }).nullable(),
|
||||
});
|
||||
const pr = releasePullRequestSchema.omit({ issues: true }).extend({
|
||||
closingIssuesReferences: connection(issue),
|
||||
});
|
||||
const PAGE = "pageInfo { hasNextPage endCursor }";
|
||||
const ACTOR = "login __typename";
|
||||
const AUTHORS = `authors(first:100, after:$cursor) { nodes { name email user { ${ACTOR} } } ${PAGE} }`;
|
||||
const ASSOCIATED = `associatedPullRequests(first:100, after:$cursor) { nodes { number mergeCommit { oid } } ${PAGE} }`;
|
||||
const ISSUES = `closingIssuesReferences(first:100, after:$cursor) { nodes { number url author { ${ACTOR} } repository { nameWithOwner } } ${PAGE} }`;
|
||||
|
||||
async function query(run, repository, selection, cursor = null) {
|
||||
const [owner, name] = repository.split("/");
|
||||
const response = JSON.parse(
|
||||
await run(
|
||||
"gh",
|
||||
["api", "graphql", "--input", "-"],
|
||||
JSON.stringify({
|
||||
query: `query($owner:String!,$name:String!,$cursor:String) { repository(owner:$owner,name:$name) { ${selection} } }`,
|
||||
variables: { owner, name, cursor },
|
||||
}),
|
||||
),
|
||||
);
|
||||
if (response.errors?.length)
|
||||
throw new Error(
|
||||
`GitHub inventory query failed: ${JSON.stringify(response.errors)}`,
|
||||
);
|
||||
if (!response.data?.repository)
|
||||
throw new Error(`GitHub repository unavailable: ${repository}`);
|
||||
return response.data.repository;
|
||||
}
|
||||
|
||||
async function remainingPages(run, repository, selection, field) {
|
||||
const initial = field.initial;
|
||||
const nodes = [...initial.nodes];
|
||||
let page = initial;
|
||||
const cursors = new Set();
|
||||
while (page.pageInfo.hasNextPage) {
|
||||
const cursor = page.pageInfo.endCursor;
|
||||
if (!cursor || cursors.has(cursor))
|
||||
throw new Error(`GitHub returned a non-advancing ${field.name} cursor`);
|
||||
cursors.add(cursor);
|
||||
const response = await query(run, repository, selection, cursor);
|
||||
page = field.schema.parse(response.item?.[field.name]);
|
||||
nodes.push(...page.nodes);
|
||||
}
|
||||
return nodes;
|
||||
}
|
||||
|
||||
/** GitHub's commit authors include co-authors; unresolved identities remain explicit. */
|
||||
export async function collectGitHubCredits(run, repository, commits) {
|
||||
const shipped = new Set(commits.map((commit) => commit.sha));
|
||||
const numbers = new Set();
|
||||
const result = [];
|
||||
// Small query batches limit transient GraphQL work, never the history examined.
|
||||
for (let offset = 0; offset < commits.length; offset += 20) {
|
||||
const batch = commits.slice(offset, offset + 20);
|
||||
const selection = batch
|
||||
.map(
|
||||
(commit, index) =>
|
||||
`c${index}: object(oid:${JSON.stringify(commit.sha)}) { ... on Commit { oid ${AUTHORS} ${ASSOCIATED} } }`,
|
||||
)
|
||||
.join("\n");
|
||||
const response = await query(run, repository, selection);
|
||||
for (const [index, commit] of batch.entries()) {
|
||||
const value = z
|
||||
.object({
|
||||
oid: z.literal(commit.sha),
|
||||
authors: connection(author),
|
||||
associatedPullRequests: connection(associated),
|
||||
})
|
||||
.parse(response[`c${index}`]);
|
||||
const select = (fields) =>
|
||||
`item: object(oid:${JSON.stringify(commit.sha)}) { ... on Commit { ${fields} } }`;
|
||||
const authors = await remainingPages(run, repository, select(AUTHORS), {
|
||||
name: "authors",
|
||||
schema: connection(author),
|
||||
initial: value.authors,
|
||||
});
|
||||
const associations = await remainingPages(
|
||||
run,
|
||||
repository,
|
||||
select(ASSOCIATED),
|
||||
{
|
||||
name: "associatedPullRequests",
|
||||
schema: connection(associated),
|
||||
initial: value.associatedPullRequests,
|
||||
},
|
||||
);
|
||||
const pullRequests = [
|
||||
...new Set(
|
||||
associations
|
||||
.filter((item) => shipped.has(item.mergeCommit?.oid))
|
||||
.map((item) => item.number),
|
||||
),
|
||||
];
|
||||
for (const number of pullRequests) numbers.add(number);
|
||||
result.push({ ...commit, authors, pullRequests });
|
||||
}
|
||||
}
|
||||
const pullRequests = [];
|
||||
const sorted = [...numbers].sort((a, b) => a - b);
|
||||
for (let offset = 0; offset < sorted.length; offset += 20) {
|
||||
const batch = sorted.slice(offset, offset + 20);
|
||||
const fields = `number title url author { ${ACTOR} } mergeCommit { oid } ${ISSUES}`;
|
||||
const response = await query(
|
||||
run,
|
||||
repository,
|
||||
batch
|
||||
.map(
|
||||
(number, index) =>
|
||||
`p${index}: pullRequest(number:${number}) { ${fields} }`,
|
||||
)
|
||||
.join("\n"),
|
||||
);
|
||||
for (const [index, number] of batch.entries()) {
|
||||
const value = pr.parse(response[`p${index}`]);
|
||||
if (value.number !== number || !shipped.has(value.mergeCommit.oid))
|
||||
throw new Error(`PR #${number} changed outside the selected history`);
|
||||
const issues = await remainingPages(
|
||||
run,
|
||||
repository,
|
||||
`item: pullRequest(number:${number}) { ${ISSUES} }`,
|
||||
{
|
||||
name: "closingIssuesReferences",
|
||||
schema: connection(issue),
|
||||
initial: value.closingIssuesReferences,
|
||||
},
|
||||
);
|
||||
const { closingIssuesReferences: _connection, ...metadata } = value;
|
||||
pullRequests.push({ ...metadata, issues });
|
||||
}
|
||||
}
|
||||
return { commits: result, pullRequests };
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
import { valid } from "semver";
|
||||
import { z } from "zod";
|
||||
import {
|
||||
releaseInventorySchema,
|
||||
verifyInventoryHistory,
|
||||
verifyReleaseNotes,
|
||||
} from "./release-notes.mjs";
|
||||
|
||||
const sha = z.string().regex(/^[a-f0-9]{40}$/u);
|
||||
const repository = z.string().regex(/^[\w.-]+\/[\w.-]+$/u);
|
||||
const pullRequest = z.object({
|
||||
number: z.number().int().positive(),
|
||||
body: z.string(),
|
||||
merged_at: z.string().nullable(),
|
||||
merge_commit_sha: sha.nullable(),
|
||||
head: z.object({
|
||||
ref: z.string(),
|
||||
repo: z.object({ full_name: z.string() }).nullable(),
|
||||
}),
|
||||
labels: z.array(z.object({ name: z.string() })),
|
||||
});
|
||||
|
||||
/** Read the reviewed Git snapshot and live PR before Release Please can mutate tags. */
|
||||
export async function verifyNotesPublication(run, options) {
|
||||
const repo = repository.parse(options.repository);
|
||||
const source = sha.parse(
|
||||
(
|
||||
await run("git", [
|
||||
"rev-parse",
|
||||
"--verify",
|
||||
"--end-of-options",
|
||||
`${options.source}^{commit}`,
|
||||
])
|
||||
).trim(),
|
||||
);
|
||||
const read = (path) => run("git", ["show", `${source}:${path}`]);
|
||||
const { version } = z
|
||||
.object({ version: z.string() })
|
||||
.parse(JSON.parse(await read("package.json")));
|
||||
if (valid(version) !== version || version.includes("+"))
|
||||
throw new Error("Invalid reviewed package version");
|
||||
let candidates;
|
||||
if (options.pr !== undefined) {
|
||||
const number = z.coerce.number().int().positive().parse(options.pr);
|
||||
candidates = [
|
||||
pullRequest.parse(
|
||||
JSON.parse(await run("gh", ["api", `repos/${repo}/pulls/${number}`])),
|
||||
),
|
||||
];
|
||||
} else {
|
||||
candidates = z
|
||||
.array(z.array(pullRequest))
|
||||
.parse(
|
||||
JSON.parse(
|
||||
await run("gh", [
|
||||
"api",
|
||||
"--paginate",
|
||||
"--slurp",
|
||||
`repos/${repo}/commits/${source}/pulls?per_page=100`,
|
||||
]),
|
||||
),
|
||||
)
|
||||
.flat();
|
||||
}
|
||||
const matching = candidates.filter(
|
||||
(pr) =>
|
||||
pr.merged_at !== null &&
|
||||
pr.merge_commit_sha === source &&
|
||||
pr.head.repo?.full_name.toLowerCase() === repo.toLowerCase() &&
|
||||
pr.head.ref.startsWith("release-please--branches--"),
|
||||
);
|
||||
if (matching.length !== 1)
|
||||
throw new Error(
|
||||
"Expected exactly one merged Release Please PR for the reviewed source",
|
||||
);
|
||||
const pr = matching[0];
|
||||
const path = `docs/releases/${version}.contributions.json`;
|
||||
const present =
|
||||
(
|
||||
await run("git", ["ls-tree", "--name-only", source, "--", path])
|
||||
).trim() === path;
|
||||
if (!present) {
|
||||
if (pr.labels.some((label) => label.name === "rea:release-notes-finalized"))
|
||||
throw new Error(
|
||||
"Finalized release PR is missing its contribution inventory",
|
||||
);
|
||||
return { status: "legacy", version, pullRequest: pr.number };
|
||||
}
|
||||
const inventory = releaseInventorySchema.parse(JSON.parse(await read(path)));
|
||||
if (
|
||||
inventory.version !== version ||
|
||||
inventory.repository.toLowerCase() !== repo.toLowerCase()
|
||||
)
|
||||
throw new Error(
|
||||
"Contribution inventory differs from the reviewed release version or repository",
|
||||
);
|
||||
const baseline = z
|
||||
.object({ ".": z.string() })
|
||||
.parse(
|
||||
JSON.parse(
|
||||
await run("git", ["show", `${source}^1:.release-please-manifest.json`]),
|
||||
),
|
||||
)["."];
|
||||
if (valid(baseline) !== baseline || baseline.includes("+"))
|
||||
throw new Error("Invalid release baseline version");
|
||||
const baselineSha = (
|
||||
await run("git", ["rev-parse", `refs/tags/rea-agents-${baseline}^{commit}`])
|
||||
).trim();
|
||||
if (inventory.base !== baselineSha)
|
||||
throw new Error(
|
||||
"Contribution inventory does not start at the published baseline tag",
|
||||
);
|
||||
await verifyInventoryHistory(run, inventory, source);
|
||||
return {
|
||||
status: "verified",
|
||||
...verifyReleaseNotes(inventory, await read("CHANGELOG.md"), pr.body),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,442 @@
|
||||
import { valid } from "semver";
|
||||
import { z } from "zod";
|
||||
|
||||
const sha = z.string().regex(/^[a-f0-9]{40}$/u);
|
||||
const actor = z.object({
|
||||
login: z.string().regex(/^[A-Za-z0-9][A-Za-z0-9-]*(?:\[bot\])?$/u),
|
||||
__typename: z.string(),
|
||||
});
|
||||
const author = z.object({
|
||||
name: z.string(),
|
||||
email: z.string(),
|
||||
user: actor.nullable(),
|
||||
});
|
||||
const commit = z.object({
|
||||
sha,
|
||||
subject: z.string(),
|
||||
body: z.string(),
|
||||
parents: z.array(sha),
|
||||
authors: z.array(author),
|
||||
pullRequests: z.array(z.number().int().positive()),
|
||||
});
|
||||
const issue = z.object({
|
||||
number: z.number().int().positive(),
|
||||
url: z.url(),
|
||||
author: actor.nullable(),
|
||||
repository: z.object({ nameWithOwner: z.string() }),
|
||||
});
|
||||
const pullRequest = z.object({
|
||||
number: z.number().int().positive(),
|
||||
title: z.string(),
|
||||
url: z.url(),
|
||||
author: actor.nullable(),
|
||||
mergeCommit: z.object({ oid: sha }),
|
||||
issues: z.array(issue),
|
||||
});
|
||||
export {
|
||||
author as releaseAuthorSchema,
|
||||
issue as releaseIssueSchema,
|
||||
pullRequest as releasePullRequestSchema,
|
||||
};
|
||||
export const releaseInventorySchema = z.object({
|
||||
schemaVersion: z.literal(1),
|
||||
repository: z.string().regex(/^[\w.-]+\/[\w.-]+$/u),
|
||||
version: z
|
||||
.string()
|
||||
.refine(
|
||||
(value) => valid(value) === value && !value.includes("+"),
|
||||
"Expected exact release SemVer",
|
||||
),
|
||||
base: sha,
|
||||
target: sha,
|
||||
commits: z.array(commit),
|
||||
pullRequests: z.array(pullRequest),
|
||||
});
|
||||
|
||||
export async function readReleaseHistory(run, base, target) {
|
||||
if (
|
||||
(await run("git", ["rev-parse", "--is-shallow-repository"])).trim() !==
|
||||
"false"
|
||||
)
|
||||
throw new Error(
|
||||
"Release inventory requires complete Git history; fetch --unshallow first",
|
||||
);
|
||||
const resolve = async (ref) =>
|
||||
sha.parse(
|
||||
(
|
||||
await run("git", [
|
||||
"rev-parse",
|
||||
"--verify",
|
||||
"--end-of-options",
|
||||
`${ref}^{commit}`,
|
||||
])
|
||||
).trim(),
|
||||
);
|
||||
const baseSha = await resolve(base);
|
||||
const targetSha = await resolve(target);
|
||||
await run("git", ["merge-base", "--is-ancestor", baseSha, targetSha]);
|
||||
const text = await run("git", [
|
||||
"log",
|
||||
"--reverse",
|
||||
"--topo-order",
|
||||
"-z",
|
||||
"--format=%H%x00%s%x00%B%x00%P",
|
||||
`${baseSha}..${targetSha}`,
|
||||
]);
|
||||
const fields = text.split("\0");
|
||||
if (fields.pop() !== "" || fields.length % 4)
|
||||
throw new Error("Malformed Git release history");
|
||||
const commits = [];
|
||||
for (let index = 0; index < fields.length; index += 4) {
|
||||
commits.push({
|
||||
sha: sha.parse(fields[index]),
|
||||
subject: fields[index + 1],
|
||||
body: fields[index + 2],
|
||||
parents: fields[index + 3].split(" ").filter(Boolean),
|
||||
});
|
||||
}
|
||||
return { base: baseSha, target: targetSha, commits };
|
||||
}
|
||||
|
||||
// REA history maps Anthropic and Cursor's agent trailers to ordinary User actors.
|
||||
// Keep those source identities in the inventory while excluding automation credit.
|
||||
const AGENT_ACCOUNTS = new Set(["claude", "cursoragent"]);
|
||||
// REA's maintainer requested external-contributor thanks without self-credit.
|
||||
const MAINTAINER_ACCOUNTS = new Set(["morluto"]);
|
||||
const human = (value) =>
|
||||
value?.__typename === "User" &&
|
||||
!value.login.endsWith("[bot]") &&
|
||||
!AGENT_ACCOUNTS.has(value.login.toLowerCase()) &&
|
||||
!MAINTAINER_ACCOUNTS.has(value.login.toLowerCase())
|
||||
? value.login
|
||||
: undefined;
|
||||
const unique = (values) =>
|
||||
[...new Set(values.filter(Boolean))].sort((a, b) =>
|
||||
a.toLowerCase().localeCompare(b.toLowerCase()),
|
||||
);
|
||||
|
||||
function prCredits(inventory, pr) {
|
||||
return unique([
|
||||
human(pr.author),
|
||||
...pr.issues.map((item) => human(item.author)),
|
||||
...inventory.commits
|
||||
.filter(
|
||||
(item) =>
|
||||
item.parents.length === 1 && item.pullRequests.includes(pr.number),
|
||||
)
|
||||
.flatMap((item) => item.authors.map((identity) => human(identity.user))),
|
||||
]);
|
||||
}
|
||||
|
||||
function thanks(handles) {
|
||||
return handles.length
|
||||
? ` Thanks ${handles.map((handle) => `@${handle}`).join(", ")}.`
|
||||
: "";
|
||||
}
|
||||
|
||||
export function contributionRecord(raw) {
|
||||
const inventory = releaseInventorySchema.parse(raw);
|
||||
const lines = [
|
||||
"### Complete contribution record",
|
||||
"",
|
||||
`History: \`${inventory.base}..${inventory.target}\`; ${inventory.pullRequests.length} merged PRs, ${inventory.commits.length} commits examined.`,
|
||||
"",
|
||||
];
|
||||
for (const pr of inventory.pullRequests) {
|
||||
const issues = pr.issues
|
||||
.map(
|
||||
(item) =>
|
||||
`[${item.repository.nameWithOwner}#${item.number}](${item.url})`,
|
||||
)
|
||||
.join(", ");
|
||||
lines.push(
|
||||
`- **PR [#${pr.number}](${pr.url})**${issues ? ` Related ${issues}.` : ""}${thanks(prCredits(inventory, pr))}`,
|
||||
);
|
||||
}
|
||||
const direct = inventory.commits.filter(
|
||||
(item) => item.pullRequests.length === 0 && item.parents.length <= 1,
|
||||
);
|
||||
if (direct.length) {
|
||||
lines.push("", "#### Direct contributions", "");
|
||||
for (const item of direct)
|
||||
lines.push(
|
||||
`- [${item.sha.slice(0, 7)}](https://github.com/${inventory.repository}/commit/${item.sha})${thanks(unique(item.authors.map((identity) => human(identity.user))))}`,
|
||||
);
|
||||
}
|
||||
return `${lines.join("\n")}\n`;
|
||||
}
|
||||
|
||||
/** Keep the full authorship snapshot available without duplicating public thanks. */
|
||||
export function contributionSummary(raw) {
|
||||
const inventory = releaseInventorySchema.parse(raw);
|
||||
return `Full contribution record: [${inventory.commits.length} commits and ${inventory.pullRequests.length} merged PR${inventory.pullRequests.length === 1 ? "" : "s"}](https://github.com/${inventory.repository}/blob/rea-agents-${inventory.version}/docs/releases/${inventory.version}.contributions.json).\n`;
|
||||
}
|
||||
|
||||
export function selectedReleaseNotes(text, version) {
|
||||
const headings = [...text.matchAll(/^## \[([^\]]+)\].*$/gmu)];
|
||||
if (headings[0]?.[1] !== version)
|
||||
throw new Error(`Notes must start with release [${version}]`);
|
||||
return text
|
||||
.slice(headings[0].index, headings[1]?.index ?? text.length)
|
||||
.trim();
|
||||
}
|
||||
|
||||
// Notes have one Markdown list item per line; fenced examples are never credit sources.
|
||||
function proseBullets(text) {
|
||||
let fence;
|
||||
return text.split("\n").filter((line) => {
|
||||
const marker = /^\s*(`{3,}|~{3,})/u.exec(line)?.[1];
|
||||
if (marker) {
|
||||
if (!fence) fence = marker;
|
||||
else if (marker[0] === fence[0] && marker.length >= fence.length)
|
||||
fence = undefined;
|
||||
return false;
|
||||
}
|
||||
return !fence && /^[-*] /u.test(line);
|
||||
});
|
||||
}
|
||||
|
||||
function bulletPrs(bullet, repository) {
|
||||
// Resolve explicit links first so cross-repository numbers cannot acquire local credit.
|
||||
const local = [];
|
||||
const withoutLinks = bullet.replace(
|
||||
/\[([^\]]*)\]\(https:\/\/github\.com\/([^/]+\/[^/]+)\/(?:pull|issues)\/(\d+)\)/gu,
|
||||
(_match, _label, repo, number) => {
|
||||
if (repo.toLowerCase() === repository.toLowerCase())
|
||||
local.push(Number(number));
|
||||
return "";
|
||||
},
|
||||
);
|
||||
for (const match of withoutLinks.matchAll(/(?<![\w/])#(\d+)\b/gu))
|
||||
local.push(Number(match[1]));
|
||||
return new Set(local);
|
||||
}
|
||||
|
||||
function requiredCredits(inventory, bullet, references) {
|
||||
const required = [];
|
||||
for (const pr of inventory.pullRequests.filter((item) =>
|
||||
references.has(item.number),
|
||||
)) {
|
||||
for (const handle of prCredits(inventory, pr))
|
||||
required.push({ handle, context: `PR #${pr.number}` });
|
||||
}
|
||||
for (const issue of inventory.pullRequests.flatMap((item) => item.issues)) {
|
||||
const mentioned =
|
||||
(issue.repository.nameWithOwner.toLowerCase() ===
|
||||
inventory.repository.toLowerCase() &&
|
||||
references.has(issue.number)) ||
|
||||
bullet.includes(`](${issue.url})`);
|
||||
const handle = human(issue.author);
|
||||
if (mentioned && handle)
|
||||
required.push({
|
||||
handle,
|
||||
context: `issue ${issue.repository.nameWithOwner}#${issue.number}`,
|
||||
});
|
||||
}
|
||||
for (const match of bullet.matchAll(
|
||||
/https:\/\/github\.com\/([^/]+\/[^/]+)\/commit\/([a-f0-9]{7,40})\b/gu,
|
||||
)) {
|
||||
if (match[1].toLowerCase() !== inventory.repository.toLowerCase()) continue;
|
||||
const matches = inventory.commits.filter((item) =>
|
||||
item.sha.startsWith(match[2]),
|
||||
);
|
||||
if (matches.length !== 1)
|
||||
throw new Error(`Unknown or ambiguous release commit ${match[2]}`);
|
||||
if (matches[0].parents.length > 1) continue;
|
||||
for (const identity of matches[0].authors) {
|
||||
const handle = human(identity.user);
|
||||
if (handle)
|
||||
required.push({ handle, context: `commit ${matches[0].sha}` });
|
||||
}
|
||||
}
|
||||
return required;
|
||||
}
|
||||
|
||||
function verifyEditorialCredits(inventory, prose) {
|
||||
const known = new Set(inventory.pullRequests.map((pr) => pr.number));
|
||||
const issues = new Set(
|
||||
inventory.pullRequests.flatMap((pr) =>
|
||||
pr.issues
|
||||
.filter(
|
||||
(item) =>
|
||||
item.repository.nameWithOwner.toLowerCase() ===
|
||||
inventory.repository.toLowerCase(),
|
||||
)
|
||||
.map((item) => item.number),
|
||||
),
|
||||
);
|
||||
const eligible = new Set(
|
||||
unique([
|
||||
...inventory.pullRequests.flatMap((pr) => prCredits(inventory, pr)),
|
||||
...inventory.commits.flatMap((item) =>
|
||||
item.authors.map((identity) => human(identity.user)),
|
||||
),
|
||||
]).map((handle) => handle.toLowerCase()),
|
||||
);
|
||||
const errors = [];
|
||||
const requiredByContext = new Map();
|
||||
const thankedByContext = new Map();
|
||||
for (const bullet of proseBullets(prose)) {
|
||||
const references = bulletPrs(bullet, inventory.repository);
|
||||
const start = bullet.indexOf("Thanks ");
|
||||
const thanked = new Set(
|
||||
[
|
||||
...bullet
|
||||
.slice(start < 0 ? bullet.length : start)
|
||||
.matchAll(/@([A-Za-z0-9-]+(?:\[bot\])?)/gu),
|
||||
].map((match) => match[1].toLowerCase()),
|
||||
);
|
||||
for (const number of references)
|
||||
if (!known.has(number) && !issues.has(number))
|
||||
errors.push(`Unknown release reference #${number}`);
|
||||
for (const { handle, context } of requiredCredits(
|
||||
inventory,
|
||||
bullet,
|
||||
references,
|
||||
)) {
|
||||
const required = requiredByContext.get(context) ?? new Set();
|
||||
required.add(handle);
|
||||
requiredByContext.set(context, required);
|
||||
const credited = thankedByContext.get(context) ?? new Set();
|
||||
for (const login of thanked) credited.add(login);
|
||||
thankedByContext.set(context, credited);
|
||||
}
|
||||
for (const handle of thanked)
|
||||
if (!eligible.has(handle))
|
||||
errors.push(`Unverified human credit @${handle}`);
|
||||
}
|
||||
for (const [context, required] of requiredByContext)
|
||||
for (const handle of required)
|
||||
if (!thankedByContext.get(context)?.has(handle.toLowerCase()))
|
||||
errors.push(`Missing Thanks @${handle} for ${context}`);
|
||||
return errors;
|
||||
}
|
||||
|
||||
export function verifyReleaseNotes(raw, text, body) {
|
||||
const inventory = releaseInventorySchema.parse(raw);
|
||||
const notes = selectedReleaseNotes(text, inventory.version);
|
||||
// Release Please's inline issue/PR body limit (pull-request-overflow-handler).
|
||||
if (notes.length > 65_536 || (body !== undefined && body.length > 65_536))
|
||||
throw new Error(
|
||||
"Release notes exceed GitHub's inline PR body limit; overflow-link bodies are not supported by this checker",
|
||||
);
|
||||
if (notes.length > 125_000 || Buffer.byteLength(notes) > 125_000)
|
||||
throw new Error(
|
||||
"Release notes exceed GitHub's body budget; shorten prose without dropping credit",
|
||||
);
|
||||
const record = contributionRecord(inventory).trim();
|
||||
const summary = contributionSummary(inventory).trim();
|
||||
const start = notes.endsWith(summary)
|
||||
? notes.length - summary.length
|
||||
: notes.indexOf("### Complete contribution record");
|
||||
if (
|
||||
start < 0 ||
|
||||
(notes.slice(start).trim() !== record &&
|
||||
notes.slice(start).trim() !== summary)
|
||||
)
|
||||
throw new Error(
|
||||
"Contribution record differs from inventory; regenerate it with release-notes record",
|
||||
);
|
||||
const prose = notes.slice(0, start);
|
||||
const errors = verifyEditorialCredits(inventory, prose);
|
||||
for (const heading of ["Highlights", "Changes", "Fixes"])
|
||||
if (!prose.includes(`### ${heading}\n`))
|
||||
errors.push(`Missing ${heading} section`);
|
||||
if (body !== undefined) {
|
||||
const bodyNotes = selectedReleaseNotes(body, inventory.version)
|
||||
.replace(
|
||||
/\n---\nThis PR was generated with \[Release Please\][\s\S]*$/u,
|
||||
"",
|
||||
)
|
||||
.trim();
|
||||
if (bodyNotes !== notes)
|
||||
errors.push("Release PR body differs from CHANGELOG.md");
|
||||
}
|
||||
if (errors.length) throw new Error(errors.join("\n"));
|
||||
return {
|
||||
version: inventory.version,
|
||||
pullRequests: inventory.pullRequests.length,
|
||||
commits: inventory.commits.length,
|
||||
unresolvedAuthors: inventory.commits.flatMap((item) =>
|
||||
item.authors
|
||||
.filter((identity) => identity.user === null)
|
||||
.map((identity) => ({ sha: item.sha, ...identity })),
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
/** Reject stale inventories and product changes made after the editorial target. */
|
||||
export async function verifyInventoryHistory(run, raw, source) {
|
||||
const inventory = releaseInventorySchema.parse(raw);
|
||||
const history = await readReleaseHistory(
|
||||
run,
|
||||
inventory.base,
|
||||
inventory.target,
|
||||
);
|
||||
const original = inventory.commits.map(
|
||||
({ authors: _authors, pullRequests: _prs, ...item }) => item,
|
||||
);
|
||||
if (JSON.stringify(history.commits) !== JSON.stringify(original))
|
||||
throw new Error("Inventory does not match complete local Git history");
|
||||
const shas = new Set(history.commits.map((item) => item.sha));
|
||||
const numbers = new Set();
|
||||
for (const pr of inventory.pullRequests) {
|
||||
if (numbers.has(pr.number) || !shas.has(pr.mergeCommit.oid))
|
||||
throw new Error(`Duplicate or out-of-range PR #${pr.number}`);
|
||||
numbers.add(pr.number);
|
||||
}
|
||||
for (const item of inventory.commits)
|
||||
for (const number of item.pullRequests)
|
||||
if (!numbers.has(number))
|
||||
throw new Error(`Commit refers to missing PR #${number}`);
|
||||
if (source !== undefined) {
|
||||
const resolved = sha.parse(
|
||||
(
|
||||
await run("git", [
|
||||
"rev-parse",
|
||||
"--verify",
|
||||
"--end-of-options",
|
||||
`${source}^{commit}`,
|
||||
])
|
||||
).trim(),
|
||||
);
|
||||
const parent = (await run("git", ["rev-parse", `${resolved}^1`])).trim();
|
||||
if (parent !== inventory.target)
|
||||
throw new Error(
|
||||
"Release base moved after notes target; regenerate inventory for the reviewed merge's first parent",
|
||||
);
|
||||
await run("git", [
|
||||
"merge-base",
|
||||
"--is-ancestor",
|
||||
inventory.target,
|
||||
resolved,
|
||||
]);
|
||||
const allowed = new Set([
|
||||
"CHANGELOG.md",
|
||||
".release-please-manifest.json",
|
||||
"package.json",
|
||||
"package-lock.json",
|
||||
"server.json",
|
||||
"src/generatedPackageMetadata.ts",
|
||||
"docs/installation.md",
|
||||
`docs/releases/${inventory.version}.contributions.json`,
|
||||
]);
|
||||
const paths = (
|
||||
await run("git", [
|
||||
"diff",
|
||||
"--name-only",
|
||||
"--no-renames",
|
||||
"-z",
|
||||
inventory.target,
|
||||
resolved,
|
||||
])
|
||||
)
|
||||
.split("\0")
|
||||
.filter(Boolean);
|
||||
const product = paths.filter((path) => !allowed.has(path));
|
||||
if (product.length)
|
||||
throw new Error(
|
||||
`Product changed after notes target; regenerate inventory: ${product.join(", ")}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
+16
-1
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { access } from "node:fs/promises";
|
||||
import { dirname, resolve } from "node:path";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
// Route production MCP before importing Incur. Incur owns registration helpers
|
||||
@@ -15,6 +15,21 @@ const isMcpMode =
|
||||
args.length === 1 && (args[0] === "--mcp" || args[0] === "mcp");
|
||||
const isMcpDoctorMode = args[0] === "mcp" && args[1] === "doctor";
|
||||
const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
|
||||
// Playwright's registry assumes one of its three desktop host platforms when
|
||||
// it chooses the browser cache directory. Termux reports `android`, although
|
||||
// its Node runtime and filesystem are POSIX-compatible. Keep the cache path
|
||||
// explicit so REA can start on Termux. Browser launch still requires a
|
||||
// caller-supplied Android-compatible Chromium executable.
|
||||
if (
|
||||
process.platform === "android" &&
|
||||
process.env.PLAYWRIGHT_BROWSERS_PATH === undefined
|
||||
) {
|
||||
const cacheRoot =
|
||||
process.env.XDG_CACHE_HOME ?? join(process.env.HOME ?? ".", ".cache");
|
||||
process.env.PLAYWRIGHT_BROWSERS_PATH = join(cacheRoot, "ms-playwright");
|
||||
}
|
||||
|
||||
const runtimeFiles = isMcpMode
|
||||
? ["dist/main.js"]
|
||||
: isMcpDoctorMode
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
#!/usr/bin/env node
|
||||
import { execFile } from "node:child_process";
|
||||
import { readFile, writeFile } from "node:fs/promises";
|
||||
import { parseArgs, promisify } from "node:util";
|
||||
import { collectGitHubCredits } from "./lib/release-notes-github.mjs";
|
||||
import { verifyNotesPublication } from "./lib/release-notes-publication.mjs";
|
||||
import {
|
||||
contributionRecord,
|
||||
contributionSummary,
|
||||
readReleaseHistory,
|
||||
releaseInventorySchema,
|
||||
selectedReleaseNotes,
|
||||
verifyInventoryHistory,
|
||||
verifyReleaseNotes,
|
||||
} from "./lib/release-notes.mjs";
|
||||
|
||||
const execFileAsync = promisify(execFile);
|
||||
async function run(command, args, input) {
|
||||
const pending = execFileAsync(command, args, {
|
||||
encoding: "utf8",
|
||||
maxBuffer: 64 * 1024 * 1024,
|
||||
});
|
||||
pending.child.stdin.end(input);
|
||||
return (await pending).stdout;
|
||||
}
|
||||
|
||||
async function createInventory(values) {
|
||||
if (!values.repository || !values.base || !values.target || !values.version)
|
||||
throw new Error(
|
||||
"inventory requires --repository, --base, --target and --version",
|
||||
);
|
||||
// Parse caller-selected boundaries before making network requests.
|
||||
const history = await readReleaseHistory(run, values.base, values.target);
|
||||
releaseInventorySchema.omit({ commits: true, pullRequests: true }).parse({
|
||||
schemaVersion: 1,
|
||||
repository: values.repository,
|
||||
version: values.version,
|
||||
base: history.base,
|
||||
target: history.target,
|
||||
});
|
||||
const credits = await collectGitHubCredits(
|
||||
run,
|
||||
values.repository,
|
||||
history.commits,
|
||||
);
|
||||
const inventory = releaseInventorySchema.parse({
|
||||
schemaVersion: 1,
|
||||
repository: values.repository,
|
||||
version: values.version,
|
||||
base: history.base,
|
||||
target: history.target,
|
||||
...credits,
|
||||
});
|
||||
await verifyInventoryHistory(run, inventory);
|
||||
await writeFile(values.manifest, `${JSON.stringify(inventory, null, 2)}\n`, {
|
||||
flag: "wx",
|
||||
});
|
||||
process.stdout.write(
|
||||
`Inventoried ${inventory.commits.length} commits and ${inventory.pullRequests.length} merged PRs. Saved ${values.manifest}\n`,
|
||||
);
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const { values, positionals } = parseArgs({
|
||||
allowPositionals: true,
|
||||
options: {
|
||||
repository: { type: "string" },
|
||||
pr: { type: "string" },
|
||||
base: { type: "string" },
|
||||
target: { type: "string" },
|
||||
version: { type: "string" },
|
||||
manifest: { type: "string" },
|
||||
notes: { type: "string", default: "CHANGELOG.md" },
|
||||
"pr-body": { type: "string" },
|
||||
source: { type: "string" },
|
||||
output: { type: "string" },
|
||||
summary: { type: "boolean", default: false },
|
||||
},
|
||||
});
|
||||
const [command] = positionals;
|
||||
if (command === "publication" && positionals.length === 1) {
|
||||
if (!values.repository || !values.source)
|
||||
throw new Error(
|
||||
"publication requires --repository and --source; --pr selects the merged release PR",
|
||||
);
|
||||
process.stdout.write(
|
||||
`${JSON.stringify(await verifyNotesPublication(run, values), null, 2)}\n`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (
|
||||
positionals.length !== 1 ||
|
||||
!["inventory", "record", "check", "render"].includes(command) ||
|
||||
!values.manifest
|
||||
)
|
||||
throw new Error(
|
||||
"Usage: release-notes inventory|record|check|render --manifest FILE [--repository OWNER/REPO --base TAG --target SHA --version VERSION] [--notes FILE --pr-body FILE --source SHA --output FILE] [--summary for record]",
|
||||
);
|
||||
if (command === "inventory") {
|
||||
await createInventory(values);
|
||||
return;
|
||||
}
|
||||
const inventory = releaseInventorySchema.parse(
|
||||
JSON.parse(await readFile(values.manifest, "utf8")),
|
||||
);
|
||||
await verifyInventoryHistory(run, inventory, values.source);
|
||||
if (command === "record") {
|
||||
const record = values.summary
|
||||
? contributionSummary(inventory)
|
||||
: contributionRecord(inventory);
|
||||
if (values.output)
|
||||
await writeFile(values.output, record, {
|
||||
flag: "wx",
|
||||
});
|
||||
else process.stdout.write(record);
|
||||
return;
|
||||
}
|
||||
const notes = await readFile(values.notes, "utf8");
|
||||
const body = values["pr-body"]
|
||||
? await readFile(values["pr-body"], "utf8")
|
||||
: undefined;
|
||||
const report = verifyReleaseNotes(inventory, notes, body);
|
||||
if (command === "check") {
|
||||
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
|
||||
return;
|
||||
}
|
||||
if (!values.output)
|
||||
throw new Error(
|
||||
"render requires --output (an existing file is never overwritten)",
|
||||
);
|
||||
const rendered = selectedReleaseNotes(notes, inventory.version);
|
||||
await writeFile(values.output, `${rendered}\n`, { flag: "wx" });
|
||||
}
|
||||
|
||||
await main().catch((error) => {
|
||||
process.stderr.write(
|
||||
`Release notes failed: ${error instanceof Error ? error.message : String(error)}\n`,
|
||||
);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
@@ -0,0 +1,481 @@
|
||||
// Run the website's browser scripts against a small document model, so missing
|
||||
// page-owned messages are checked without a browser.
|
||||
import assert from "node:assert/strict";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { test } from "node:test";
|
||||
import { setImmediate as nextTurn } from "node:timers/promises";
|
||||
import { pathToFileURL } from "node:url";
|
||||
|
||||
const ASSETS = new URL("../website/public/assets/", import.meta.url);
|
||||
|
||||
class Element {
|
||||
constructor(tag, attributes = {}, children = [], text = "") {
|
||||
this.tagName = tag.toUpperCase();
|
||||
this.attributes = new Map(Object.entries(attributes));
|
||||
this.children = [];
|
||||
this.parent = null;
|
||||
this.listeners = new Map();
|
||||
this.ownText = text;
|
||||
this.hidden = "hidden" in attributes;
|
||||
this.disabled = "disabled" in attributes;
|
||||
this.checked = "checked" in attributes;
|
||||
this.value = attributes.value ?? "";
|
||||
for (const child of children) this.append(child);
|
||||
}
|
||||
|
||||
append(child) {
|
||||
child.parent = this;
|
||||
this.children.push(child);
|
||||
}
|
||||
|
||||
get textContent() {
|
||||
return (
|
||||
this.ownText + this.children.map((child) => child.textContent).join("")
|
||||
);
|
||||
}
|
||||
|
||||
set textContent(value) {
|
||||
this.ownText = String(value);
|
||||
this.children = [];
|
||||
}
|
||||
|
||||
get id() {
|
||||
return this.getAttribute("id") ?? "";
|
||||
}
|
||||
|
||||
get lang() {
|
||||
return this.getAttribute("lang") ?? "";
|
||||
}
|
||||
|
||||
get dataset() {
|
||||
const values = {};
|
||||
for (const [name, value] of this.attributes) {
|
||||
if (!name.startsWith("data-")) continue;
|
||||
const key = name
|
||||
.slice(5)
|
||||
.replace(/-([a-z])/g, (_, letter) => letter.toUpperCase());
|
||||
values[key] = value;
|
||||
}
|
||||
return values;
|
||||
}
|
||||
|
||||
getAttribute(name) {
|
||||
return this.attributes.has(name) ? this.attributes.get(name) : null;
|
||||
}
|
||||
|
||||
setAttribute(name, value) {
|
||||
this.attributes.set(name, String(value));
|
||||
}
|
||||
|
||||
removeAttribute(name) {
|
||||
this.attributes.delete(name);
|
||||
}
|
||||
|
||||
addEventListener(type, listener) {
|
||||
this.listeners.set(type, [...(this.listeners.get(type) ?? []), listener]);
|
||||
}
|
||||
|
||||
dispatch(type, event = {}) {
|
||||
for (const listener of this.listeners.get(type) ?? []) {
|
||||
listener({ preventDefault() {}, repeat: false, ...event });
|
||||
}
|
||||
}
|
||||
|
||||
click() {
|
||||
this.dispatch("click");
|
||||
}
|
||||
|
||||
focus() {}
|
||||
|
||||
descendants() {
|
||||
return this.children.flatMap((child) => [child, ...child.descendants()]);
|
||||
}
|
||||
|
||||
matches(selector) {
|
||||
return selector
|
||||
.split(",")
|
||||
.some((part) => matchesComplex(this, part.trim()));
|
||||
}
|
||||
|
||||
closest(selector) {
|
||||
for (let node = this; node !== null; node = node.parent) {
|
||||
if (node.matches(selector)) return node;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
querySelectorAll(selector) {
|
||||
return this.descendants().filter((node) => node.matches(selector));
|
||||
}
|
||||
|
||||
querySelector(selector) {
|
||||
return this.querySelectorAll(selector)[0] ?? null;
|
||||
}
|
||||
|
||||
getContext() {
|
||||
return this.context ?? null;
|
||||
}
|
||||
}
|
||||
|
||||
// Supports the selector forms the website scripts use: tag, #id, .class and
|
||||
// [attribute], [attribute="value"] or [attribute^="value"], joined by spaces.
|
||||
function matchesCompound(node, compound) {
|
||||
const pattern =
|
||||
/^([a-z0-9]*)((?:#[\w-]+|\.[\w-]+|\[[\w-]+(?:\^?="[^"]*")?\])*)$/i;
|
||||
const parsed = pattern.exec(compound);
|
||||
if (parsed === null) throw new Error(`Unsupported selector: ${compound}`);
|
||||
const [, tag, rest] = parsed;
|
||||
if (tag && node.tagName !== tag.toUpperCase()) return false;
|
||||
for (const [, id, className, name, operator, value] of rest.matchAll(
|
||||
/#([\w-]+)|\.([\w-]+)|\[([\w-]+)(?:(\^?=)"([^"]*)")?\]/g,
|
||||
)) {
|
||||
if (id !== undefined) {
|
||||
if (node.id !== id) return false;
|
||||
continue;
|
||||
}
|
||||
if (className !== undefined) {
|
||||
const classes = (node.getAttribute("class") ?? "").split(/\s+/);
|
||||
if (!classes.includes(className)) return false;
|
||||
continue;
|
||||
}
|
||||
const actual = node.getAttribute(name);
|
||||
if (actual === null) return false;
|
||||
if (operator === "=" && actual !== value) return false;
|
||||
if (operator === "^=" && !actual.startsWith(value)) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
function matchesComplex(node, selector) {
|
||||
const parts = selector.split(/\s+/);
|
||||
if (!matchesCompound(node, parts.at(-1))) return false;
|
||||
let ancestor = node.parent;
|
||||
for (const part of parts.slice(0, -1).reverse()) {
|
||||
while (ancestor !== null && !matchesCompound(ancestor, part))
|
||||
ancestor = ancestor.parent;
|
||||
if (ancestor === null) return false;
|
||||
ancestor = ancestor.parent;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
const h = (tag, attributes, ...children) => {
|
||||
const text = children.filter((child) => typeof child === "string").join("");
|
||||
return new Element(
|
||||
tag,
|
||||
attributes ?? {},
|
||||
children.filter((child) => child instanceof Element),
|
||||
text,
|
||||
);
|
||||
};
|
||||
|
||||
// Install a fresh document and browser globals, then run one script.
|
||||
async function run(
|
||||
script,
|
||||
body,
|
||||
{ lang = "en", clipboard = "resolve", stars = 1234 } = {},
|
||||
) {
|
||||
const root = h("html", { lang }, h("body", {}, ...body));
|
||||
const timers = [];
|
||||
const written = [];
|
||||
const canvasText = [];
|
||||
for (const canvas of root.querySelectorAll("canvas")) {
|
||||
canvas.context = new Proxy(
|
||||
{ fillText: (text) => canvasText.push(text) },
|
||||
{ get: (target, name) => target[name] ?? (() => {}), set: () => true },
|
||||
);
|
||||
}
|
||||
const document = {
|
||||
documentElement: root,
|
||||
hidden: false,
|
||||
getElementById: (id) =>
|
||||
root.descendants().find((node) => node.id === id) ?? null,
|
||||
querySelector: (selector) => root.querySelector(selector),
|
||||
querySelectorAll: (selector) => root.querySelectorAll(selector),
|
||||
createRange: () => ({ selectNodeContents() {} }),
|
||||
addEventListener() {},
|
||||
};
|
||||
const window = {
|
||||
scrollY: 0,
|
||||
innerHeight: 900,
|
||||
location: { hash: "" },
|
||||
addEventListener() {},
|
||||
getSelection: () => ({ removeAllRanges() {}, addRange() {} }),
|
||||
matchMedia: () => ({ matches: false, addEventListener() {} }),
|
||||
requestAnimationFrame: () => 1,
|
||||
scrollTo() {},
|
||||
setTimeout: (callback) => timers.push(callback),
|
||||
};
|
||||
Object.assign(globalThis, {
|
||||
document,
|
||||
window,
|
||||
requestAnimationFrame: () => 1,
|
||||
cancelAnimationFrame() {},
|
||||
Path2D: class {},
|
||||
fetch: async () => ({
|
||||
ok: true,
|
||||
json: async () => ({ stargazers_count: stars }),
|
||||
}),
|
||||
});
|
||||
Object.defineProperty(globalThis, "navigator", {
|
||||
configurable: true,
|
||||
value: {
|
||||
clipboard: {
|
||||
writeText: async (text) => {
|
||||
if (clipboard === "reject") throw new Error("denied");
|
||||
written.push(text);
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
await import(
|
||||
`${pathToFileURL(new URL(script, ASSETS).pathname)}?load=${randomUUID()}`
|
||||
);
|
||||
await nextTurn();
|
||||
return { root, timers, written, canvasText };
|
||||
}
|
||||
|
||||
const copyBlock = () => [
|
||||
h("button", { "data-copy": "command" }, "Copy"),
|
||||
h("pre", {}, h("code", { id: "command" }, "npx rea-agents setup")),
|
||||
];
|
||||
const copyStatus = (messages = {}) =>
|
||||
h("p", {
|
||||
id: "copy-status",
|
||||
...Object.fromEntries(
|
||||
Object.entries(messages).map(([key, value]) => [`data-${key}`, value]),
|
||||
),
|
||||
});
|
||||
const englishCopyMessages = {
|
||||
copied: "Copied",
|
||||
selected: "Selected",
|
||||
"status-copied": "Code copied to clipboard.",
|
||||
"status-selected": "Code selected. Use your keyboard to copy it.",
|
||||
};
|
||||
|
||||
test("copy buttons report and restore localized labels", async () => {
|
||||
const { root, timers, written } = await run("main.js", [
|
||||
...copyBlock(),
|
||||
copyStatus(englishCopyMessages),
|
||||
]);
|
||||
const button = root.querySelector("[data-copy]");
|
||||
button.click();
|
||||
await nextTurn();
|
||||
assert.deepEqual(written, ["npx rea-agents setup"]);
|
||||
assert.equal(button.textContent, "Copied");
|
||||
assert.equal(
|
||||
root.querySelector("#copy-status").textContent,
|
||||
"Code copied to clipboard.",
|
||||
);
|
||||
timers.forEach((callback) => callback());
|
||||
assert.equal(button.textContent, "Copy");
|
||||
});
|
||||
|
||||
test("copy buttons keep their label when messages are missing", async () => {
|
||||
for (const [body, clipboard] of [
|
||||
[[...copyBlock(), copyStatus()], "resolve"],
|
||||
[[...copyBlock(), copyStatus()], "reject"],
|
||||
[
|
||||
[...copyBlock(), copyStatus({ copied: "", "status-copied": "" })],
|
||||
"resolve",
|
||||
],
|
||||
[copyBlock(), "resolve"],
|
||||
]) {
|
||||
const { root, timers, written } = await run("main.js", body, { clipboard });
|
||||
const button = root.querySelector("[data-copy]");
|
||||
button.click();
|
||||
await nextTurn();
|
||||
assert.equal(written.length, clipboard === "resolve" ? 1 : 0);
|
||||
assert.equal(button.textContent, "Copy");
|
||||
assert.equal(root.querySelector("#copy-status")?.textContent ?? "", "");
|
||||
timers.forEach((callback) => callback());
|
||||
assert.equal(button.textContent, "Copy");
|
||||
}
|
||||
});
|
||||
|
||||
const githubLink = (labels = {}) =>
|
||||
h(
|
||||
"a",
|
||||
{
|
||||
class: "nav-github",
|
||||
"aria-label": "REA on GitHub",
|
||||
title: "REA on GitHub",
|
||||
...labels,
|
||||
},
|
||||
h("span", { class: "nav-github-count", hidden: "" }),
|
||||
);
|
||||
|
||||
test("the star label follows the page language and plural rules", async () => {
|
||||
const labels = {
|
||||
"data-label-one": "REA on GitHub ({count} star)",
|
||||
"data-label-other": "REA on GitHub ({count} stars)",
|
||||
};
|
||||
for (const [stars, lang, expected] of [
|
||||
[1, "en", "REA on GitHub (1 star)"],
|
||||
[1234, "en", "REA on GitHub (1,234 stars)"],
|
||||
[1234, "de", "REA on GitHub (1.234 stars)"],
|
||||
]) {
|
||||
const { root } = await run("main.js", [githubLink(labels)], {
|
||||
stars,
|
||||
lang,
|
||||
});
|
||||
await nextTurn();
|
||||
assert.equal(
|
||||
root.querySelector(".nav-github").getAttribute("aria-label"),
|
||||
expected,
|
||||
);
|
||||
}
|
||||
// Languages without a "one" category use the "other" label.
|
||||
const { root } = await run(
|
||||
"main.js",
|
||||
[githubLink({ "data-label-other": "{count}" })],
|
||||
{ stars: 1, lang: "ja" },
|
||||
);
|
||||
await nextTurn();
|
||||
assert.equal(
|
||||
root.querySelector(".nav-github").getAttribute("aria-label"),
|
||||
"1",
|
||||
);
|
||||
});
|
||||
|
||||
test("the star label stays unchanged without a localized label", async () => {
|
||||
const { root } = await run("main.js", [githubLink()], { stars: 1 });
|
||||
await nextTurn();
|
||||
const link = root.querySelector(".nav-github");
|
||||
assert.equal(link.getAttribute("aria-label"), "REA on GitHub");
|
||||
assert.equal(link.getAttribute("title"), "REA on GitHub");
|
||||
assert.equal(link.querySelector(".nav-github-count").hidden, true);
|
||||
});
|
||||
|
||||
const percentDemo = (messages) =>
|
||||
h(
|
||||
"figure",
|
||||
{ "data-percent-demo": "", ...messages },
|
||||
h(
|
||||
"div",
|
||||
{ class: "percent-operation", hidden: "" },
|
||||
h(
|
||||
"button",
|
||||
{ "data-percent-operation": "add", "aria-pressed": "true" },
|
||||
"200 + 10%",
|
||||
),
|
||||
h(
|
||||
"button",
|
||||
{ "data-percent-operation": "multiply", "aria-pressed": "false" },
|
||||
"200 × 10%",
|
||||
),
|
||||
),
|
||||
h("span", { "data-percent-expression": "" }, "200 +"),
|
||||
h("strong", { "data-percent-display": "" }, "20"),
|
||||
h(
|
||||
"div",
|
||||
{ class: "percent-controls", hidden: "" },
|
||||
h("button", { "data-percent-key": "percent" }, "%"),
|
||||
h("button", { "data-percent-key": "equals", disabled: "" }, "="),
|
||||
h("button", { "data-percent-key": "reset" }, "Reset"),
|
||||
),
|
||||
h(
|
||||
"p",
|
||||
{ "data-percent-status": "" },
|
||||
"10% of 200 = 20. Then 200 + 20 = 220.",
|
||||
),
|
||||
);
|
||||
|
||||
test("the percent demo uses page messages and keeps text when one is missing", async () => {
|
||||
const { root } = await run("home.js", [
|
||||
percentDemo({
|
||||
"data-msg-input": "Press % to see what happens to 10.",
|
||||
"data-msg-result": "200 {operator} {percent} = {result}.",
|
||||
}),
|
||||
]);
|
||||
const status = root.querySelector("[data-percent-status]");
|
||||
const key = (name) =>
|
||||
root.querySelector(`[data-percent-key="${name}"]`).click();
|
||||
assert.equal(status.textContent, "Press % to see what happens to 10.");
|
||||
key("percent");
|
||||
assert.equal(status.textContent, "Press % to see what happens to 10.");
|
||||
key("equals");
|
||||
assert.equal(status.textContent, "200 + 20 = 220.");
|
||||
|
||||
const bare = await run("home.js", [percentDemo({})]);
|
||||
const bareStatus = bare.root.querySelector("[data-percent-status]");
|
||||
for (const name of ["percent", "equals", "reset"]) {
|
||||
bare.root.querySelector(`[data-percent-key="${name}"]`).click();
|
||||
assert.equal(
|
||||
bareStatus.textContent,
|
||||
"10% of 200 = 20. Then 200 + 20 = 220.",
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
const dinoDemo = (messages) =>
|
||||
h(
|
||||
"figure",
|
||||
{ "data-dino-demo": "", ...messages },
|
||||
h("svg", { class: "dino-fallback" }),
|
||||
h("canvas", { hidden: "" }),
|
||||
h(
|
||||
"div",
|
||||
{ class: "dino-controls", hidden: "" },
|
||||
h("button", { "data-dino-start": "" }, "Play"),
|
||||
h("button", { "data-dino-jump": "" }, "Jump ↑"),
|
||||
h("button", { "data-dino-reset": "" }, "Restart"),
|
||||
h("output", { "data-dino-speed": "" }, "6.0"),
|
||||
h("input", { "data-dino-slider": "", value: "6" }),
|
||||
h("input", { "data-dino-auto": "", checked: "" }),
|
||||
),
|
||||
h(
|
||||
"p",
|
||||
{ "data-dino-status": "" },
|
||||
"Original rule: start at 6, then increase toward 13.",
|
||||
),
|
||||
);
|
||||
|
||||
test("the dinosaur game never shows a missing message", async () => {
|
||||
const { root, canvasText } = await run("dino-demo.js", [
|
||||
dinoDemo({}),
|
||||
h("button", { "data-speed-check": "", hidden: "" }, "Replay"),
|
||||
h("p", { "data-speed-check-status": "" }, "Not replayed yet."),
|
||||
]);
|
||||
const start = root.querySelector("[data-dino-start]");
|
||||
const status = root.querySelector("[data-dino-status]");
|
||||
const slider = root.querySelector("[data-dino-slider]");
|
||||
start.click();
|
||||
start.click();
|
||||
slider.value = "9.5";
|
||||
slider.dispatch("input");
|
||||
root.querySelector("[data-dino-reset]").click();
|
||||
assert.equal(start.textContent, "Play");
|
||||
assert.equal(
|
||||
status.textContent,
|
||||
"Original rule: start at 6, then increase toward 13.",
|
||||
);
|
||||
assert.deepEqual(canvasText, []);
|
||||
root.querySelector("[data-speed-check]").click();
|
||||
assert.equal(
|
||||
root.querySelector("[data-speed-check-status]").textContent,
|
||||
"Not replayed yet.",
|
||||
);
|
||||
});
|
||||
|
||||
test("the dinosaur game fills placeholders from page messages", async () => {
|
||||
const { root, canvasText } = await run("dino-demo.js", [
|
||||
dinoDemo({
|
||||
"data-msg-pause": "Pause",
|
||||
"data-msg-fixed-running":
|
||||
"Fixed speed: {speed}. Space or Jump clears a cactus.",
|
||||
"data-msg-distance": "DISTANCE {distance}",
|
||||
}),
|
||||
]);
|
||||
const start = root.querySelector("[data-dino-start]");
|
||||
const slider = root.querySelector("[data-dino-slider]");
|
||||
start.click();
|
||||
assert.equal(start.textContent, "Pause");
|
||||
slider.value = "9.5";
|
||||
slider.dispatch("input");
|
||||
assert.equal(
|
||||
root.querySelector("[data-dino-status]").textContent,
|
||||
"Fixed speed: 9.5. Space or Jump clears a cactus.",
|
||||
);
|
||||
assert.ok(canvasText.includes("DISTANCE 0000"));
|
||||
});
|
||||
+212
-2
@@ -2,7 +2,10 @@
|
||||
|
||||
import contextlib
|
||||
import importlib.util
|
||||
from html import escape
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import tempfile
|
||||
@@ -137,7 +140,7 @@ class WebsiteMaintenanceChecks(unittest.TestCase):
|
||||
def test_shared_edits_preserve_body_and_catch_drift_before_rewriting(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
for name in ("public", "templates"):
|
||||
for name in ("public", "templates", "locales"):
|
||||
shutil.copytree(ROOT / "website" / name, root / "website" / name)
|
||||
page = root / "website/public/blog/touhou-reconstruction/index.html"
|
||||
original = page.read_text()
|
||||
@@ -159,7 +162,7 @@ class WebsiteMaintenanceChecks(unittest.TestCase):
|
||||
def test_new_page_escapes_metadata_stays_a_draft_and_joins_sitemap_when_ready(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
for name in ("public", "templates"):
|
||||
for name in ("public", "templates", "locales"):
|
||||
shutil.copytree(ROOT / "website" / name, root / "website" / name)
|
||||
site = root / "website/public"
|
||||
page = WEBSITE.new_page(root, "blog/new-example", 'A <test> & "result"', 'Check "quotes" & <markup>.')
|
||||
@@ -242,5 +245,212 @@ class WebsiteMaintenanceChecks(unittest.TestCase):
|
||||
worker.join()
|
||||
|
||||
|
||||
|
||||
def site_copy(directory):
|
||||
"""Copy the authored website inputs without existing translated pages."""
|
||||
root = Path(directory)
|
||||
for name in ("public", "templates", "locales"):
|
||||
shutil.copytree(ROOT / "website" / name, root / "website" / name)
|
||||
for code in WEBSITE.LANGUAGES:
|
||||
if code != "en":
|
||||
shutil.rmtree(root / "website/public" / code, ignore_errors=True)
|
||||
return root
|
||||
|
||||
|
||||
def zh_hant(key):
|
||||
"""Return one zh-Hant UI string, so assertions follow the locale file."""
|
||||
strings = json.loads((ROOT / "website/locales/zh-hant.json").read_text())["strings"]
|
||||
return escape(strings[key])
|
||||
|
||||
|
||||
def quietly(function, *arguments):
|
||||
"""Run a command helper without its progress output."""
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
return function(*arguments)
|
||||
|
||||
|
||||
def mark_reviewed(path):
|
||||
"""Apply the reviewer's change: reviewed status and date, without noindex.
|
||||
|
||||
A reviewed translation has its own search title, so give the test page one.
|
||||
"""
|
||||
source = re.sub(r"<title>(.*?)</title>", r"<title>\1 (zh-Hant)</title>", path.read_text(), count=1)
|
||||
source = source.replace('status="draft"', 'status="reviewed" reviewed="2026-10-11"', 1)
|
||||
path.write_text(source.replace(' <meta name="robots" content="noindex" />\n', "", 1))
|
||||
|
||||
|
||||
def translation_errors(root):
|
||||
"""Return the verifier's translation errors and warnings for a site copy."""
|
||||
return VERIFY.check_translations(root, VERIFY.parse_pages(root / "website/public"))
|
||||
|
||||
|
||||
class WebsiteTranslationChecks(unittest.TestCase):
|
||||
"""Create translations through the contributor command and check what readers get."""
|
||||
|
||||
def test_translate_creates_an_unlinked_noindex_draft(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = site_copy(directory)
|
||||
site = root / "website/public"
|
||||
english = (site / "get-started/index.html").read_text()
|
||||
page = quietly(WEBSITE.new_translation, root, "zh-hant", "get-started")
|
||||
source = page.read_text()
|
||||
self.assertEqual(page, site / "zh-hant/get-started/index.html")
|
||||
self.assertIn('<html lang="zh-Hant">', source)
|
||||
self.assertIn('<meta name="robots" content="noindex" />', source)
|
||||
self.assertEqual(
|
||||
WEBSITE.translation_record(source),
|
||||
{"source-sha256": WEBSITE.source_digest(english), "status": "draft"},
|
||||
)
|
||||
self.assertIn('href="../../assets/styles.css"', source)
|
||||
self.assertIn('<a class="brand" href="../../"', source)
|
||||
self.assertIn(
|
||||
'<a href="../../zh-hant/get-started/" class="nav-start" aria-current="page">'
|
||||
f'{zh_hant("navGetStarted")}</a>',
|
||||
source,
|
||||
)
|
||||
self.assertIn(
|
||||
f'<a href="../../showcase/" hreflang="en">{zh_hant("navShowcases")}'
|
||||
'<span class="nav-badge" lang="en">EN</span></a>',
|
||||
source,
|
||||
)
|
||||
self.assertIn('<li><a href="../../get-started/" hreflang="en" lang="en">English</a></li>', source)
|
||||
self.assertIn('<meta property="og:locale" content="zh_TW" />', source)
|
||||
self.assertNotIn('rel="alternate"', source)
|
||||
self.assertIn(WEBSITE.NOTICE_START + WEBSITE.NOTICE_END, source)
|
||||
self.assertEqual(english, (site / "get-started/index.html").read_text())
|
||||
errors, warnings = translation_errors(root)
|
||||
self.assertEqual(errors, [])
|
||||
self.assertLessEqual({file for file, _ in warnings}, {"website/locales/zh-hant.json"})
|
||||
self.assertEqual(VERIFY.check_site(site)[3], [])
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
PREPARE.prepare_sitemap(site)
|
||||
self.assertNotIn("zh-hant", (site / "sitemap.xml").read_text())
|
||||
for language, route, message in (
|
||||
("zh-hant", "get-started", "already exists"),
|
||||
("en", "faq", "Choose one of"),
|
||||
("ja", "missing", "No English page"),
|
||||
("ja", "zh-hant/get-started", "without a language directory"),
|
||||
("ja", "../outside", "directory route"),
|
||||
):
|
||||
with self.subTest(language=language, route=route):
|
||||
with self.assertRaisesRegex(ValueError, message):
|
||||
quietly(WEBSITE.new_translation, root, language, route)
|
||||
|
||||
def test_reviewed_translations_join_menus_hreflang_and_the_sitemap(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = site_copy(directory)
|
||||
site = root / "website/public"
|
||||
mark_reviewed(quietly(WEBSITE.new_translation, root, "zh-hant", "get-started"))
|
||||
quietly(WEBSITE.sync_pages, root)
|
||||
english = (site / "get-started/index.html").read_text()
|
||||
for tag, url in (
|
||||
("en", "https://rea.tools/get-started/"),
|
||||
("zh-Hant", "https://rea.tools/zh-hant/get-started/"),
|
||||
("x-default", "https://rea.tools/get-started/"),
|
||||
):
|
||||
self.assertIn(f'<link rel="alternate" hreflang="{tag}" href="{url}" />', english)
|
||||
self.assertIn('<meta property="og:locale:alternate" content="zh_TW" />', english)
|
||||
self.assertIn(
|
||||
'<li><a href="../zh-hant/get-started/" hreflang="zh-Hant" lang="zh-Hant">繁體中文</a></li>', english
|
||||
)
|
||||
self.assertNotIn("language-menu", (site / "faq/index.html").read_text())
|
||||
translated = (site / "zh-hant/get-started/index.html").read_text()
|
||||
self.assertIn('<meta property="og:locale:alternate" content="en_US" />', translated)
|
||||
|
||||
# A reviewed home page becomes the menu's fallback for untranslated pages.
|
||||
mark_reviewed(quietly(WEBSITE.new_translation, root, "zh-hant", "/"))
|
||||
quietly(WEBSITE.sync_pages, root)
|
||||
self.assertIn(
|
||||
'<li><a href="../zh-hant/" hreflang="zh-Hant" lang="zh-Hant">繁體中文</a></li>',
|
||||
(site / "faq/index.html").read_text(),
|
||||
)
|
||||
translated = (site / "zh-hant/get-started/index.html").read_text()
|
||||
self.assertIn('<a class="brand" href="../../zh-hant/"', translated)
|
||||
home = (site / "zh-hant/index.html").read_text()
|
||||
self.assertIn('href="get-started/"', home)
|
||||
self.assertIn('href="../showcase/"', home)
|
||||
self.assertIn('href="../assets/styles.css"', home)
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
PREPARE.prepare_sitemap(site)
|
||||
sitemap = (site / "sitemap.xml").read_text()
|
||||
self.assertIn("https://rea.tools/zh-hant/get-started/", sitemap)
|
||||
self.assertIn("https://rea.tools/zh-hant/", sitemap)
|
||||
self.assertEqual(VERIFY.check_site(site)[3], [])
|
||||
self.assertEqual(translation_errors(root)[0], [])
|
||||
|
||||
def test_english_commands_sync_and_text_changes_show_a_notice(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = site_copy(directory)
|
||||
site = root / "website/public"
|
||||
english_path = site / "get-started/index.html"
|
||||
page = quietly(WEBSITE.new_translation, root, "zh-hant", "get-started")
|
||||
english = english_path.read_text()
|
||||
identifier = WEBSITE.synced_code_ids(english)[0]
|
||||
block = WEBSITE.code_block(identifier)
|
||||
|
||||
# A command-only change is copied and keeps the translation current.
|
||||
command = block.search(english).group(2) + " --test-flag"
|
||||
english_path.write_text(block.sub(lambda match: match.group(1) + command + match.group(3), english))
|
||||
with self.assertRaisesRegex(ValueError, "Shared HTML is stale: zh-hant/get-started/index.html"):
|
||||
quietly(WEBSITE.sync_pages, root, True)
|
||||
quietly(WEBSITE.sync_pages, root)
|
||||
self.assertEqual(block.search(page.read_text()).group(2), command)
|
||||
self.assertNotIn('class="translation-notice"', page.read_text())
|
||||
self.assertEqual(translation_errors(root), ([], []))
|
||||
|
||||
# A text change marks the translation out of date.
|
||||
unchanged = english_path.read_text()
|
||||
changed = unchanged.replace(" </main>", " <p>A new English paragraph.</p>\n </main>", 1)
|
||||
english_path.write_text(changed)
|
||||
self.assertEqual(quietly(WEBSITE.sync_pages, root, True), [page])
|
||||
errors, warnings = translation_errors(root)
|
||||
self.assertEqual(errors, [])
|
||||
self.assertTrue(any("has changed since this translation" in message for _, message in warnings))
|
||||
quietly(WEBSITE.sync_pages, root)
|
||||
link = f'<a href="../../get-started/" hreflang="en">{zh_hant("staleNoticeLink")}</a>'
|
||||
self.assertIn(
|
||||
f'<p class="translation-notice" role="note">{zh_hant("staleNotice").replace("{link}", link)}</p>',
|
||||
page.read_text(),
|
||||
)
|
||||
report = quietly(WEBSITE.translation_report, root)
|
||||
self.assertIn(f' after updating the translation, record source-sha256="{WEBSITE.source_digest(changed)}"', report)
|
||||
|
||||
english_path.write_text(unchanged)
|
||||
quietly(WEBSITE.sync_pages, root)
|
||||
self.assertNotIn('class="translation-notice"', page.read_text())
|
||||
|
||||
page.write_text(block.sub(lambda match: match.group(1) + "edited" + match.group(3), page.read_text()))
|
||||
self.assertTrue(any(f"{identifier} must match" in error for error in translation_errors(root)[0]))
|
||||
|
||||
def test_translations_keep_structure_and_cjk_spacing(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = site_copy(directory)
|
||||
page = quietly(WEBSITE.new_translation, root, "zh-hant", "get-started")
|
||||
original = page.read_text()
|
||||
first, second = WEBSITE.copy_targets(original)[:2]
|
||||
|
||||
def added(paragraph):
|
||||
return (" </main>", f" {paragraph}\n </main>")
|
||||
|
||||
cases = (
|
||||
(*added("<p>完成後\n 重新啟動。</p>"), "renders as a gap"),
|
||||
('<main id="main"', '<main id="moved"', "element IDs differ"),
|
||||
(f'data-copy="{first}"', f'data-copy="{second}"', "copy buttons differ"),
|
||||
(' <meta name="robots" content="noindex" />\n', "", "draft translation needs"),
|
||||
('status="draft"', 'status="final"', "status must be draft or reviewed"),
|
||||
)
|
||||
for old, new, message in cases:
|
||||
with self.subTest(message=message):
|
||||
self.assertIn(old, original)
|
||||
page.write_text(original.replace(old, new, 1))
|
||||
self.assertTrue(any(message in error for error in translation_errors(root)[0]))
|
||||
page.write_text(original.replace(*added("<p>重新啟動 REA 用戶端。</p>"), 1))
|
||||
self.assertEqual(translation_errors(root)[0], [])
|
||||
page.write_text(original.replace(*added("<p><code>extra-command</code></p>"), 1))
|
||||
errors, warnings = translation_errors(root)
|
||||
self.assertEqual(errors, [])
|
||||
self.assertTrue(any("inline code differs" in message for _, message in warnings))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -15,6 +15,10 @@ import { promisify } from "node:util";
|
||||
import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import { PrivateRuntimeRoot } from "../dist/process/PrivateRuntimeRoot.js";
|
||||
import {
|
||||
OwnedCommandFailure,
|
||||
runOwnedCommand,
|
||||
} from "../dist/process/OwnedCommand.js";
|
||||
import { parseEvidence } from "../dist/domain/evidence.js";
|
||||
import { mcpTextValue } from "./lib/mcp-verifier-results.mjs";
|
||||
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
|
||||
@@ -56,6 +60,13 @@ const root = await PrivateRuntimeRoot.create({
|
||||
const environment = Object.fromEntries(
|
||||
Object.entries(process.env).filter(([, value]) => typeof value === "string"),
|
||||
);
|
||||
const limitedPython = join(root.path, "limited-python");
|
||||
const limitedEnvironment = (option) => ({
|
||||
...environment,
|
||||
REA_PWNTOOLS_PYTHON: limitedPython,
|
||||
REA_VERIFY_LIMITED_PYTHON: python,
|
||||
REA_VERIFY_LIMIT_OPTION: option,
|
||||
});
|
||||
const client = new Client({ name: "binary-layout-verifier", version: "1" });
|
||||
const transport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
@@ -73,6 +84,46 @@ let cases = 0;
|
||||
let bootstrapCases = 0;
|
||||
const failures = [];
|
||||
try {
|
||||
await execute(
|
||||
"gcc",
|
||||
[
|
||||
"-std=c11",
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
"-Werror",
|
||||
fileURLToPath(
|
||||
new URL("./fixtures/binary-layout-limited-python.c", import.meta.url),
|
||||
),
|
||||
"-o",
|
||||
limitedPython,
|
||||
],
|
||||
{ timeout: 30_000, maxBuffer: 1024 * 1024 },
|
||||
);
|
||||
// A live limited child must retain its configured launcher's identity so
|
||||
// timeout cleanup can still verify and terminate the entire owned group.
|
||||
await assert.rejects(
|
||||
runOwnedCommand(
|
||||
{
|
||||
command: limitedPython,
|
||||
arguments: [
|
||||
"-I",
|
||||
"-c",
|
||||
"import time; print('limited child ready', flush=True); time.sleep(60)",
|
||||
],
|
||||
cwd: root.path,
|
||||
runId: `${run.run_id}-limited-python-timeout`,
|
||||
hostEnvironment: limitedEnvironment("--as=2147483648"),
|
||||
},
|
||||
{ timeoutMs: 2000, diagnosticBytes: 1024 },
|
||||
),
|
||||
(error) => {
|
||||
assert.ok(error instanceof OwnedCommandFailure);
|
||||
assert.equal(error.reason, "timeout");
|
||||
assert.equal(error.cleanupFailure, null);
|
||||
assert.ok(error.snapshot.stdout.text.includes("limited child ready"));
|
||||
return true;
|
||||
},
|
||||
);
|
||||
const bootstrapRoot = join(root.path, "bootstrap-boundary");
|
||||
await mkdir(bootstrapRoot);
|
||||
const bootstrapPath = join(bootstrapRoot, "layout.py");
|
||||
@@ -796,19 +847,12 @@ try {
|
||||
assert.deepEqual(await readFile(extendedPath), extendedObject);
|
||||
cases++;
|
||||
}
|
||||
for (const [name, option, expected] of [
|
||||
["address-space", "--as=2147483648", '"address_space_bytes": 2147483648'],
|
||||
["cpu", "--cpu=20", '"cpu_seconds": 20'],
|
||||
["file-size", "--fsize=33554432", '"file_size_bytes": 33554432'],
|
||||
for (const [option, expected] of [
|
||||
["--as=2147483648", '"address_space_bytes": 2147483648'],
|
||||
["--cpu=20", '"cpu_seconds": 20'],
|
||||
["--fsize=33554432", '"file_size_bytes": 33554432'],
|
||||
]) {
|
||||
const wrapper = join(root.path, `python-limit-${name}`);
|
||||
const quotedPython = "'" + python.replaceAll("'", "'\"'\"'") + "'";
|
||||
await writeFile(
|
||||
wrapper,
|
||||
`#!/bin/sh\nexec /usr/bin/prlimit ${option} -- ${quotedPython} "$@"\n`,
|
||||
{ mode: 0o700 },
|
||||
);
|
||||
const limitedEnvironment = { ...environment, REA_PWNTOOLS_PYTHON: wrapper };
|
||||
const selectedEnvironment = limitedEnvironment(option);
|
||||
const limitedClient = new Client({
|
||||
name: "limited-layout-verifier",
|
||||
version: "1",
|
||||
@@ -816,7 +860,7 @@ try {
|
||||
const limitedTransport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
args: [entrypoint, "mcp"],
|
||||
env: limitedEnvironment,
|
||||
env: selectedEnvironment,
|
||||
stderr: "pipe",
|
||||
});
|
||||
try {
|
||||
@@ -826,7 +870,7 @@ try {
|
||||
mode,
|
||||
join(root.path, "protected"),
|
||||
undefined,
|
||||
limitedEnvironment,
|
||||
selectedEnvironment,
|
||||
limitedClient,
|
||||
);
|
||||
assert.ok(value.limitations.some((item) => item.includes(expected)));
|
||||
@@ -862,17 +906,29 @@ try {
|
||||
sectionHeavy.writeUInt16LE(sectionCount, 60);
|
||||
const sectionHeavyPath = join(root.path, "section-heavy.o");
|
||||
await writeFile(sectionHeavyPath, sectionHeavy);
|
||||
const memoryWrapper = join(root.path, "python-memory-constraint");
|
||||
const quotedPython = "'" + python.replaceAll("'", "'\"'\"'") + "'";
|
||||
await writeFile(
|
||||
memoryWrapper,
|
||||
`#!/bin/sh\nexec /usr/bin/prlimit --as=100663296 -- ${quotedPython} "$@"\n`,
|
||||
{ mode: 0o700 },
|
||||
// pwntools materializes each PT_LOAD zero-filled tail during inspection.
|
||||
// One allocation larger than the whole address-space budget fails promptly;
|
||||
// a large section count instead races CPU work against the command deadline.
|
||||
const memoryHeavy = Buffer.from(await readFile(join(root.path, "protected")));
|
||||
const memoryPhOffset = Number(memoryHeavy.readBigUInt64LE(32));
|
||||
const memoryPhSize = memoryHeavy.readUInt16LE(54);
|
||||
const memoryPhCount = memoryHeavy.readUInt16LE(56);
|
||||
const lastLoadIndex = Array.from(
|
||||
{ length: memoryPhCount },
|
||||
(_, index) => index,
|
||||
).findLast(
|
||||
(index) =>
|
||||
memoryHeavy.readUInt32LE(memoryPhOffset + index * memoryPhSize) === 1,
|
||||
);
|
||||
const memoryEnvironment = {
|
||||
...environment,
|
||||
REA_PWNTOOLS_PYTHON: memoryWrapper,
|
||||
};
|
||||
assert.notEqual(lastLoadIndex, undefined);
|
||||
const memorySizeOffset = memoryPhOffset + lastLoadIndex * memoryPhSize + 40;
|
||||
memoryHeavy.writeBigUInt64LE(
|
||||
memoryHeavy.readBigUInt64LE(memorySizeOffset) + 128n * 1024n * 1024n,
|
||||
memorySizeOffset,
|
||||
);
|
||||
const memoryHeavyPath = join(root.path, "memory-zero-tail");
|
||||
await writeFile(memoryHeavyPath, memoryHeavy);
|
||||
const memoryEnvironment = limitedEnvironment("--as=100663296");
|
||||
const memoryClient = new Client({
|
||||
name: "memory-layout-verifier",
|
||||
version: "1",
|
||||
@@ -888,7 +944,7 @@ try {
|
||||
for (const mode of ["cli", "mcp"]) {
|
||||
const error = await inspect(
|
||||
mode,
|
||||
sectionHeavyPath,
|
||||
memoryHeavyPath,
|
||||
"resource_constraint",
|
||||
memoryEnvironment,
|
||||
memoryClient,
|
||||
@@ -903,7 +959,7 @@ try {
|
||||
assert.ok(error.remediation.action.includes("memory"));
|
||||
cases++;
|
||||
}
|
||||
assert.deepEqual(await readFile(sectionHeavyPath), sectionHeavy);
|
||||
assert.deepEqual(await readFile(memoryHeavyPath), memoryHeavy);
|
||||
} finally {
|
||||
try {
|
||||
await memoryClient.close();
|
||||
@@ -924,13 +980,7 @@ try {
|
||||
cpuHeavy.writeBigUInt64LE(BigInt(cpuSectionCount), sectionOffset + 32);
|
||||
const cpuHeavyPath = join(root.path, "cpu-section-heavy.o");
|
||||
await writeFile(cpuHeavyPath, cpuHeavy);
|
||||
const cpuWrapper = join(root.path, "python-cpu-constraint");
|
||||
await writeFile(
|
||||
cpuWrapper,
|
||||
`#!/bin/sh\nexec /usr/bin/prlimit --cpu=1: --core=0: -- ${quotedPython} "$@"\n`,
|
||||
{ mode: 0o700 },
|
||||
);
|
||||
const cpuEnvironment = { ...environment, REA_PWNTOOLS_PYTHON: cpuWrapper };
|
||||
const cpuEnvironment = limitedEnvironment("--cpu=1:");
|
||||
const cpuClient = new Client({ name: "cpu-layout-verifier", version: "1" });
|
||||
const cpuTransport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
@@ -963,13 +1013,7 @@ try {
|
||||
}
|
||||
}
|
||||
for (const limit of [1024, 1]) {
|
||||
const wrapper = join(root.path, `python-file-size-${limit}`);
|
||||
await writeFile(
|
||||
wrapper,
|
||||
`#!/bin/sh\nexec /usr/bin/prlimit --fsize=${limit}: --core=0: -- ${quotedPython} "$@"\n`,
|
||||
{ mode: 0o700 },
|
||||
);
|
||||
const fileEnvironment = { ...environment, REA_PWNTOOLS_PYTHON: wrapper };
|
||||
const fileEnvironment = limitedEnvironment(`--fsize=${limit}:`);
|
||||
const fileClient = new Client({
|
||||
name: "file-size-layout-verifier",
|
||||
version: "1",
|
||||
|
||||
+145
-97
@@ -38,104 +38,147 @@ let update;
|
||||
let repositorySkill;
|
||||
let report;
|
||||
|
||||
const runStage = async (name, action) => {
|
||||
const started = performance.now();
|
||||
process.stderr.write(`[verify:package] ${name}: start\n`);
|
||||
const result = await action();
|
||||
process.stderr.write(
|
||||
`[verify:package] ${name}: complete (${Math.round(performance.now() - started)} ms)\n`,
|
||||
);
|
||||
return result;
|
||||
};
|
||||
|
||||
try {
|
||||
repositorySkill = await verifyRepositorySkill({ root, workspace });
|
||||
({ tarball } = await verifyPackagePack({ root, workspace }));
|
||||
const environmentData = await verifyPackageEnvironment({
|
||||
root,
|
||||
workspace,
|
||||
evidenceRoot,
|
||||
referenceRoot,
|
||||
});
|
||||
const { cli, packageRunnerCli } = await verifyPackageInstall({
|
||||
tarball,
|
||||
prefix: environmentData.prefix,
|
||||
workspace,
|
||||
environment: environmentData.environment,
|
||||
});
|
||||
update = await verifyPackageUpdate({
|
||||
prefix: environmentData.prefix,
|
||||
packageRoot: join(
|
||||
environmentData.prefix,
|
||||
"lib",
|
||||
"node_modules",
|
||||
PRODUCT_IDENTITY.packageName,
|
||||
),
|
||||
tarball,
|
||||
workspace,
|
||||
environment: environmentData.environment,
|
||||
});
|
||||
mcpStartup = await measureMcpStartup({
|
||||
command: cli,
|
||||
args: ["mcp"],
|
||||
environment: {
|
||||
...environmentData.environment,
|
||||
},
|
||||
policy: MCP_STARTUP_POLICY,
|
||||
});
|
||||
mcpModuleLoading = await profileMcpModuleLoading({
|
||||
command: cli,
|
||||
args: ["mcp"],
|
||||
environment: {
|
||||
...environmentData.environment,
|
||||
},
|
||||
policy: MCP_STARTUP_POLICY,
|
||||
packageName: PRODUCT_IDENTITY.packageName,
|
||||
});
|
||||
const { supportedSetupHost, hopperSetupSupported } =
|
||||
await verifyPackageDiscovery({
|
||||
repositorySkill = await runStage("repository skill", () =>
|
||||
verifyRepositorySkill({ root, workspace }),
|
||||
);
|
||||
({ tarball } = await runStage("package archive", () =>
|
||||
verifyPackagePack({ root, workspace }),
|
||||
));
|
||||
const environmentData = await runStage("package environment", () =>
|
||||
verifyPackageEnvironment({
|
||||
root,
|
||||
workspace,
|
||||
evidenceRoot,
|
||||
referenceRoot,
|
||||
}),
|
||||
);
|
||||
const { cli, packageRunnerCli } = await runStage("package install", () =>
|
||||
verifyPackageInstall({
|
||||
tarball,
|
||||
prefix: environmentData.prefix,
|
||||
workspace,
|
||||
environment: environmentData.environment,
|
||||
}),
|
||||
);
|
||||
update = await runStage("package update", () =>
|
||||
verifyPackageUpdate({
|
||||
prefix: environmentData.prefix,
|
||||
packageRoot: join(
|
||||
environmentData.prefix,
|
||||
"lib",
|
||||
"node_modules",
|
||||
PRODUCT_IDENTITY.packageName,
|
||||
),
|
||||
tarball,
|
||||
workspace,
|
||||
environment: environmentData.environment,
|
||||
}),
|
||||
);
|
||||
mcpStartup = await runStage("MCP startup", () =>
|
||||
measureMcpStartup({
|
||||
command: cli,
|
||||
args: ["mcp"],
|
||||
environment: {
|
||||
...environmentData.environment,
|
||||
},
|
||||
policy: MCP_STARTUP_POLICY,
|
||||
}),
|
||||
);
|
||||
mcpModuleLoading = await runStage("MCP module loading", () =>
|
||||
profileMcpModuleLoading({
|
||||
command: cli,
|
||||
args: ["mcp"],
|
||||
environment: {
|
||||
...environmentData.environment,
|
||||
},
|
||||
policy: MCP_STARTUP_POLICY,
|
||||
packageName: PRODUCT_IDENTITY.packageName,
|
||||
}),
|
||||
);
|
||||
const { supportedSetupHost, hopperSetupSupported } = await runStage(
|
||||
"package discovery",
|
||||
() =>
|
||||
verifyPackageDiscovery({
|
||||
cli,
|
||||
environment: environmentData.environment,
|
||||
}),
|
||||
);
|
||||
const { artifactArchive } = await runStage("artifact and Electron CLI", () =>
|
||||
verifyPackageArtifactAndElectron({
|
||||
cli,
|
||||
workspace,
|
||||
environment: environmentData.environment,
|
||||
}),
|
||||
);
|
||||
await runStage("managed analysis CLI", () =>
|
||||
verifyManaged({
|
||||
cli,
|
||||
workspace,
|
||||
environment: environmentData.environment,
|
||||
}),
|
||||
);
|
||||
await runStage("unknown provider CLI", () =>
|
||||
verifyUnknownProvider({
|
||||
cli,
|
||||
environment: environmentData.environment,
|
||||
});
|
||||
const { artifactArchive } = await verifyPackageArtifactAndElectron({
|
||||
cli,
|
||||
workspace,
|
||||
environment: environmentData.environment,
|
||||
});
|
||||
await verifyManaged({
|
||||
cli,
|
||||
workspace,
|
||||
environment: environmentData.environment,
|
||||
});
|
||||
await verifyUnknownProvider({
|
||||
cli,
|
||||
environment: environmentData.environment,
|
||||
});
|
||||
await verifyPackagePlatform({
|
||||
cli,
|
||||
environment: environmentData.environment,
|
||||
});
|
||||
await verifyPackageCapabilitiesAndSearch({
|
||||
cli,
|
||||
environment: environmentData.environment,
|
||||
});
|
||||
await verifyPackageEvidence({
|
||||
cli,
|
||||
evidenceRoot,
|
||||
referenceRoot,
|
||||
environment: environmentData.environment,
|
||||
});
|
||||
await verifyPackageSetup({
|
||||
cli,
|
||||
packageRunnerCli,
|
||||
environment: environmentData.environment,
|
||||
home: environmentData.home,
|
||||
npxLog: environmentData.npxLog,
|
||||
claudeConfig: environmentData.claudeConfig,
|
||||
codexConfig: environmentData.codexConfig,
|
||||
cursorConfig: environmentData.cursorConfig,
|
||||
codexTarget: environmentData.codexTarget,
|
||||
cursorTarget: environmentData.cursorTarget,
|
||||
supportedSetupHost,
|
||||
hopperSetupSupported,
|
||||
root,
|
||||
});
|
||||
await verifyPackageMcp({
|
||||
cli,
|
||||
environment: environmentData.environment,
|
||||
evidenceRoot,
|
||||
artifactArchive,
|
||||
});
|
||||
}),
|
||||
);
|
||||
await runStage("platform CLI", () =>
|
||||
verifyPackagePlatform({
|
||||
cli,
|
||||
environment: environmentData.environment,
|
||||
}),
|
||||
);
|
||||
await runStage("capabilities and search CLI", () =>
|
||||
verifyPackageCapabilitiesAndSearch({
|
||||
cli,
|
||||
environment: environmentData.environment,
|
||||
}),
|
||||
);
|
||||
await runStage("evidence CLI", () =>
|
||||
verifyPackageEvidence({
|
||||
cli,
|
||||
evidenceRoot,
|
||||
referenceRoot,
|
||||
environment: environmentData.environment,
|
||||
}),
|
||||
);
|
||||
await runStage("setup and uninstall CLI", () =>
|
||||
verifyPackageSetup({
|
||||
cli,
|
||||
packageRunnerCli,
|
||||
environment: environmentData.environment,
|
||||
home: environmentData.home,
|
||||
npxLog: environmentData.npxLog,
|
||||
claudeConfig: environmentData.claudeConfig,
|
||||
codexConfig: environmentData.codexConfig,
|
||||
cursorConfig: environmentData.cursorConfig,
|
||||
codexTarget: environmentData.codexTarget,
|
||||
cursorTarget: environmentData.cursorTarget,
|
||||
supportedSetupHost,
|
||||
hopperSetupSupported,
|
||||
root,
|
||||
}),
|
||||
);
|
||||
await runStage("MCP tools and evidence", () =>
|
||||
verifyPackageMcp({
|
||||
cli,
|
||||
environment: environmentData.environment,
|
||||
evidenceRoot,
|
||||
artifactArchive,
|
||||
}),
|
||||
);
|
||||
report = {
|
||||
cli: true,
|
||||
analysisCli: true,
|
||||
@@ -173,8 +216,13 @@ try {
|
||||
mcpModuleLoading,
|
||||
};
|
||||
} finally {
|
||||
await rm(workspace, { recursive: true, force: true });
|
||||
await runStage("workspace cleanup", () =>
|
||||
rm(workspace, { recursive: true, force: true }),
|
||||
);
|
||||
}
|
||||
const verifierLineage = await runStage("verifier lineage", () =>
|
||||
completeVerifierRun(verifierRun),
|
||||
);
|
||||
process.stdout.write(
|
||||
`${JSON.stringify({ verifier_run: await completeVerifierRun(verifierRun), ...report })}\n`,
|
||||
`${JSON.stringify({ verifier_run: verifierLineage, ...report })}\n`,
|
||||
);
|
||||
|
||||
@@ -175,6 +175,7 @@ const callerDirectory = join(workspace, "caller cwd with spaces");
|
||||
const scriptCollisions = [
|
||||
"ReaGhidraBridge.java",
|
||||
"ReaGhidraPrepareCom.java",
|
||||
"ReaGhidraApplySeeds.java",
|
||||
"ReaGhidraNoReturnFix.java",
|
||||
];
|
||||
await mkdir(callerDirectory);
|
||||
|
||||
@@ -45,6 +45,7 @@ import { verifyBrowserDomDestinations } from "./lib/browser-dom-destinations-e2e
|
||||
import { verifyBrowserCaptureMetadataBudget } from "./lib/browser-capture-metadata-budget-e2e.mjs";
|
||||
import { artifactCliEvidence, artifactMcpResult } from "./lib/artifact-e2e.mjs";
|
||||
import { verifyScenarioEnvironment } from "./lib/browser-scenario-environment-e2e.mjs";
|
||||
import { verifyScenarioStorage } from "./lib/browser-scenario-storage-e2e.mjs";
|
||||
|
||||
const REAL_BROWSER_STARTUP_TIMEOUT_MS = 60_000;
|
||||
const SCENARIO_SECRET_VALUE = "rea-browser-verifier-secret";
|
||||
@@ -64,6 +65,7 @@ try {
|
||||
executable,
|
||||
[
|
||||
"--headless=new",
|
||||
"--password-store=basic",
|
||||
...(process.env.REA_BROWSER_NO_SANDBOX === "true"
|
||||
? ["--no-sandbox"]
|
||||
: []),
|
||||
@@ -275,11 +277,18 @@ try {
|
||||
throw new Error("Scenario attachment terminated its external browser");
|
||||
|
||||
const profilesBefore = await scenarioProfiles();
|
||||
process.stderr.write("Browser verifier: scenario environment\n");
|
||||
const scenarioEnvironment = await verifyScenarioEnvironment(
|
||||
endpoint,
|
||||
target,
|
||||
site.origin,
|
||||
);
|
||||
process.stderr.write("Browser verifier: scenario storage\n");
|
||||
const scenarioStorage = await verifyScenarioStorage({
|
||||
executable,
|
||||
endpoint,
|
||||
targetId: target,
|
||||
});
|
||||
const launchedScenario = await createBrowserScenarioProvider(
|
||||
process.env,
|
||||
).captureScenario(
|
||||
@@ -332,7 +341,9 @@ try {
|
||||
`Browser scenario included unselected structured value: ${unselectedValue}`,
|
||||
);
|
||||
|
||||
process.stderr.write("Browser verifier: large screenshot\n");
|
||||
const largeScreenshot = await verifyLargeScreenshotE2e(endpoint, site.origin);
|
||||
process.stderr.write("Browser verifier: popup coverage\n");
|
||||
const popupEvents = await verifyPopupEventCoverage(executable);
|
||||
const networkEvidence = await verifyBrowserNetworkEvidence(
|
||||
executable,
|
||||
@@ -354,7 +365,11 @@ try {
|
||||
scripts: observed.value.scripts.items.length,
|
||||
networkRequests: observed.value.network.requests.length,
|
||||
consoleEvents: observed.value.console.events.length,
|
||||
websocketEvents: observed.value.network.websocket_events.length,
|
||||
websocketConnections: observed.value.network.websocket_connections.length,
|
||||
websocketFrames: observed.value.network.websocket_connections.reduce(
|
||||
(total, connection) => total + connection.events.length,
|
||||
0,
|
||||
),
|
||||
bundleScripts: bundle.capture.scripts_analyzed,
|
||||
sourceMaps: bundle.observations.source_maps.processed,
|
||||
bundle_cli_and_stdio_mcp: true,
|
||||
@@ -373,6 +388,7 @@ try {
|
||||
browserScenarioLaunchCleanup: "terminated-owned-process",
|
||||
scenarioFailure,
|
||||
scenarioEnvironment,
|
||||
scenarioStorage,
|
||||
verified: true,
|
||||
};
|
||||
} finally {
|
||||
|
||||
@@ -21,6 +21,10 @@ import {
|
||||
} from "./verify-real-ghidra-assertions.mjs";
|
||||
|
||||
async function functionCall(client, operation, parameters) {
|
||||
return (await functionObservationCall(client, operation, parameters)).value;
|
||||
}
|
||||
|
||||
async function functionObservationCall(client, operation, parameters) {
|
||||
const input = parseGhidraFunctionInput(operation, parameters);
|
||||
if (!input.ok) throw input.error;
|
||||
const called = await client.callTool(operation, input.value);
|
||||
@@ -153,6 +157,57 @@ export async function verifyDebugFunctionOperations(
|
||||
const denseSwitchDossier = await functionCall(client, "analyze_function", {
|
||||
procedure: denseSwitchProcedure.address,
|
||||
});
|
||||
const terminalProcedure = requireProcedure(
|
||||
procedures,
|
||||
"rea_ghidra_inventory_terminal_call",
|
||||
);
|
||||
const terminalObservations = await Promise.all(
|
||||
[
|
||||
"procedure_info",
|
||||
"procedure_assembly",
|
||||
"read_function_instructions",
|
||||
"procedure_pseudo_code",
|
||||
"analyze_function",
|
||||
].map((operation) =>
|
||||
functionObservationCall(client, operation, {
|
||||
...(operation === "analyze_function" ? {} : { document: null }),
|
||||
procedure: terminalProcedure.address,
|
||||
}),
|
||||
),
|
||||
);
|
||||
const terminalLimitations = terminalObservations.map((result) =>
|
||||
result.limitations.find(
|
||||
(limitation) =>
|
||||
/terminal call at 0x[0-9a-f]+/i.test(limitation) &&
|
||||
/ to 0x[0-9a-f]+/i.test(limitation) &&
|
||||
limitation.includes("abort") &&
|
||||
limitation.includes("hasNoReturn=true"),
|
||||
),
|
||||
);
|
||||
if (
|
||||
terminalLimitations.some((limitation) => limitation === undefined) ||
|
||||
terminalLimitations.some(
|
||||
(limitation) =>
|
||||
!limitation.includes("external=") ||
|
||||
!limitation.includes("thunk=") ||
|
||||
!limitation.includes("fallthrough is excluded") ||
|
||||
!limitation.includes("not an independent verification"),
|
||||
)
|
||||
)
|
||||
throw new Error(
|
||||
`Ghidra no-return boundary evidence was missing callee, address, or flag provenance: ${JSON.stringify(terminalObservations.map(({ limitations }) => limitations))}`,
|
||||
);
|
||||
const returningObservation = await functionObservationCall(
|
||||
client,
|
||||
"procedure_info",
|
||||
{ document: null, procedure: leaf.address },
|
||||
);
|
||||
if (
|
||||
returningObservation.limitations.some((limitation) =>
|
||||
limitation.includes("puts"),
|
||||
)
|
||||
)
|
||||
throw new Error("Ghidra marked an ordinary returning call as terminal");
|
||||
// The debug fixture is compiled -O0, and GCC 16 and later lower dense
|
||||
// switches to compare chains at -O0, so this dossier asserts only its
|
||||
// complete shape; the optimized jump-table variant proves dense jump-table
|
||||
@@ -179,6 +234,8 @@ export async function verifyDebugFunctionOperations(
|
||||
instruction_window: {
|
||||
returned: instructionWindow.instructions.length,
|
||||
},
|
||||
terminal_call_limitations: terminalLimitations,
|
||||
returning_call_limitations: returningObservation.limitations,
|
||||
cancellation,
|
||||
timeout,
|
||||
concurrency: concurrent,
|
||||
@@ -617,6 +674,113 @@ export async function verifyRelativeSwitch(client, procedures, entrySize) {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Oracle for relative-switch-signed.S: signed halfword byte offsets from a base
|
||||
* inside the case bodies, a CBZ route for the zero entry, and a B.CS bound.
|
||||
*/
|
||||
const SIGNED_RELATIVE_SWITCH_CASES = [
|
||||
{ value: 0, immediate: 200 },
|
||||
{ value: 1, immediate: 201 },
|
||||
{ value: 2, immediate: 202 },
|
||||
{ value: 3, immediate: 250 },
|
||||
{ value: 4, immediate: 204 },
|
||||
];
|
||||
|
||||
export async function verifyRelativeSignedSwitch(client, selectorBias = 0) {
|
||||
const names = await inventoryCall(client, "list_names", {
|
||||
document: null,
|
||||
address: null,
|
||||
});
|
||||
const procedure = requireProcedure(names, "rea_relative_switch_signed");
|
||||
const dossier = await functionCall(client, "analyze_function", {
|
||||
procedure: procedure.address,
|
||||
});
|
||||
const table = dossier.native_api?.jump_tables.find((item) =>
|
||||
item.data_sources.some(
|
||||
(source) =>
|
||||
source.entry_size_bytes === 2 &&
|
||||
source.entry_count === SIGNED_RELATIVE_SWITCH_CASES.length,
|
||||
),
|
||||
);
|
||||
if (table === undefined)
|
||||
throw new Error(
|
||||
`Exact signed halfword relative table unavailable: ${JSON.stringify(dossier.native_api)}`,
|
||||
);
|
||||
const expectedValues = SIGNED_RELATIVE_SWITCH_CASES.map(
|
||||
({ value }) => value + selectorBias,
|
||||
);
|
||||
if (
|
||||
table.mappings.some(
|
||||
(mapping) =>
|
||||
mapping.case_value !== null &&
|
||||
!expectedValues.includes(mapping.case_value),
|
||||
)
|
||||
)
|
||||
throw new Error(
|
||||
`Signed relative switch emitted an unsupported case label: ${JSON.stringify(table)}`,
|
||||
);
|
||||
if (table.mappings.length !== SIGNED_RELATIVE_SWITCH_CASES.length)
|
||||
throw new Error(
|
||||
`Signed relative switch has unexpected mappings: ${JSON.stringify(table)}`,
|
||||
);
|
||||
if (selectorBias !== 0) {
|
||||
if (
|
||||
!table.limitations.some((limitation) =>
|
||||
limitation.includes("table indices are not proven"),
|
||||
) ||
|
||||
!table.mappings.some((mapping) => mapping.case_value === null) ||
|
||||
table.mappings.some((mapping) => mapping.case_value === selectorBias + 3)
|
||||
)
|
||||
throw new Error(
|
||||
`Normalized selector must retain unknown case evidence: ${JSON.stringify(table)}`,
|
||||
);
|
||||
}
|
||||
for (const { value: tableIndex, immediate } of SIGNED_RELATIVE_SWITCH_CASES) {
|
||||
const value = tableIndex + selectorBias;
|
||||
// Without a proven relationship between the machine index and the typed
|
||||
// switch selector, the CBZ-only case must stay unknown rather than being
|
||||
// assigned the normalized index or Ghidra's unreachable base destination.
|
||||
if (selectorBias !== 0 && tableIndex === 3) continue;
|
||||
const mapping = table.mappings.find((item) => item.case_value === value);
|
||||
if (
|
||||
mapping === undefined ||
|
||||
mapping.confidence !== "high" ||
|
||||
(selectorBias !== 0 &&
|
||||
!mapping.evidence.some(
|
||||
(evidence) => evidence.source === "ghidra-clang-case-token",
|
||||
))
|
||||
)
|
||||
throw new Error(
|
||||
`Signed relative switch case ${value} was not verified: ${JSON.stringify(table)}`,
|
||||
);
|
||||
const instruction = await functionCall(
|
||||
client,
|
||||
"inspect_native_instruction",
|
||||
{ address: mapping.target_address },
|
||||
);
|
||||
if (
|
||||
instruction.status !== "decoded" ||
|
||||
!instruction.operands.some((operand) =>
|
||||
operand.components.some(
|
||||
(component) =>
|
||||
component.kind === "immediate" &&
|
||||
component.value === `0x${immediate.toString(16)}`,
|
||||
),
|
||||
)
|
||||
)
|
||||
throw new Error(
|
||||
`Signed relative switch target does not implement source case ${value}: ${JSON.stringify(instruction)}`,
|
||||
);
|
||||
}
|
||||
return {
|
||||
entry_size: 2,
|
||||
signed: true,
|
||||
selector_bias: selectorBias,
|
||||
dispatch_address: table.dispatch_address,
|
||||
mappings: table.mappings.length,
|
||||
};
|
||||
}
|
||||
|
||||
export async function verifyNativeValueTrace(client, procedures, target) {
|
||||
const entry = requireProcedure(procedures, "rea_ghidra_inventory_entry");
|
||||
const names = await inventoryCall(client, "list_names", {
|
||||
|
||||
@@ -0,0 +1,227 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { mkdir, mkdtemp, readFile, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { promisify } from "node:util";
|
||||
|
||||
import { parseBinaryTarget } from "../dist/application/BinaryTargetResolver.js";
|
||||
import { parseConfig } from "../dist/config/parseConfig.js";
|
||||
import { GhidraClient } from "../dist/ghidra/GhidraClient.js";
|
||||
import { inspectGhidraInstallation } from "../dist/ghidra/GhidraInstallation.js";
|
||||
import { SUPPORTED_GHIDRA_VERSION } from "../dist/ghidra/GhidraInstallationPolicy.js";
|
||||
import { GhidraProvider } from "../dist/ghidra/GhidraProvider.js";
|
||||
import { silentLogger } from "../dist/logger.js";
|
||||
import { cleanupOwnedProcessGroup } from "../dist/process/ProcessOwnership.js";
|
||||
import {
|
||||
ProviderProcessSupervisor,
|
||||
spawnOwnedProviderProcess,
|
||||
} from "../dist/process/ProviderProcess.js";
|
||||
import { completeVerifierRun, createVerifierRun } from "./lib/verifier-run.mjs";
|
||||
|
||||
const exec = promisify(execFile);
|
||||
const run = createVerifierRun();
|
||||
const procedureName = (name) =>
|
||||
process.platform === "darwin" ? `_${name}` : name;
|
||||
assert.ok(
|
||||
process.platform === "linux" || process.platform === "darwin",
|
||||
"This regression lane requires Linux or macOS",
|
||||
);
|
||||
assert.ok(
|
||||
process.arch === "x64" || process.arch === "arm64",
|
||||
"This regression lane requires x86-64 or arm64",
|
||||
);
|
||||
const config = parseConfig(process.env);
|
||||
if (!config.ok) throw config.error;
|
||||
const installation = inspectGhidraInstallation({
|
||||
environment: process.env,
|
||||
installDir: config.value.ghidraInstallDir,
|
||||
javaHome: config.value.ghidraJavaHome,
|
||||
});
|
||||
assert.equal(installation.status, "available", JSON.stringify(installation));
|
||||
assert.equal(installation.providerVersion, SUPPORTED_GHIDRA_VERSION);
|
||||
const fixtures = fileURLToPath(
|
||||
new URL("../tests/conformance/ghidra", import.meta.url),
|
||||
);
|
||||
const bridge = fileURLToPath(new URL("../bridge/ghidra", import.meta.url));
|
||||
const workspace = await mkdtemp(join(tmpdir(), "rea-ghidra-noreturn-"));
|
||||
let cleaned = true;
|
||||
let report;
|
||||
const digest = (bytes) => createHash("sha256").update(bytes).digest("hex");
|
||||
try {
|
||||
const targetPath = join(
|
||||
workspace,
|
||||
process.platform === "darwin" ? "noreturn.dylib" : "noreturn.so",
|
||||
);
|
||||
const targetOptions =
|
||||
process.platform === "darwin"
|
||||
? [
|
||||
"-dynamiclib",
|
||||
"-undefined",
|
||||
"dynamic_lookup",
|
||||
"-U_FORTIFY_SOURCE",
|
||||
"-D_FORTIFY_SOURCE=0",
|
||||
]
|
||||
: ["-shared"];
|
||||
await exec(process.env.REA_CC ?? "cc", [
|
||||
...targetOptions,
|
||||
"-fPIC",
|
||||
"-O0",
|
||||
"-fno-builtin",
|
||||
join(fixtures, "no-return.c"),
|
||||
"-o",
|
||||
targetPath,
|
||||
]);
|
||||
const targetDigest = digest(await readFile(targetPath));
|
||||
const sourceDigests = Object.fromEntries(
|
||||
await Promise.all(
|
||||
["ReaGhidraBridge.java", "ReaGhidraNoReturnFix.java"].map(
|
||||
async (name) => [name, digest(await readFile(join(bridge, name)))],
|
||||
),
|
||||
),
|
||||
);
|
||||
const project = join(workspace, "project");
|
||||
await mkdir(project);
|
||||
const launch = await spawnOwnedProviderProcess({
|
||||
command: installation.analyzeHeadlessPath,
|
||||
arguments: [
|
||||
project,
|
||||
"NoReturn",
|
||||
"-import",
|
||||
targetPath,
|
||||
"-readOnly",
|
||||
"-deleteProject",
|
||||
"-scriptPath",
|
||||
`${fixtures};${bridge}`,
|
||||
"-postScript",
|
||||
join(fixtures, "ReaNoReturnProbe.java"),
|
||||
bridge,
|
||||
],
|
||||
runId: run.run_id,
|
||||
expectedCommand: null,
|
||||
});
|
||||
const supervisor = new ProviderProcessSupervisor({
|
||||
...launch,
|
||||
ownsProcessLifetime: true,
|
||||
cleanup: () => cleanupOwnedProcessGroup(launch.ownership),
|
||||
});
|
||||
let probe;
|
||||
try {
|
||||
assert.ok(
|
||||
await supervisor.waitForExit(120000),
|
||||
"No-return probe timed out",
|
||||
);
|
||||
assert.ok(
|
||||
await supervisor.waitForOutputClose(1000),
|
||||
"No-return probe output did not close",
|
||||
);
|
||||
const snapshot = supervisor.snapshot();
|
||||
const output = `${snapshot.stdout.text}\n${snapshot.stderr.text}`;
|
||||
assert.equal(snapshot.exitCode, 0, output);
|
||||
const reports = [
|
||||
...output.matchAll(/REA_NO_RETURN_PROBE_JSON (\{[^\r\n]*\})/gu),
|
||||
];
|
||||
assert.equal(reports.length, 1, output);
|
||||
probe = JSON.parse(reports[0][1]);
|
||||
assert.equal(probe.status, "passed");
|
||||
assert.equal(probe.ghidra_version, SUPPORTED_GHIDRA_VERSION);
|
||||
assert.equal(probe.target_sha256, targetDigest);
|
||||
for (const [name, expected] of Object.entries(sourceDigests))
|
||||
assert.equal(probe[name], expected);
|
||||
assert.ok(probe.checks.includes("independent_entry_preserved"));
|
||||
assert.ok(probe.checks.includes("propagated_flag_repaired_rea_tls_outer"));
|
||||
assert.ok(probe.checks.includes("recovered_pseudocode"));
|
||||
} finally {
|
||||
const stopped = await supervisor.stop();
|
||||
cleaned = stopped.status === "verified-cleanup";
|
||||
assert.ok(
|
||||
cleaned,
|
||||
`Owned cleanup incomplete; retained ${workspace}: ${JSON.stringify(stopped)}`,
|
||||
);
|
||||
}
|
||||
|
||||
// Exercise the actual Java server, authenticated transport, request queue and Evidence
|
||||
// projection. This read-only TCP lane also runs where AF_UNIX is unavailable; it does
|
||||
// not claim Windows authority or validation of the default Linux Unix-socket transport.
|
||||
const provider = new GhidraProvider(
|
||||
config.value,
|
||||
silentLogger,
|
||||
process.env,
|
||||
undefined,
|
||||
(options) =>
|
||||
new GhidraClient({ ...options, transport: "authenticated-loopback-tcp" }),
|
||||
);
|
||||
const target = await parseBinaryTarget(targetPath);
|
||||
if (!target.ok) throw target.error;
|
||||
const profile = await provider.resolveAnalysisProfile(target.value);
|
||||
if (!profile.ok) throw profile.error;
|
||||
assert.ok(profile.value.profile);
|
||||
assert.equal(
|
||||
profile.value.profile.parameters.no_return_repair,
|
||||
"returning-imports-decoded-return-v2",
|
||||
);
|
||||
const client = provider.createClient(target.value, profile.value.profile);
|
||||
cleaned = false;
|
||||
const observations = [];
|
||||
const failures = [];
|
||||
try {
|
||||
for (const operation of [
|
||||
"procedure_pseudo_code",
|
||||
"procedure_assembly",
|
||||
"procedure_info",
|
||||
"read_function_instructions",
|
||||
"analyze_function",
|
||||
]) {
|
||||
const result = await client.execute(operation, {
|
||||
procedure: procedureName("rea_abort"),
|
||||
});
|
||||
if (!result.ok) throw result.error;
|
||||
const warnings = result.value.limitations.filter((value) =>
|
||||
/terminal call/u.test(value),
|
||||
);
|
||||
assert.ok(
|
||||
warnings.some(
|
||||
(value) => /abort/u.test(value) && /fallthrough/u.test(value),
|
||||
),
|
||||
operation,
|
||||
);
|
||||
observations.push({ operation, warnings });
|
||||
}
|
||||
const healthy = await client.execute("procedure_pseudo_code", {
|
||||
procedure: procedureName("rea_tls"),
|
||||
});
|
||||
if (!healthy.ok) throw healthy.error;
|
||||
assert.ok(
|
||||
!healthy.value.limitations.some((value) => /terminal call/u.test(value)),
|
||||
);
|
||||
} catch (cause) {
|
||||
failures.push(cause);
|
||||
}
|
||||
const closed = await client.close();
|
||||
cleaned = closed.ok;
|
||||
if (!closed.ok) failures.push(closed.error);
|
||||
if (failures.length > 0)
|
||||
throw new AggregateError(
|
||||
failures,
|
||||
"No-return Evidence verification failed",
|
||||
);
|
||||
assert.equal(digest(await readFile(targetPath)), targetDigest);
|
||||
report = {
|
||||
status: "passed",
|
||||
probe,
|
||||
observations,
|
||||
scope:
|
||||
"Real Ghidra ELF or Mach-O analysis with injected flags; real read-only TCP Evidence projection. No target execution; exact Amethyst artifact and Windows are unverified.",
|
||||
};
|
||||
} finally {
|
||||
if (cleaned) await rm(workspace, { recursive: true, force: true });
|
||||
}
|
||||
console.log(
|
||||
JSON.stringify({
|
||||
...report,
|
||||
cleanup: "complete",
|
||||
verifier_run: await completeVerifierRun(run),
|
||||
}),
|
||||
);
|
||||
@@ -322,7 +322,7 @@ async function functionOperation(operation, parameters) {
|
||||
const result = parseGhidraFunctionResult(operation, called.value);
|
||||
if (!result.ok) throw result.error;
|
||||
observed.add(operation);
|
||||
return result.value;
|
||||
return result.value.value;
|
||||
}
|
||||
|
||||
function assertSession(session, targetSha256, profileDigest) {
|
||||
|
||||
@@ -38,6 +38,7 @@ import {
|
||||
verifyJumpTableDenseSwitch,
|
||||
verifyNativeTypeLayout,
|
||||
verifyRelativeSwitch,
|
||||
verifyRelativeSignedSwitch,
|
||||
verifyNativeValueTrace,
|
||||
} from "./verify-real-ghidra-function.mjs";
|
||||
const exec = promisify(execFile);
|
||||
@@ -173,6 +174,36 @@ try {
|
||||
]);
|
||||
}
|
||||
}
|
||||
const signedRelativeSource = fileURLToPath(
|
||||
new URL(
|
||||
"../tests/conformance/ghidra/relative-switch-signed.S",
|
||||
import.meta.url,
|
||||
),
|
||||
);
|
||||
for (const selectorBias of [0, 2, 10]) {
|
||||
for (const format of [
|
||||
"elf",
|
||||
...(expectedNativeTarget.architecture === "arm64" ? ["mach-o"] : []),
|
||||
]) {
|
||||
const targetPath = join(
|
||||
fixtureRoot,
|
||||
`relative-signed-${selectorBias}-${format}.o`,
|
||||
);
|
||||
await exec(clang, [
|
||||
...(format === "elf" ? ["--target=aarch64-linux-gnu"] : []),
|
||||
`-DREA_SELECTOR_BIAS=${selectorBias}`,
|
||||
"-c",
|
||||
signedRelativeSource,
|
||||
"-o",
|
||||
targetPath,
|
||||
]);
|
||||
crossTargets.push([
|
||||
targetPath,
|
||||
`relative-signed-${selectorBias}`,
|
||||
{ format, architecture: "arm64" },
|
||||
]);
|
||||
}
|
||||
}
|
||||
|
||||
await exec(clang, [
|
||||
"--target=aarch64-linux-gnu",
|
||||
@@ -522,25 +553,30 @@ async function verifyTarget(targetPath, variant, expectedTarget = null) {
|
||||
client,
|
||||
})
|
||||
: null;
|
||||
const probes = variant.startsWith("relative-")
|
||||
? await verifyRelativeSwitch(
|
||||
const probes = variant.startsWith("relative-signed-")
|
||||
? await verifyRelativeSignedSwitch(
|
||||
client,
|
||||
procedures,
|
||||
Number(variant.slice(-1)),
|
||||
Number(variant.split("-").at(-1)),
|
||||
)
|
||||
: variant === "type-layout"
|
||||
? await verifyNativeTypeLayout(client, names)
|
||||
: variant === "jump-table"
|
||||
? await verifyJumpTableDenseSwitch(client, procedures)
|
||||
: variant === "custom" || variant === "aarch64-jump-table"
|
||||
? null
|
||||
: await verifyInventoryOperations({
|
||||
client,
|
||||
variant,
|
||||
procedures,
|
||||
names,
|
||||
strings,
|
||||
});
|
||||
: variant.startsWith("relative-")
|
||||
? await verifyRelativeSwitch(
|
||||
client,
|
||||
procedures,
|
||||
Number(variant.slice(-1)),
|
||||
)
|
||||
: variant === "type-layout"
|
||||
? await verifyNativeTypeLayout(client, names)
|
||||
: variant === "jump-table"
|
||||
? await verifyJumpTableDenseSwitch(client, procedures)
|
||||
: variant === "custom" || variant === "aarch64-jump-table"
|
||||
? null
|
||||
: await verifyInventoryOperations({
|
||||
client,
|
||||
variant,
|
||||
procedures,
|
||||
names,
|
||||
strings,
|
||||
});
|
||||
const nativeValues =
|
||||
variant === "debug"
|
||||
? await verifyNativeValueTrace(client, procedures, parsedTarget.value)
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFile } from "node:child_process";
|
||||
import { access, mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { access, writeFile } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
import { promisify } from "node:util";
|
||||
|
||||
@@ -25,8 +24,6 @@ const electronPath = join(fixtureRoot, "electron");
|
||||
const environment = {
|
||||
...process.env,
|
||||
};
|
||||
const temporaryRoot = await mkdtemp(join(tmpdir(), "rea-readiness-"));
|
||||
|
||||
await access(nativePath);
|
||||
const nativeEvidence = await runCli(["inspect-artifact", nativePath, "--json"]);
|
||||
const javascriptCliEvidence = await runCli([
|
||||
@@ -135,7 +132,6 @@ try {
|
||||
);
|
||||
} finally {
|
||||
await client.close();
|
||||
await rm(temporaryRoot, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
async function runCli(arguments_) {
|
||||
|
||||
+221
-8
@@ -1,6 +1,8 @@
|
||||
"""Check website references, search metadata and the sole production publisher."""
|
||||
|
||||
from collections import Counter
|
||||
from html.parser import HTMLParser
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import struct
|
||||
@@ -10,7 +12,21 @@ from urllib.parse import unquote, urlsplit
|
||||
import xml.etree.ElementTree as ET
|
||||
from zipfile import BadZipFile, ZipFile
|
||||
|
||||
from website import NOTES_FIXTURES, SITE_ORIGIN, authored_markdown, page_url, sync_pages
|
||||
from website import (
|
||||
LANGUAGES,
|
||||
NOTES_FIXTURES,
|
||||
SITE_ORIGIN,
|
||||
SiteState,
|
||||
authored_markdown,
|
||||
code_block,
|
||||
copy_targets,
|
||||
page_language,
|
||||
page_url,
|
||||
source_digest,
|
||||
sync_pages,
|
||||
synced_code_ids,
|
||||
translation_record,
|
||||
)
|
||||
|
||||
LEGACY_ORIGIN = "morluto.github.io"
|
||||
SITE_TEXT_URL = re.compile(
|
||||
@@ -167,8 +183,9 @@ def check_search_metadata(site, pages):
|
||||
title = page.titles[0].strip() if len(page.titles) == 1 else ""
|
||||
if not title:
|
||||
errors.append(f"{label}: needs exactly one nonempty head title")
|
||||
if page.lang != "en" or page.h1_count != 1:
|
||||
errors.append(f"{label}: needs lang=en and exactly one h1")
|
||||
tag = LANGUAGES[page_language(site, path)[0]].tag
|
||||
if page.lang != tag or page.h1_count != 1:
|
||||
errors.append(f"{label}: needs lang={tag} and exactly one h1")
|
||||
if label in NOTES_FIXTURES and not page.noindex():
|
||||
errors.append(
|
||||
f"{label}: example application must have a head noindex directive"
|
||||
@@ -280,15 +297,21 @@ def check_search_metadata(site, pages):
|
||||
return len(indexable), errors
|
||||
|
||||
|
||||
def check_site(site):
|
||||
"""Check assets, HTML fragment destinations and SVG syntax."""
|
||||
def parse_pages(site):
|
||||
"""Parse every HTML page once for reference, metadata and translation checks."""
|
||||
pages = {}
|
||||
errors = []
|
||||
references = 0
|
||||
for path in sorted(site.rglob("*.html")):
|
||||
parser = HtmlReferences()
|
||||
parser.feed(path.read_text(encoding="utf-8"))
|
||||
pages[path] = parser
|
||||
return pages
|
||||
|
||||
|
||||
def check_site(site):
|
||||
"""Check assets, HTML fragment destinations and SVG syntax."""
|
||||
pages = parse_pages(site)
|
||||
errors = []
|
||||
references = 0
|
||||
if site / "index.html" not in pages:
|
||||
errors.append("website/public/index.html is missing")
|
||||
|
||||
@@ -346,6 +369,190 @@ def check_site(site):
|
||||
return len(pages), references, indexable, errors
|
||||
|
||||
|
||||
# Han, kana and CJK punctuation; a space between two of them renders as a gap.
|
||||
CJK_GAP = re.compile(
|
||||
r"([、-〿-ヿ㐀-䶿一-鿿豈-!-])"
|
||||
r"\s+([、-〿-ヿ㐀-䶿一-鿿豈-!-])"
|
||||
)
|
||||
CODE_TAGS = {"code", "kbd", "pre", "script", "style"}
|
||||
|
||||
|
||||
class TranslationText(HTMLParser):
|
||||
"""Collect prose, inline code and message attributes for comparing translations."""
|
||||
|
||||
def __init__(self, synced):
|
||||
super().__init__()
|
||||
self.synced = set(synced)
|
||||
self.stack = []
|
||||
self.prose = []
|
||||
self.inline_code = Counter()
|
||||
self.messages = Counter()
|
||||
|
||||
def handle_starttag(self, tag, attrs):
|
||||
attributes = dict(attrs)
|
||||
self.messages.update(name for name in attributes if name.startswith("data-msg-"))
|
||||
if tag in CODE_TAGS:
|
||||
self.stack.append([tag, attributes.get("id"), ""])
|
||||
|
||||
def handle_startendtag(self, tag, attrs):
|
||||
attributes = dict(attrs)
|
||||
self.messages.update(name for name in attributes if name.startswith("data-msg-"))
|
||||
|
||||
def handle_endtag(self, tag):
|
||||
if self.stack and self.stack[-1][0] == tag:
|
||||
name, identifier, text = self.stack.pop()
|
||||
if name in ("code", "kbd") and identifier not in self.synced:
|
||||
self.inline_code[" ".join(text.split())] += 1
|
||||
if self.stack:
|
||||
self.stack[-1][2] += text
|
||||
|
||||
def handle_data(self, data):
|
||||
if self.stack:
|
||||
self.stack[-1][2] += data
|
||||
else:
|
||||
self.prose.append(data)
|
||||
|
||||
|
||||
def translation_warning(path, root, message):
|
||||
"""Return a warning tied to a repository file."""
|
||||
return (path.relative_to(root).as_posix(), message)
|
||||
|
||||
|
||||
def check_translations(root, pages):
|
||||
"""Check translation records, copied code, structure parity and CJK spacing.
|
||||
|
||||
A changed English source is a warning: its translation keeps working, shows a
|
||||
notice after sync and is updated later. Errors cover what sync or the
|
||||
translator controls directly.
|
||||
"""
|
||||
site = root / "website/public"
|
||||
errors = []
|
||||
warnings = []
|
||||
try:
|
||||
state = SiteState(root)
|
||||
except ValueError as error:
|
||||
return [str(error)], []
|
||||
for path, page in pages.items():
|
||||
language, english = page_language(site, path)
|
||||
if language == "en":
|
||||
continue
|
||||
label = path.relative_to(site).as_posix()
|
||||
source = path.read_text(encoding="utf-8")
|
||||
record = translation_record(source)
|
||||
english_path = site / english
|
||||
if not english_path.is_file():
|
||||
errors.append(f"{label}: translates {english}, which does not exist")
|
||||
continue
|
||||
english_source = english_path.read_text(encoding="utf-8")
|
||||
if record is None:
|
||||
errors.append(f"{label}: needs one <!-- website:translation source-sha256=... status=... --> record")
|
||||
continue
|
||||
status = record.get("status")
|
||||
if status not in ("draft", "reviewed"):
|
||||
errors.append(f"{label}: translation status must be draft or reviewed")
|
||||
if status == "draft" and not page.noindex():
|
||||
errors.append(f"{label}: a draft translation needs <meta name=\"robots\" content=\"noindex\" />")
|
||||
if status == "reviewed":
|
||||
if page.noindex():
|
||||
errors.append(f"{label}: remove noindex from a reviewed translation")
|
||||
if not re.fullmatch(r"\d{4}-\d{2}-\d{2}", record.get("reviewed", "")):
|
||||
errors.append(f"{label}: a reviewed translation needs reviewed=\"YYYY-MM-DD\"")
|
||||
fresh = record.get("source-sha256") == source_digest(english_source)
|
||||
if not fresh:
|
||||
warnings.append(translation_warning(
|
||||
path, root,
|
||||
f"The English page {english} has changed since this translation. Update the translation, "
|
||||
"then set source-sha256 to the new value from python3 scripts/website.py translations.",
|
||||
))
|
||||
|
||||
synced = synced_code_ids(english_source)
|
||||
for identifier in synced:
|
||||
expected = code_block(identifier).search(english_source).group(2)
|
||||
found = code_block(identifier).findall(source)
|
||||
if len(found) != 1 or found[0][1] != expected:
|
||||
errors.append(f"{label}: code block {identifier} must match the English page; run sync")
|
||||
|
||||
english_page = pages[english_path]
|
||||
differences = []
|
||||
if page.ids != english_page.ids:
|
||||
differences.append(
|
||||
f"element IDs differ (missing {sorted(english_page.ids - page.ids)}, extra {sorted(page.ids - english_page.ids)})"
|
||||
)
|
||||
if copy_targets(source) != copy_targets(english_source):
|
||||
differences.append("copy buttons differ from the English page")
|
||||
translated = TranslationText(synced)
|
||||
translated.feed(source)
|
||||
original = TranslationText(synced)
|
||||
original.feed(english_source)
|
||||
if translated.messages != original.messages:
|
||||
differences.append("data-msg-* message attributes differ from the English page")
|
||||
for difference in differences:
|
||||
if fresh:
|
||||
errors.append(f"{label}: {difference}")
|
||||
else:
|
||||
warnings.append(translation_warning(path, root, difference))
|
||||
if fresh and translated.inline_code != original.inline_code:
|
||||
changed = sorted((translated.inline_code - original.inline_code) + (original.inline_code - translated.inline_code))
|
||||
warnings.append(translation_warning(
|
||||
path, root, f"inline code differs from the English page: {', '.join(repr(text) for text in changed[:5])}",
|
||||
))
|
||||
|
||||
if language in ("ja", "zh-hans", "zh-hant"):
|
||||
for text in translated.prose:
|
||||
for match in CJK_GAP.finditer(text):
|
||||
errors.append(
|
||||
f"{label}: whitespace between {match.group(1)!r} and {match.group(2)!r} renders as a gap; "
|
||||
"keep CJK text on one line without spaces"
|
||||
)
|
||||
|
||||
for reference in page.references:
|
||||
url = urlsplit(reference)
|
||||
if url.scheme or url.netloc or not url.path or reference.startswith("/"):
|
||||
continue
|
||||
target = (path.parent / unquote(url.path)).resolve()
|
||||
if target.is_dir():
|
||||
target /= "index.html"
|
||||
if not target.is_relative_to(site.resolve()):
|
||||
continue
|
||||
target_language, target_english = page_language(site.resolve(), target)
|
||||
if (
|
||||
target_language == "en"
|
||||
and target_english != english
|
||||
and state.translations.get((language, target_english)) == "reviewed"
|
||||
):
|
||||
warnings.append(translation_warning(
|
||||
path, root, f"links to the English {target_english}; link to its reviewed translation instead",
|
||||
))
|
||||
|
||||
english_strings = state.locales["en"]["strings"]
|
||||
for code in LANGUAGES:
|
||||
if code == "en" or not any(key[0] == code for key in state.translations):
|
||||
continue
|
||||
locale = state.locales[code]
|
||||
same = sorted(
|
||||
key for key, value in locale["strings"].items()
|
||||
if value == english_strings[key] and key not in locale.get("sameAsEnglish", [])
|
||||
)
|
||||
if same:
|
||||
warnings.append((f"website/locales/{code}.json", f"UI strings still in English: {', '.join(same)}"))
|
||||
return errors, warnings
|
||||
|
||||
|
||||
def report_warnings(warnings):
|
||||
"""Print warnings, as GitHub annotations and a job summary when running in Actions."""
|
||||
for file, message in warnings:
|
||||
print(f"Warning: {file}: {message}")
|
||||
if os.environ.get("GITHUB_ACTIONS") != "true" or not warnings:
|
||||
return
|
||||
for file, message in warnings:
|
||||
print(f"::warning file={file}::{message}")
|
||||
summary = os.environ.get("GITHUB_STEP_SUMMARY")
|
||||
if summary:
|
||||
with open(summary, "a", encoding="utf-8") as output:
|
||||
output.write("### Website warnings\n\n")
|
||||
output.writelines(f"- `{file}`: {message}\n" for file, message in warnings)
|
||||
|
||||
|
||||
def check_publisher(root):
|
||||
"""Detect a second Pages action that could replace the website."""
|
||||
expected = root / ".github/workflows/website-pages.yml"
|
||||
@@ -404,10 +611,16 @@ def main():
|
||||
errors.extend(check_example_archive(root / "website/public"))
|
||||
errors.extend(check_publisher(root))
|
||||
errors.extend(check_repository_links(root))
|
||||
translation_errors, warnings = check_translations(root, parse_pages(root / "website/public"))
|
||||
errors.extend(translation_errors)
|
||||
try:
|
||||
sync_pages(root, check=True)
|
||||
for path in sync_pages(root, check=True):
|
||||
warnings.append(translation_warning(
|
||||
path, root, "the out-of-date notice needs refreshing; run python3 scripts/website.py sync",
|
||||
))
|
||||
except (OSError, ValueError, KeyError) as error:
|
||||
errors.append(str(error))
|
||||
report_warnings(warnings)
|
||||
if errors:
|
||||
print("Website checks failed:\n" + "\n".join(errors), file=sys.stderr)
|
||||
return 1
|
||||
|
||||
@@ -6,6 +6,7 @@ import { createHash } from "node:crypto";
|
||||
import { once } from "node:events";
|
||||
import {
|
||||
access,
|
||||
copyFile,
|
||||
mkdir,
|
||||
mkdtemp,
|
||||
readFile,
|
||||
@@ -16,7 +17,7 @@ import {
|
||||
open,
|
||||
} from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { dirname, join, parse, resolve } from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { verifyWindowsProtocolInput } from "./lib/windows-protocol-input.mjs";
|
||||
|
||||
@@ -32,6 +33,9 @@ const native = requireWindowsNativeAuthority();
|
||||
const { WindowsPrivateRuntime, windowsPrivateRuntime } = await import(
|
||||
pathToFileURL(join(packageRoot, "dist/windows/WindowsPrivateRuntime.js"))
|
||||
);
|
||||
const { ghidraSessionRoot } = await import(
|
||||
pathToFileURL(join(packageRoot, "dist/ghidra/GhidraSessionRoot.js"))
|
||||
);
|
||||
const workspace = await mkdtemp(join(tmpdir(), "rea-native-conformance-"));
|
||||
const environment = Object.entries(process.env)
|
||||
.filter(([, value]) => value !== undefined)
|
||||
@@ -77,6 +81,31 @@ const wait = async (child) => {
|
||||
};
|
||||
const report = { ok: false, artifact: "verified", controls: {}, processes: {} };
|
||||
try {
|
||||
// Exercise the loader in a process whose image is not named node.exe. The
|
||||
// ordinary Windows lane otherwise cannot detect eager Node-API imports.
|
||||
const renamedHost = join(workspace, "rea-renamed-node.exe");
|
||||
const renamedHostFixture = join(workspace, "renamed-host.mjs");
|
||||
const loaderUrl = pathToFileURL(
|
||||
join(packageRoot, "dist/windows/WindowsNativeLoader.js"),
|
||||
).href;
|
||||
await copyFile(process.execPath, renamedHost);
|
||||
await writeFile(
|
||||
renamedHostFixture,
|
||||
`import {basename} from 'node:path';
|
||||
import {requireWindowsNativeAuthority} from ${JSON.stringify(loaderUrl)};
|
||||
const {inspection}=requireWindowsNativeAuthority();
|
||||
process.stdout.write(JSON.stringify({executable:basename(process.execPath),abiVersion:inspection.abiVersion}));\n`,
|
||||
);
|
||||
const renamedHostResult = execFileSync(renamedHost, [renamedHostFixture], {
|
||||
encoding: "utf8",
|
||||
timeout: 20_000,
|
||||
windowsHide: true,
|
||||
});
|
||||
assert.deepEqual(JSON.parse(renamedHostResult), {
|
||||
executable: "rea-renamed-node.exe",
|
||||
abiVersion: 1,
|
||||
});
|
||||
report.controls.renamedHostNodeApi = true;
|
||||
const sourceDirectory = join(workspace, "source");
|
||||
await mkdir(sourceDirectory);
|
||||
const source = join(sourceDirectory, "target.bin");
|
||||
@@ -110,6 +139,47 @@ try {
|
||||
])
|
||||
assert.throws(() => native.call("open", [rejected]));
|
||||
report.controls.ordinaryDriveSeparatorsAndRequestedIdentity = true;
|
||||
const driveRoot = parse(workspace).root;
|
||||
const runtimeParent = ghidraSessionRoot({
|
||||
base: join(driveRoot, ".Tmp", "scratch"),
|
||||
});
|
||||
assert.equal(runtimeParent, driveRoot);
|
||||
const driveRootRuntime = native.call("runtime_create", [
|
||||
runtimeParent,
|
||||
"rea-drive-root-",
|
||||
]);
|
||||
try {
|
||||
assert.equal(driveRootRuntime.path.startsWith(`${driveRoot}\\`), false);
|
||||
assert.equal(dirname(driveRootRuntime.path), driveRoot);
|
||||
const snapshot = await native.call("runtime_snapshot", [
|
||||
driveRootRuntime.handle,
|
||||
source,
|
||||
"probe.bin",
|
||||
]);
|
||||
const sourceBytes = await readFile(source);
|
||||
assert.equal(
|
||||
snapshot.sha256,
|
||||
createHash("sha256").update(sourceBytes).digest("hex"),
|
||||
);
|
||||
assert.equal(snapshot.source.requestedPath, source);
|
||||
const readback = native.call("runtime_open", [
|
||||
driveRootRuntime.handle,
|
||||
"probe.bin",
|
||||
]);
|
||||
try {
|
||||
assert.equal(readback.size, sourceBytes.byteLength);
|
||||
assert.deepEqual(
|
||||
native.call("read", [readback.handle, 0, 65536]),
|
||||
sourceBytes.subarray(0, 65536),
|
||||
);
|
||||
} finally {
|
||||
native.call("close", [readback.handle]);
|
||||
}
|
||||
} finally {
|
||||
native.call("runtime_close", [driveRootRuntime.handle]);
|
||||
}
|
||||
assert.equal(await exists(driveRootRuntime.path), false);
|
||||
report.controls.driveRootRuntimeParent = true;
|
||||
const snapshotOwner = WindowsPrivateRuntime.create(
|
||||
workspace.replace(/\\/u, "/"),
|
||||
"single-flight-",
|
||||
|
||||
@@ -26,22 +26,49 @@ export async function verifyElectronToolContracts({
|
||||
await mkdtemp(join(tmpdir(), "rea-electron-e2e-")),
|
||||
);
|
||||
const main = join(root, "main.cjs");
|
||||
const preload = join(root, "preload.cjs");
|
||||
const ready = join(root, "ready");
|
||||
const entrypoint = join(repositoryRoot, "scripts", "rea.mjs");
|
||||
const runId = randomUUID();
|
||||
await writeFile(
|
||||
main,
|
||||
`const {app,BrowserWindow}=require("electron");
|
||||
app.whenReady().then(()=>{const window=new BrowserWindow();
|
||||
window.loadFile("renderer.html");require("node:fs").writeFileSync(${JSON.stringify(ready)},"ready");});`,
|
||||
`const electron = require("electron");
|
||||
const path = require("node:path");
|
||||
const { app, BrowserWindow, ipcMain } = electron;
|
||||
const Window = BrowserWindow;
|
||||
const mainIpc = ipcMain;
|
||||
app.whenReady().then(() => {
|
||||
const window = new Window({
|
||||
webPreferences: { preload: path.join(__dirname, "preload.cjs") },
|
||||
});
|
||||
window.loadFile("renderer.html");
|
||||
require("node:fs").writeFileSync(${JSON.stringify(ready)}, "ready");
|
||||
});
|
||||
mainIpc.handle("fixture:echo", (_event, value) => ({ value }));`,
|
||||
);
|
||||
await writeFile(
|
||||
join(root, "renderer.html"),
|
||||
'<!doctype html><button id="run">Run</button><script src="renderer.js"></script>',
|
||||
`<!doctype html>
|
||||
<button id="run">Run</button>
|
||||
<output id="result"></output>
|
||||
<script src="renderer.js"></script>`,
|
||||
);
|
||||
await writeFile(
|
||||
preload,
|
||||
`const electron = require("electron");
|
||||
const { contextBridge, ipcRenderer } = electron;
|
||||
const bridge = contextBridge;
|
||||
const rendererIpc = ipcRenderer;
|
||||
bridge.exposeInMainWorld("fixture", {
|
||||
echo: (value) => rendererIpc.invoke("fixture:echo", value),
|
||||
});`,
|
||||
);
|
||||
await writeFile(
|
||||
join(root, "renderer.js"),
|
||||
'document.querySelector("#run").addEventListener("click",()=>console.log("fixture-click"));',
|
||||
`document.querySelector("#run").addEventListener("click", async () => {
|
||||
const result = await window.fixture.echo("renderer");
|
||||
document.querySelector("#result").textContent = result.value;
|
||||
});`,
|
||||
);
|
||||
const transport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
@@ -135,6 +162,25 @@ window.loadFile("renderer.html");require("node:fs").writeFileSync(${JSON.stringi
|
||||
const staticAnalysis = await call("analyze_javascript_application", {
|
||||
input_path: root,
|
||||
});
|
||||
assert.equal(staticAnalysis.normalized_result.summary.browser_windows, 1);
|
||||
assert.equal(
|
||||
staticAnalysis.normalized_result.summary.preload_entrypoints,
|
||||
1,
|
||||
);
|
||||
assert.equal(
|
||||
staticAnalysis.normalized_result.summary.context_bridge_apis,
|
||||
1,
|
||||
);
|
||||
assert.equal(staticAnalysis.normalized_result.summary.ipc.main_handlers, 1);
|
||||
assert.equal(
|
||||
staticAnalysis.normalized_result.summary.ipc.renderer_transmissions,
|
||||
1,
|
||||
);
|
||||
assert.equal(
|
||||
staticAnalysis.normalized_result.summary.ipc
|
||||
.paired_renderer_transmissions,
|
||||
1,
|
||||
);
|
||||
const reconciled = await call("reconcile_javascript_runtime", {
|
||||
static_layers: [
|
||||
{
|
||||
@@ -169,7 +215,7 @@ window.loadFile("renderer.html");require("node:fs").writeFileSync(${JSON.stringi
|
||||
executable_path: executable,
|
||||
application_path: main,
|
||||
application_root: root,
|
||||
actions: [{ step_id: "settle", kind: "wait", duration_ms: 50 }],
|
||||
actions: [{ step_id: "ipc", kind: "click", selector: "#run" }],
|
||||
};
|
||||
const active = await client.callTool(
|
||||
{ name: "capture_electron_scenario", arguments: input },
|
||||
@@ -192,6 +238,13 @@ window.loadFile("renderer.html");require("node:fs").writeFileSync(${JSON.stringi
|
||||
assert.equal(capture.application.cleanup, "terminated-owned-process");
|
||||
}
|
||||
assert.equal(capture.actions[0].status, "completed");
|
||||
assert.ok(
|
||||
capture.ipc.events.some(
|
||||
({ kind, channel }) =>
|
||||
kind === "main-handler-invocation" && channel === "fixture:echo",
|
||||
),
|
||||
JSON.stringify(capture.ipc),
|
||||
);
|
||||
assert.equal(
|
||||
capture.coverage.status,
|
||||
"partial_attach",
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { isAbsolute, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import {
|
||||
spawnOwnedProviderProcess,
|
||||
ProviderProcessSupervisor,
|
||||
} from "../../../dist/process/ProviderProcess.js";
|
||||
import { cleanupOwnedProcessGroup } from "../../../dist/process/ProcessOwnership.js";
|
||||
import { createVerifierRun } from "../../lib/verifier-run.mjs";
|
||||
|
||||
const installation = process.env.GHIDRA_INSTALL_DIR;
|
||||
assert.ok(
|
||||
installation && isAbsolute(installation),
|
||||
"Set GHIDRA_INSTALL_DIR to a local Ghidra installation",
|
||||
);
|
||||
assert.notEqual(
|
||||
process.platform,
|
||||
"win32",
|
||||
"This direct headless regression requires a POSIX host; Windows uses its controlled provider lane",
|
||||
);
|
||||
const workspace = await mkdtemp(join(tmpdir(), "rea-ghidra-seed-regression-"));
|
||||
const run = createVerifierRun();
|
||||
let supervisor;
|
||||
try {
|
||||
const target = join(workspace, "seed-functions.bin");
|
||||
const seeds = join(workspace, "seeds.tsv");
|
||||
// Falling through from the lower entry reaches the higher entry. Creating
|
||||
// the higher function first is what keeps the two function bodies separate.
|
||||
await writeFile(target, Buffer.from([0x90, 0x90, 0xc3]));
|
||||
await writeFile(
|
||||
seeds,
|
||||
[
|
||||
"0x0\tfunction\tseed_low",
|
||||
"0x100000000\tfunction\tout_of_range",
|
||||
"0x1\tfunction\tseed_high",
|
||||
"0x1000\tlabel\tseed_global",
|
||||
"0x1000\tcode",
|
||||
"",
|
||||
].join("\n"),
|
||||
);
|
||||
const launch = await spawnOwnedProviderProcess({
|
||||
command: join(installation, "support", "analyzeHeadless"),
|
||||
arguments: [
|
||||
workspace,
|
||||
"seed-regression",
|
||||
"-import",
|
||||
target,
|
||||
"-loader",
|
||||
"BinaryLoader",
|
||||
"-processor",
|
||||
"x86:LE:32:default",
|
||||
"-noanalysis",
|
||||
"-readOnly",
|
||||
"-deleteProject",
|
||||
"-scriptPath",
|
||||
[
|
||||
fileURLToPath(new URL("../../../bridge/ghidra", import.meta.url)),
|
||||
fileURLToPath(
|
||||
new URL("../../../tests/conformance/ghidra", import.meta.url),
|
||||
),
|
||||
].join(";"),
|
||||
"-postScript",
|
||||
"ReaSeedRegressionProbe.java",
|
||||
seeds,
|
||||
],
|
||||
runId: run.run_id,
|
||||
expectedCommand: null,
|
||||
});
|
||||
supervisor = new ProviderProcessSupervisor({
|
||||
...launch,
|
||||
ownsProcessLifetime: true,
|
||||
cleanup: () => cleanupOwnedProcessGroup(launch.ownership),
|
||||
});
|
||||
assert.ok(
|
||||
await supervisor.waitForExit(180_000),
|
||||
"Ghidra seed regression exceeded its startup deadline",
|
||||
);
|
||||
const snapshot = supervisor.snapshot();
|
||||
const output = `${snapshot.stdout.text}\n${snapshot.stderr.text}`;
|
||||
assert.equal(snapshot.exitCode, 0, output);
|
||||
// Headless can exit zero after a script fails, so require the probe's marker.
|
||||
assert.ok(output.includes("REA_SEED_REGRESSION_OK "), output);
|
||||
} finally {
|
||||
if (supervisor !== undefined) {
|
||||
const stopped = await supervisor.stop();
|
||||
assert.notEqual(stopped.status, "incomplete", JSON.stringify(stopped));
|
||||
supervisor.dispose();
|
||||
}
|
||||
await rm(workspace, { recursive: true, force: true });
|
||||
}
|
||||
console.log(
|
||||
"Real Ghidra seed regression passed: descending function order, out-of-range seed, BSS label, BSS code rejection, and owned-process cleanup.",
|
||||
);
|
||||
@@ -0,0 +1,86 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import assert from "node:assert/strict";
|
||||
import { execFile } from "node:child_process";
|
||||
import { createServer } from "node:http";
|
||||
import { promisify } from "node:util";
|
||||
|
||||
const execute = promisify(execFile);
|
||||
assert.equal(process.platform, "android", "Run this verifier inside Termux");
|
||||
assert.equal(process.env.PLAYWRIGHT_BROWSERS_PATH, undefined);
|
||||
// Import no browser provider here: the public dispatcher must establish its
|
||||
// Android cache fallback before Playwright initializes in each subprocess.
|
||||
const { stdout: doctorOutput } = await execute(
|
||||
process.execPath,
|
||||
["scripts/rea.mjs", "mcp", "doctor", "--format", "json"],
|
||||
{ timeout: 120_000, maxBuffer: 8 * 1024 * 1024 },
|
||||
);
|
||||
assert.equal(JSON.parse(doctorOutput).healthy, true, doctorOutput);
|
||||
const executable = `${process.env.PREFIX}/bin/chromium`;
|
||||
const { stdout: chromiumVersion } = await execute(executable, ["--version"], {
|
||||
timeout: 30_000,
|
||||
});
|
||||
const marker = "rea-termux-browser-smoke";
|
||||
const server = createServer((_request, response) => {
|
||||
response.writeHead(200, { "Content-Type": "text/html; charset=utf-8" });
|
||||
response.end(`<!doctype html><html><body><h1>${marker}</h1></body></html>`);
|
||||
});
|
||||
await new Promise((resolve, reject) => {
|
||||
server.once("error", reject);
|
||||
server.listen(0, "127.0.0.1", resolve);
|
||||
});
|
||||
try {
|
||||
const url = `http://127.0.0.1:${server.address().port}/smoke`;
|
||||
const scenario = {
|
||||
browser: { mode: "launch", executable_path: executable },
|
||||
start_url: { url },
|
||||
actions: [
|
||||
{
|
||||
step_id: "ready",
|
||||
action: "wait_for",
|
||||
locator: { kind: "css", selector: "h1" },
|
||||
state: "visible",
|
||||
},
|
||||
],
|
||||
capture: { at_end: ["url", "dom"] },
|
||||
};
|
||||
const { stdout } = await execute(
|
||||
process.execPath,
|
||||
[
|
||||
"scripts/rea.mjs",
|
||||
"capture-browser-scenario",
|
||||
JSON.stringify(scenario),
|
||||
"--json",
|
||||
],
|
||||
{ timeout: 180_000, maxBuffer: 16 * 1024 * 1024 },
|
||||
);
|
||||
const evidence = JSON.parse(stdout);
|
||||
assert.equal(evidence.error, undefined, stdout);
|
||||
const capture = evidence.normalized_result;
|
||||
assert.ok(
|
||||
capture.steps.every((step) => step.status === "completed"),
|
||||
stdout,
|
||||
);
|
||||
const artifacts = capture.steps.at(-1).artifacts;
|
||||
assert.equal(artifacts.url.state, "captured", stdout);
|
||||
assert.equal(artifacts.url.value.url, url);
|
||||
assert.equal(artifacts.dom.state, "captured", stdout);
|
||||
assert.ok(artifacts.dom.value.text.includes(`<h1>${marker}</h1>`), stdout);
|
||||
assert.equal(capture.browser.cleanup, "terminated-owned-process");
|
||||
console.log(
|
||||
JSON.stringify({
|
||||
platform: process.platform,
|
||||
architecture: process.arch,
|
||||
node: process.version,
|
||||
chromium: chromiumVersion.trim(),
|
||||
mcp_doctor: "healthy",
|
||||
url_capture: "passed",
|
||||
dom_capture: "passed",
|
||||
cleanup: capture.browser.cleanup,
|
||||
}),
|
||||
);
|
||||
} finally {
|
||||
await new Promise((resolve, reject) =>
|
||||
server.close((cause) => (cause ? reject(cause) : resolve())),
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
#!/usr/bin/env bash
|
||||
# Host-side orchestration. Root is used only to stage/read files and send the
|
||||
# intent; TermuxService launches all package/build/browser work as the app UID.
|
||||
set -euo pipefail
|
||||
|
||||
staging="${RUNNER_TEMP:?}/rea-termux"
|
||||
app_home=/data/data/com.termux/files/home
|
||||
prefix=/data/data/com.termux/files/usr
|
||||
mkdir -p "$staging/diagnostics"
|
||||
collect_diagnostics() {
|
||||
adb logcat -d > "$staging/diagnostics/logcat.txt" 2>&1 || true
|
||||
adb exec-out cat "$app_home/rea-ci/run.log" > "$staging/diagnostics/termux.log" 2>&1 || true
|
||||
adb exec-out cat "$app_home/rea-ci/status" > "$staging/diagnostics/status.txt" 2>&1 || true
|
||||
}
|
||||
trap collect_diagnostics EXIT
|
||||
|
||||
adb root
|
||||
adb wait-for-device
|
||||
adb install "$staging/termux-app_v0.118.3+github-debug_x86_64.apk"
|
||||
adb shell am start -n com.termux/.app.TermuxActivity
|
||||
# First launch unpacks the APK's bundled bootstrap. Do not replace it with a
|
||||
# root-installed userland: that would miss Termux's own initialization.
|
||||
deadline=$((SECONDS + 180))
|
||||
until adb shell test -x "$prefix/bin/bash"; do
|
||||
if (( SECONDS >= deadline )); then
|
||||
echo 'Termux bootstrap did not finish within 180 seconds' >&2
|
||||
exit 1
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
|
||||
git archive --format=tar HEAD > "$staging/source.tar"
|
||||
adb push "$staging/source.tar" /data/local/tmp/rea-source.tar
|
||||
adb push scripts/verify/termux/in-app.sh /data/local/tmp/rea-in-app.sh
|
||||
adb shell chmod 644 /data/local/tmp/rea-source.tar /data/local/tmp/rea-in-app.sh
|
||||
app_uid=$(adb shell stat -c %u /data/data/com.termux | tr -d '\r')
|
||||
[[ "$app_uid" =~ ^[0-9]+$ && "$app_uid" != 0 ]]
|
||||
adb shell mkdir -p "$app_home/rea-ci" "$app_home/.termux"
|
||||
adb shell "echo allow-external-apps=true > $app_home/.termux/termux.properties"
|
||||
adb shell cp /data/local/tmp/rea-in-app.sh "$app_home/rea-ci/in-app.sh"
|
||||
adb shell cp /data/local/tmp/rea-source.tar "$app_home/rea-ci/source.tar"
|
||||
adb shell chown -R "$app_uid:$app_uid" "$app_home/rea-ci" "$app_home/.termux"
|
||||
adb shell restorecon -RF "$app_home/rea-ci" "$app_home/.termux"
|
||||
# Reload the external-command policy in a fresh app process before sending it.
|
||||
adb shell am force-stop com.termux
|
||||
adb shell am start -n com.termux/.app.TermuxActivity
|
||||
# Root may send this permission-protected intent; the service executes as the
|
||||
# application, rather than running Node/Chromium through adb shell or su.
|
||||
adb shell am startservice \
|
||||
-n com.termux/.app.RunCommandService \
|
||||
-a com.termux.RUN_COMMAND \
|
||||
--es com.termux.RUN_COMMAND_PATH "$prefix/bin/bash" \
|
||||
--esa com.termux.RUN_COMMAND_ARGUMENTS "-l,$app_home/rea-ci/in-app.sh" \
|
||||
--es com.termux.RUN_COMMAND_WORKDIR "$app_home" \
|
||||
--ez com.termux.RUN_COMMAND_BACKGROUND true
|
||||
|
||||
deadline=$((SECONDS + 2100))
|
||||
until adb shell test -f "$app_home/rea-ci/status"; do
|
||||
if (( SECONDS >= deadline )); then
|
||||
echo 'Termux verification exceeded 35 minutes' >&2
|
||||
exit 1
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
adb exec-out cat "$app_home/rea-ci/run.log"
|
||||
status=$(adb shell cat "$app_home/rea-ci/status" | tr -d '\r')
|
||||
[[ "$status" == 0 ]]
|
||||
@@ -0,0 +1,33 @@
|
||||
#!/data/data/com.termux/files/usr/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
report="$HOME/rea-ci"
|
||||
exec > "$report/run.log" 2>&1
|
||||
# Publish the result only after all commands have settled. The host reads it as
|
||||
# root; no shared-storage permission or helper Android application is needed.
|
||||
trap 'result=$?; printf "%s\n" "$result" > "$report/status"' EXIT
|
||||
set -x
|
||||
[[ "$(id -u)" != 0 ]]
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
export HUSKY=0
|
||||
pkg update -y
|
||||
pkg upgrade -y
|
||||
pkg install -y nodejs-lts npm git x11-repo termux-exec
|
||||
pkg install -y chromium
|
||||
mkdir -p "$report/source"
|
||||
tar -xf "$report/source.tar" -C "$report/source"
|
||||
cd "$report/source"
|
||||
node -e 'if (process.platform !== "android") throw new Error("Expected Android Node.js")'
|
||||
node -e 'const expected = require("node:fs").readFileSync(".nvmrc", "utf8").trim().replace(/^v/, ""); if (process.versions.node !== expected) throw new Error(`Termux Node ${process.versions.node} differs from .nvmrc ${expected}`)'
|
||||
# Android uses Termux's Bionic-linked Node distribution; the desktop .nvmrc
|
||||
# binary cannot run here. Keep npm at the repository's exact pinned version.
|
||||
npm install --prefix "$report/toolchain" --no-package-lock --ignore-scripts \
|
||||
"$(node -p 'JSON.parse(require("node:fs").readFileSync("package.json", "utf8")).packageManager')"
|
||||
export PATH="$report/toolchain/node_modules/.bin:$PATH"
|
||||
node --version
|
||||
npm --version
|
||||
dpkg-query -W nodejs-lts chromium termux-exec
|
||||
npm ci
|
||||
npm run build:termux
|
||||
unset PLAYWRIGHT_BROWSERS_PATH npm_config_playwright_browsers_path npm_package_config_playwright_browsers_path
|
||||
node scripts/verify/termux/browser.mjs
|
||||
+442
-28
@@ -1,26 +1,53 @@
|
||||
"""Maintain shared HTML and run the static website's contributor commands."""
|
||||
|
||||
import argparse
|
||||
from collections import namedtuple
|
||||
from functools import partial
|
||||
from hashlib import sha256
|
||||
from html import escape
|
||||
from html.parser import HTMLParser
|
||||
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
|
||||
import json
|
||||
from pathlib import Path, PurePosixPath
|
||||
import posixpath
|
||||
import re
|
||||
from string import Template
|
||||
import subprocess
|
||||
import sys
|
||||
from textwrap import indent
|
||||
from urllib.parse import quote, urlsplit
|
||||
from urllib.parse import quote, urlsplit, urlunsplit
|
||||
|
||||
|
||||
SITE_ORIGIN = "https://rea.tools"
|
||||
NOTES_FIXTURES = {"examples/notes-electron/index.html", "examples/notes-web/index.html"}
|
||||
# Each label is a key in website/locales/<language>.json.
|
||||
NAVIGATION = (
|
||||
("Showcases", "showcase/"),
|
||||
("Blog", "blog/"),
|
||||
("Get started", "get-started/"),
|
||||
("Guides", "guides/"),
|
||||
("FAQ", "faq/"),
|
||||
("navShowcases", "showcase/"),
|
||||
("navBlog", "blog/"),
|
||||
("navGetStarted", "get-started/"),
|
||||
("navGuides", "guides/"),
|
||||
("navFaq", "faq/"),
|
||||
)
|
||||
Language = namedtuple("Language", "tag name og_locale")
|
||||
# English stays at the root; every other language lives under its directory.
|
||||
# The order is the language menu's order.
|
||||
LANGUAGES = {
|
||||
"en": Language("en", "English", "en_US"),
|
||||
"zh-hans": Language("zh-Hans", "简体中文", "zh_CN"),
|
||||
"zh-hant": Language("zh-Hant", "繁體中文", "zh_TW"),
|
||||
"ja": Language("ja", "日本語", "ja_JP"),
|
||||
"ko": Language("ko", "한국어", "ko_KR"),
|
||||
"de": Language("de", "Deutsch", "de_DE"),
|
||||
"fr": Language("fr", "Français", "fr_FR"),
|
||||
}
|
||||
TRANSLATION_STATUSES = ("draft", "reviewed")
|
||||
TRANSLATION_RECORD = re.compile(r"<!-- website:translation((?:\s+[\w-]+=\"[^\"]*\")*)\s*-->")
|
||||
SHARED_REGION = re.compile(r"<!-- website:([\w-]+):start -->.*?<!-- website:\1:end -->", re.S)
|
||||
NOTICE_START = " <!-- website:translation-notice:start -->\n"
|
||||
NOTICE_END = " <!-- website:translation-notice:end -->"
|
||||
GLOBE_PATH = (
|
||||
"M12 3a9 9 0 1 0 0 18a9 9 0 1 0 0-18Z M3.6 9h16.8 M3.6 15h16.8 "
|
||||
"M12 3c2.4 2.5 3.6 5.5 3.6 9s-1.2 6.5-3.6 9c-2.4-2.5-3.6-5.5-3.6-9s1.2-6.5 3.6-9Z"
|
||||
)
|
||||
|
||||
|
||||
@@ -75,10 +102,14 @@ class PageMetadata(HTMLParser):
|
||||
self.titles[-1] += data
|
||||
|
||||
|
||||
def file_route(relative):
|
||||
"""Map a site-relative HTML file to its public route."""
|
||||
return relative.removesuffix("index.html") if PurePosixPath(relative).name == "index.html" else relative
|
||||
|
||||
|
||||
def page_route(site, path):
|
||||
"""Map an HTML file to its public route without a separate page registry."""
|
||||
route = path.relative_to(site).as_posix()
|
||||
return route.removesuffix("index.html") if path.name == "index.html" else route
|
||||
return file_route(path.relative_to(site).as_posix())
|
||||
|
||||
|
||||
def page_url(site, path):
|
||||
@@ -86,30 +117,227 @@ def page_url(site, path):
|
||||
return f"{SITE_ORIGIN}/{quote(page_route(site, path), safe='/')}"
|
||||
|
||||
|
||||
def shared_blocks(root, path, source):
|
||||
def page_language(site, path):
|
||||
"""Return a page's language directory and the site-relative file of its English source."""
|
||||
relative = path.relative_to(site).as_posix()
|
||||
first, separator, rest = relative.partition("/")
|
||||
if separator and first in LANGUAGES and first != "en":
|
||||
return first, rest
|
||||
return "en", relative
|
||||
|
||||
|
||||
def load_locales(root):
|
||||
"""Read the UI strings for every language and require the same keys as English."""
|
||||
directory = root / "website/locales"
|
||||
locales = {}
|
||||
for code in LANGUAGES:
|
||||
path = directory / f"{code}.json"
|
||||
try:
|
||||
locale = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError) as error:
|
||||
raise ValueError(f"website/locales/{code}.json: cannot read UI strings: {error}") from error
|
||||
locales[code] = locale
|
||||
expected = set(locales["en"]["strings"])
|
||||
for code, locale in locales.items():
|
||||
label = f"website/locales/{code}.json"
|
||||
strings = locale.get("strings")
|
||||
if not isinstance(strings, dict) or set(strings) != expected:
|
||||
missing = sorted(expected - set(strings or {}))
|
||||
extra = sorted(set(strings or {}) - expected)
|
||||
raise ValueError(f"{label}: strings must match en.json (missing {missing}, extra {extra})")
|
||||
if any(not isinstance(value, str) or not value.strip() for value in strings.values()):
|
||||
raise ValueError(f"{label}: every UI string must be nonempty text")
|
||||
stars = locale.get("githubStars")
|
||||
if not isinstance(stars, dict) or "other" not in stars or not set(stars) <= {"zero", "one", "two", "few", "many", "other"}:
|
||||
raise ValueError(f"{label}: githubStars needs an 'other' label and CLDR plural categories only")
|
||||
if any("{count}" not in value for value in stars.values()):
|
||||
raise ValueError(f"{label}: every githubStars label needs a {{count}} placeholder")
|
||||
if "{link}" not in strings["staleNotice"]:
|
||||
raise ValueError(f"{label}: staleNotice needs a {{link}} placeholder")
|
||||
same = locale.get("sameAsEnglish", [])
|
||||
if not isinstance(same, list) or not set(same) <= expected:
|
||||
raise ValueError(f"{label}: sameAsEnglish must list UI string keys")
|
||||
return locales
|
||||
|
||||
|
||||
def translation_record(source):
|
||||
"""Return the attributes of a translation's provenance comment, or None."""
|
||||
records = TRANSLATION_RECORD.findall(source)
|
||||
if len(records) != 1:
|
||||
return None
|
||||
return dict(re.findall(r'([\w-]+)="([^"]*)"', records[0]))
|
||||
|
||||
|
||||
def copy_targets(source):
|
||||
"""Return the IDs referenced by copy buttons, in page order."""
|
||||
return list(dict.fromkeys(re.findall(r'\bdata-copy="([^"]+)"', source)))
|
||||
|
||||
|
||||
def code_block(identifier):
|
||||
"""Match a code element by ID, capturing its opening tag, contents and end tag."""
|
||||
return re.compile(rf'(<code\b[^>]*\bid="{re.escape(identifier)}"[^>]*>)(.*?)(</code>)', re.S)
|
||||
|
||||
|
||||
def synced_code_ids(source):
|
||||
"""Return the copy targets that are code blocks; translations take them from English."""
|
||||
return [identifier for identifier in copy_targets(source) if code_block(identifier).search(source)]
|
||||
|
||||
|
||||
def source_digest(source):
|
||||
"""Hash an English page's translatable content.
|
||||
|
||||
Shared regions and copied code blocks are left out, so template edits and
|
||||
command-only changes do not mark translations out of date.
|
||||
"""
|
||||
content = SHARED_REGION.sub("", source)
|
||||
for identifier in synced_code_ids(source):
|
||||
content = code_block(identifier).sub(r"\1\3", content)
|
||||
return sha256(" ".join(content.split()).encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def sync_code_blocks(english, source, label):
|
||||
"""Copy every synced code block's contents from the English source, matched by ID."""
|
||||
for identifier in synced_code_ids(english):
|
||||
contents = code_block(identifier).search(english).group(2)
|
||||
pattern = code_block(identifier)
|
||||
if len(pattern.findall(source)) != 1:
|
||||
raise ValueError(f"{label}: needs one <code id=\"{identifier}\"> block copied from the English page")
|
||||
source = pattern.sub(lambda match: match.group(1) + contents + match.group(3), source)
|
||||
return source
|
||||
|
||||
|
||||
class SiteState:
|
||||
"""Read the locales and every translation's status once for a sync or check."""
|
||||
|
||||
def __init__(self, root):
|
||||
self.root = root
|
||||
self.site = root / "website/public"
|
||||
self.locales = load_locales(root)
|
||||
self.translations = {}
|
||||
for path in sorted(self.site.rglob("*.html")):
|
||||
language, english = page_language(self.site, path)
|
||||
if language != "en":
|
||||
record = translation_record(path.read_text(encoding="utf-8")) or {}
|
||||
self.translations[(language, english)] = record.get("status")
|
||||
|
||||
def linkable(self, language, english, from_draft=False):
|
||||
"""Report whether other pages may link to this language's version of a page."""
|
||||
if language == "en":
|
||||
return (self.site / english).is_file()
|
||||
status = self.translations.get((language, english))
|
||||
return status == "reviewed" or (from_draft and status == "draft")
|
||||
|
||||
def alternates(self, english):
|
||||
"""Return the languages whose versions of a page join its hreflang cluster."""
|
||||
return [code for code in LANGUAGES if code == "en" or self.translations.get((code, english)) == "reviewed"]
|
||||
|
||||
|
||||
def language_url(language, english):
|
||||
"""Return the canonical URL of one language's version of a page."""
|
||||
prefix = "" if language == "en" else f"{language}/"
|
||||
return f"{SITE_ORIGIN}/{quote(prefix + file_route(english), safe='/')}"
|
||||
|
||||
|
||||
def shared_blocks(root, path, source, state=None):
|
||||
"""Render shared regions while keeping the page's body and metadata authored."""
|
||||
site = root / "website/public"
|
||||
state = state or SiteState(root)
|
||||
site = state.site
|
||||
metadata = PageMetadata()
|
||||
metadata.feed(source)
|
||||
for name, values in (("title", metadata.titles), ("description", metadata.descriptions)):
|
||||
if len(values) != 1 or not values[0].strip():
|
||||
raise ValueError(f"{path.relative_to(site)}: needs one nonempty head {name}")
|
||||
language, english = page_language(site, path)
|
||||
strings = state.locales[language]["strings"]
|
||||
depth = len(path.relative_to(site).parent.parts)
|
||||
base = "../" * depth or "./"
|
||||
route = page_route(site, path)
|
||||
route = file_route(english)
|
||||
draft = language != "en" and state.translations.get((language, english)) == "draft"
|
||||
|
||||
def href(target_language, target):
|
||||
prefix = "" if target_language == "en" else f"{target_language}/"
|
||||
return base + file_route(prefix + target)
|
||||
|
||||
def localized(target):
|
||||
"""Prefer this language's version of a target page, else the English page."""
|
||||
if language != "en" and state.linkable(language, target, from_draft=draft):
|
||||
return language, href(language, target)
|
||||
return "en", href("en", target)
|
||||
|
||||
links = []
|
||||
for label, destination in NAVIGATION:
|
||||
for key, destination in NAVIGATION:
|
||||
target_language, url = localized(destination + "index.html")
|
||||
attributes = ' class="nav-start"' if destination == "get-started/" else ""
|
||||
if route.startswith(destination):
|
||||
attributes += ' aria-current="page"'
|
||||
links.append(f' <a href="{base}{destination}"{attributes}>{label}</a>')
|
||||
badge = ""
|
||||
if target_language != language:
|
||||
attributes += ' hreflang="en"'
|
||||
badge = '<span class="nav-badge" lang="en">EN</span>'
|
||||
links.append(f' <a href="{url}"{attributes}>{escape(strings[key])}{badge}</a>')
|
||||
|
||||
entries = []
|
||||
for code, details in LANGUAGES.items():
|
||||
if code == language or state.linkable(code, english):
|
||||
url = href(code, english)
|
||||
elif state.linkable(code, "index.html"):
|
||||
url = href(code, "index.html")
|
||||
else:
|
||||
continue
|
||||
current = ' aria-current="true"' if code == language else ""
|
||||
entries.append(
|
||||
f' <li><a href="{url}" hreflang="{details.tag}" lang="{details.tag}"{current}>{details.name}</a></li>'
|
||||
)
|
||||
language_menu = ""
|
||||
if len(entries) > 1:
|
||||
language_menu = "\n".join((
|
||||
"",
|
||||
' <details class="language-menu">',
|
||||
" <summary>",
|
||||
' <svg width="18" height="18" viewBox="0 0 24 24" aria-hidden="true" focusable="false">',
|
||||
f' <path fill="none" stroke="currentColor" stroke-width="1.6" d="{GLOBE_PATH}" />',
|
||||
" </svg>",
|
||||
f' <span class="sr-only">{escape(strings["language"])}: </span>'
|
||||
f'<span class="language-current">{LANGUAGES[language].name}</span>',
|
||||
" </summary>",
|
||||
' <ul class="language-list">',
|
||||
*entries,
|
||||
" </ul>",
|
||||
" </details>",
|
||||
))
|
||||
|
||||
members = [] if draft else state.alternates(english)
|
||||
hreflang = ""
|
||||
locale_alternates = ""
|
||||
if len(members) > 1:
|
||||
for code in [*members, "x-default"]:
|
||||
tag = LANGUAGES[code].tag if code != "x-default" else code
|
||||
url = language_url(code if code != "x-default" else "en", english)
|
||||
hreflang += f'\n<!-- prettier-ignore -->\n<link rel="alternate" hreflang="{tag}" href="{escape(url, quote=True)}" />'
|
||||
for code in members:
|
||||
if code != language:
|
||||
locale_alternates += (
|
||||
f'\n<!-- prettier-ignore -->\n<meta property="og:locale:alternate" content="{LANGUAGES[code].og_locale}" />'
|
||||
)
|
||||
|
||||
labels = "".join(
|
||||
f'\n data-label-{category}="{escape(label, quote=True)}"'
|
||||
for category, label in state.locales[language]["githubStars"].items()
|
||||
)
|
||||
values = {
|
||||
"base": base,
|
||||
"brand": localized("index.html")[1],
|
||||
"navigation": "\n" + "\n".join(links) + "\n ",
|
||||
"language_menu": language_menu,
|
||||
"canonical": escape(page_url(site, path), quote=True),
|
||||
"hreflang": hreflang,
|
||||
"og_locale": LANGUAGES[language].og_locale,
|
||||
"og_locale_alternates": locale_alternates,
|
||||
"origin": SITE_ORIGIN,
|
||||
"title": escape(metadata.titles[0].strip(), quote=True),
|
||||
"description": escape(metadata.descriptions[0].strip(), quote=True),
|
||||
"github_star_labels": labels,
|
||||
**{f"t_{key}": escape(value, quote=True) for key, value in strings.items()},
|
||||
}
|
||||
templates = root / "website/templates"
|
||||
return {
|
||||
@@ -118,35 +346,89 @@ def shared_blocks(root, path, source):
|
||||
}
|
||||
|
||||
|
||||
def translation_notice(state, path, english_source, record):
|
||||
"""Render the stale-translation notice, or nothing while the source hash matches."""
|
||||
if record is None or record.get("source-sha256") == source_digest(english_source):
|
||||
return ""
|
||||
language, english = page_language(state.site, path)
|
||||
strings = state.locales[language]["strings"]
|
||||
depth = len(path.relative_to(state.site).parent.parts)
|
||||
link = (
|
||||
f'<a href="{"../" * depth}{file_route(english)}" hreflang="en">'
|
||||
f'{escape(strings["staleNoticeLink"])}</a>'
|
||||
)
|
||||
message = escape(strings["staleNotice"]).replace("{link}", link)
|
||||
return f' <!-- prettier-ignore -->\n <p class="translation-notice" role="note">{message}</p>\n'
|
||||
|
||||
|
||||
def replace_region(source, start, end, block, label, name):
|
||||
"""Replace the contents of one marked region."""
|
||||
if source.count(start) != 1 or source.count(end) != 1:
|
||||
raise ValueError(f"{label}: needs one {name} region; use new-page or translate for a new page")
|
||||
before, _, remaining = source.partition(start)
|
||||
_, found, after = remaining.partition(end)
|
||||
if not found:
|
||||
raise ValueError(f"{label}: {name} region is out of order")
|
||||
return before + start + block + end + after
|
||||
|
||||
|
||||
def region_contents(source, start, end):
|
||||
"""Return the current contents of a marked region."""
|
||||
return source.partition(start)[2].partition(end)[0]
|
||||
|
||||
|
||||
def sync_pages(root, check=False):
|
||||
"""Update shared regions, or reject drift without changing any authored file."""
|
||||
site = root / "website/public"
|
||||
"""Update shared regions, or reject drift without changing any authored file.
|
||||
|
||||
Returns the pages that changed. With check=True, structural drift raises, so
|
||||
the returned pages are translations whose only drift is the stale notice;
|
||||
an out-of-date translation is a warning rather than a failure.
|
||||
"""
|
||||
state = SiteState(root)
|
||||
site = state.site
|
||||
changes = []
|
||||
notices = []
|
||||
for path in sorted(site.rglob("*.html")):
|
||||
if path.relative_to(site).as_posix() in NOTES_FIXTURES:
|
||||
continue
|
||||
label = path.relative_to(site).as_posix()
|
||||
original = path.read_text(encoding="utf-8")
|
||||
source = original
|
||||
for name, block in shared_blocks(root, path, source).items():
|
||||
start = f" <!-- website:{name}:start -->\n"
|
||||
end = f" <!-- website:{name}:end -->"
|
||||
if source.count(start) != 1 or source.count(end) != 1:
|
||||
raise ValueError(f"{path.relative_to(site)}: needs one {name} region; use new-page for a new reading page")
|
||||
before, _, remaining = source.partition(start)
|
||||
_, found, after = remaining.partition(end)
|
||||
if not found:
|
||||
raise ValueError(f"{path.relative_to(site)}: {name} region is out of order")
|
||||
source = before + start + block + end + after
|
||||
language, english = page_language(site, path)
|
||||
english_source = None
|
||||
if language != "en":
|
||||
english_path = site / english
|
||||
if not english_path.is_file():
|
||||
raise ValueError(f"{label}: translates {english}, which does not exist")
|
||||
english_source = english_path.read_text(encoding="utf-8")
|
||||
source = sync_code_blocks(english_source, source, label)
|
||||
for name, block in shared_blocks(root, path, source, state).items():
|
||||
source = replace_region(
|
||||
source, f" <!-- website:{name}:start -->\n", f" <!-- website:{name}:end -->", block, label, name
|
||||
)
|
||||
if language != "en":
|
||||
notice = translation_notice(state, path, english_source, translation_record(source))
|
||||
structural = replace_region(
|
||||
source, NOTICE_START, NOTICE_END, region_contents(original, NOTICE_START, NOTICE_END), label,
|
||||
"translation-notice",
|
||||
)
|
||||
source = replace_region(source, NOTICE_START, NOTICE_END, notice, label, "translation-notice")
|
||||
if structural == original and source != original:
|
||||
notices.append((path, source))
|
||||
continue
|
||||
if source != original:
|
||||
changes.append((path, source))
|
||||
if check and changes:
|
||||
names = ", ".join(str(path.relative_to(site)) for path, _ in changes)
|
||||
raise ValueError(f"Shared HTML is stale: {names}. Run python3 scripts/website.py sync and commit the updated pages.")
|
||||
if not check:
|
||||
for path, source in changes:
|
||||
for path, source in changes + notices:
|
||||
path.write_text(source, encoding="utf-8")
|
||||
print(f"{'Checked' if check else 'Synced'} shared HTML ({len(changes)} pages changed).")
|
||||
return [path for path, _ in changes]
|
||||
if check:
|
||||
print(f"Checked shared HTML ({len(notices)} translation notices to refresh).")
|
||||
else:
|
||||
print(f"Synced shared HTML ({len(changes) + len(notices)} pages changed).")
|
||||
return [path for path, _ in (notices if check else changes + notices)]
|
||||
|
||||
|
||||
def new_page(root, route, title, description):
|
||||
@@ -154,6 +436,8 @@ def new_page(root, route, title, description):
|
||||
parts = PurePosixPath(route)
|
||||
if not route or parts.is_absolute() or ".." in parts.parts or "\\" in route or parts.suffix:
|
||||
raise ValueError("Use a relative directory route, such as blog/my-article.")
|
||||
if parts.parts[0] in LANGUAGES:
|
||||
raise ValueError("Create the English page first, then use translate for another language.")
|
||||
site = root / "website/public"
|
||||
path = site.joinpath(*parts.parts, "index.html")
|
||||
if not path.resolve().is_relative_to(site.resolve()):
|
||||
@@ -174,11 +458,131 @@ def new_page(root, route, title, description):
|
||||
return path
|
||||
|
||||
|
||||
def relocate_references(source, english, language, state):
|
||||
"""Rewrite relative references for the translation's directory.
|
||||
|
||||
Pages that already have a version in this language link to it; everything
|
||||
else keeps pointing at the English page or asset.
|
||||
"""
|
||||
english_directory = posixpath.dirname(english)
|
||||
translation_directory = posixpath.join(language, english_directory)
|
||||
|
||||
def rewrite(match):
|
||||
value = match.group(2)
|
||||
url = urlsplit(value)
|
||||
if not value or url.scheme or url.netloc or value.startswith(("#", "/")):
|
||||
return match.group(0)
|
||||
target = posixpath.normpath(posixpath.join(english_directory, url.path))
|
||||
directory = url.path.endswith("/") or target == "."
|
||||
if directory:
|
||||
page = "index.html" if target == "." else f"{target}/index.html"
|
||||
if (language, page) in state.translations:
|
||||
target = posixpath.normpath(posixpath.join(language, target))
|
||||
relative = posixpath.relpath(target, translation_directory or ".")
|
||||
if directory:
|
||||
relative = "./" if relative == "." else relative + "/"
|
||||
return f'{match.group(1)}="{urlunsplit(("", "", relative, url.query, url.fragment))}"'
|
||||
|
||||
return re.sub(r'\b(href|src)="([^"]*)"', rewrite, source)
|
||||
|
||||
|
||||
def new_translation(root, language, route):
|
||||
"""Create a draft translation of an English page, ready to be translated in place."""
|
||||
if language not in LANGUAGES or language == "en":
|
||||
raise ValueError(f"Choose one of: {', '.join(code for code in LANGUAGES if code != 'en')}.")
|
||||
route = route.strip("/")
|
||||
parts = PurePosixPath(route) if route not in ("", ".") else PurePosixPath()
|
||||
if parts.is_absolute() or ".." in parts.parts or "\\" in route or parts.suffix:
|
||||
raise ValueError("Use the English page's directory route, such as get-started, or / for the home page.")
|
||||
if parts.parts and parts.parts[0] in LANGUAGES:
|
||||
raise ValueError("Name the English route, without a language directory.")
|
||||
english = PurePosixPath(*parts.parts, "index.html").as_posix()
|
||||
state = SiteState(root)
|
||||
site = state.site
|
||||
source_path = site / english
|
||||
if not source_path.is_file() or english in NOTES_FIXTURES:
|
||||
raise ValueError(f"No English page to translate at {english}.")
|
||||
path = site / language / english
|
||||
if path.exists():
|
||||
raise ValueError(f"Translation already exists: {path.relative_to(site)}")
|
||||
source = source_path.read_text(encoding="utf-8")
|
||||
if source.count('<html lang="en">') != 1 or source.count(" <!-- website:sharing:start -->\n") != 1:
|
||||
raise ValueError(f"{english}: needs <html lang=\"en\"> and one sharing region")
|
||||
main = re.search(r"^ {4}<main\b[^>]*>\n", source, re.M)
|
||||
if main is None:
|
||||
raise ValueError(f"{english}: needs a <main> element on its own line")
|
||||
source = source.replace('<html lang="en">', f'<html lang="{LANGUAGES[language].tag}">')
|
||||
source = relocate_references(source, english, language, state)
|
||||
record = (
|
||||
f' <!-- website:translation source-sha256="{source_digest(source_path.read_text(encoding="utf-8"))}" '
|
||||
'status="draft" -->\n'
|
||||
)
|
||||
robots = "" if '<meta name="robots"' in source else ' <meta name="robots" content="noindex" />\n'
|
||||
source = source.replace(" <!-- website:sharing:start -->\n", robots + record + " <!-- website:sharing:start -->\n")
|
||||
main = re.search(r"^ {4}<main\b[^>]*>\n", source, re.M)
|
||||
source = source[: main.end()] + NOTICE_START + NOTICE_END + "\n" + source[main.end():]
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(source, encoding="utf-8")
|
||||
sync_pages(root)
|
||||
print(
|
||||
f"Created {path.relative_to(root)} as a noindex draft. Translate its text in place following "
|
||||
"website/translation.md, then run python3 scripts/website.py check."
|
||||
)
|
||||
return path
|
||||
|
||||
|
||||
def translation_report(root, language=None):
|
||||
"""Describe each translation's review status, freshness and remaining review items."""
|
||||
state = SiteState(root)
|
||||
site = state.site
|
||||
lines = []
|
||||
for (code, english), status in sorted(state.translations.items()):
|
||||
if language and code != language:
|
||||
continue
|
||||
path = site / code / english
|
||||
source = path.read_text(encoding="utf-8")
|
||||
record = translation_record(source) or {}
|
||||
english_path = site / english
|
||||
current = source_digest(english_path.read_text(encoding="utf-8")) if english_path.is_file() else None
|
||||
fresh = record.get("source-sha256") == current
|
||||
reviewed = f" on {record['reviewed']}" if record.get("reviewed") else ""
|
||||
lines.append(f"{code}/{file_route(english) or '(home)'}: {status or 'no record'}{reviewed}, "
|
||||
f"{'up to date' if fresh else 'English source changed'}")
|
||||
if not fresh and current:
|
||||
lines.append(f" after updating the translation, record source-sha256=\"{current}\"")
|
||||
if status != "reviewed":
|
||||
for system, text in re.findall(r'<(?:em|span|strong|kbd)\b[^>]*\bdata-os-ui="([^"]+)"[^>]*>(.*?)</', source, re.S):
|
||||
lines.append(f" confirm on {system}: {' '.join(text.split())}")
|
||||
english_strings = state.locales["en"]["strings"]
|
||||
for code in LANGUAGES:
|
||||
if code == "en" or (language and code != language):
|
||||
continue
|
||||
if not any(key[0] == code for key in state.translations):
|
||||
continue
|
||||
locale = state.locales[code]
|
||||
same = sorted(
|
||||
key for key, value in locale["strings"].items()
|
||||
if value == english_strings[key] and key not in locale.get("sameAsEnglish", [])
|
||||
)
|
||||
if same:
|
||||
lines.append(f"{code}: UI strings still in English: {', '.join(same)}")
|
||||
print("\n".join(lines) or "No translated pages.")
|
||||
return lines
|
||||
|
||||
|
||||
def run_script(root, name, arguments=()):
|
||||
"""Use the active Python environment for the existing preparation and checks."""
|
||||
subprocess.run([sys.executable, str(root / "scripts" / name), *arguments], cwd=root, check=True)
|
||||
|
||||
|
||||
def run_browser_script_tests(root):
|
||||
"""Run the page-script regressions with the repository's Node.js."""
|
||||
try:
|
||||
subprocess.run(["node", str(root / "scripts/test-website-scripts.mjs")], cwd=root, check=True)
|
||||
except FileNotFoundError as error:
|
||||
raise OSError("The website script tests need Node.js; see website/README.md.") from error
|
||||
|
||||
|
||||
def preview_handler(site, prefix):
|
||||
"""Serve one static directory at root or a Pages prefix, preserving redirects."""
|
||||
class Handler(SimpleHTTPRequestHandler):
|
||||
@@ -207,6 +611,11 @@ def main():
|
||||
create.add_argument("route")
|
||||
create.add_argument("--title", required=True)
|
||||
create.add_argument("--description", required=True)
|
||||
translate = commands.add_parser("translate", help="Create a draft translation of an English page")
|
||||
translate.add_argument("language", choices=[code for code in LANGUAGES if code != "en"])
|
||||
translate.add_argument("route", help="The English page's directory route, or / for the home page")
|
||||
report = commands.add_parser("translations", help="List translation status, freshness and review items")
|
||||
report.add_argument("--lang", choices=[code for code in LANGUAGES if code != "en"])
|
||||
serve = commands.add_parser("serve", help="Prepare and preview the static site")
|
||||
serve.add_argument("--port", type=int, default=4173)
|
||||
serve.add_argument("--base-path", choices=("/", "/rea/"), default="/")
|
||||
@@ -215,12 +624,17 @@ def main():
|
||||
try:
|
||||
if args.command == "new-page":
|
||||
new_page(root, args.route, args.title, args.description)
|
||||
elif args.command == "translate":
|
||||
new_translation(root, args.language, args.route)
|
||||
elif args.command == "translations":
|
||||
translation_report(root, args.lang)
|
||||
elif args.command == "sync":
|
||||
sync_pages(root, check=args.check)
|
||||
elif args.command == "check":
|
||||
sync_pages(root, check=True)
|
||||
for name in ("prepare-website.py", "verify-website.py", "test-website.py"):
|
||||
run_script(root, name)
|
||||
run_browser_script_tests(root)
|
||||
elif args.command == "links":
|
||||
run_script(root, "audit-website-links.py", ["--report", args.report] if args.report else [])
|
||||
else:
|
||||
|
||||
+2
-2
@@ -8,13 +8,13 @@
|
||||
"url": "https://github.com/morluto/rea",
|
||||
"source": "github"
|
||||
},
|
||||
"version": "6.3.0",
|
||||
"version": "6.4.0",
|
||||
"packages": [
|
||||
{
|
||||
"registryType": "npm",
|
||||
"registryBaseUrl": "https://registry.npmjs.org",
|
||||
"identifier": "rea-agents",
|
||||
"version": "6.3.0",
|
||||
"version": "6.4.0",
|
||||
"runtimeHint": "npx",
|
||||
"transport": {
|
||||
"type": "stdio"
|
||||
|
||||
@@ -4,7 +4,7 @@ import type {
|
||||
AnalysisOperationPort,
|
||||
} from "./AnalysisProvider.js";
|
||||
import { AnalysisOutputError } from "../domain/analysisErrorCore.js";
|
||||
import { type AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import type { AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import type { JsonValue } from "../domain/jsonValue.js";
|
||||
import { analysisBookmarkSchema } from "../domain/hopperValues.js";
|
||||
import { resolveAnalysisDocument } from "./AnalysisDocument.js";
|
||||
|
||||
@@ -1,17 +1,21 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import { lstat, opendir, readFile, realpath, stat } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { promisify } from "node:util";
|
||||
|
||||
import { parseBinary } from "plist";
|
||||
import { z } from "zod";
|
||||
|
||||
import { preflightPropertyListDecode } from "../artifacts/apple/InterfaceBuilderDecodeBudget.js";
|
||||
import { BinaryTargetError } from "../domain/configurationErrors.js";
|
||||
import { isPathWithinRoot } from "../domain/localPath.js";
|
||||
import { parseXmlPropertyList } from "../domain/propertyListKeys.js";
|
||||
import {
|
||||
omitPrototypeKeys,
|
||||
parseXmlPropertyList,
|
||||
} from "../domain/propertyListKeys.js";
|
||||
import { decodeXmlPlistText } from "../domain/propertyListXmlText.js";
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
|
||||
const execFileAsync = promisify(execFile);
|
||||
/** Decoded representation bound for one binary Info.plist. */
|
||||
const BINARY_PLIST_DECODE_BYTES = 16 * 1024 * 1024;
|
||||
|
||||
/** Where one app bundle layout keeps its Info.plist and program file. */
|
||||
interface BundleLayout {
|
||||
@@ -193,9 +197,8 @@ const resolveLayoutExecutable = async (
|
||||
try {
|
||||
name =
|
||||
plist.subarray(0, 6).toString("ascii") === "bplist"
|
||||
? await (fileSystem.decodeBinaryPlist ?? readBinaryPlistExecutable)(
|
||||
plistPath,
|
||||
)
|
||||
? await (fileSystem.decodeBinaryPlist?.(plistPath) ??
|
||||
parseBinaryPlistExecutable(plist))
|
||||
: parseXmlPlistExecutable(decodeXmlPlistText(plist));
|
||||
} catch (cause: unknown) {
|
||||
return err(
|
||||
@@ -248,6 +251,10 @@ const resolveLayoutExecutable = async (
|
||||
const parseXmlPlistExecutable = (plist: string): string => {
|
||||
// An unrelated `__proto__` entry must not make the bundle unreadable.
|
||||
const { value } = parseXmlPropertyList(plist);
|
||||
return executableName(value);
|
||||
};
|
||||
|
||||
const executableName = (value: unknown): string => {
|
||||
const executable = executableEntrySchema.safeParse(value);
|
||||
if (!executable.success) throw new Error("CFBundleExecutable is missing");
|
||||
return executable.data.CFBundleExecutable;
|
||||
@@ -255,19 +262,13 @@ const parseXmlPlistExecutable = (plist: string): string => {
|
||||
|
||||
const executableEntrySchema = z.looseObject({ CFBundleExecutable: z.string() });
|
||||
|
||||
const readBinaryPlistExecutable = async (plistPath: string): Promise<string> =>
|
||||
// `-n` strips only the newline plutil appends; it requires macOS 12+.
|
||||
(
|
||||
await execFileAsync("/usr/bin/plutil", [
|
||||
"-extract",
|
||||
"CFBundleExecutable",
|
||||
"raw",
|
||||
"-n",
|
||||
"-o",
|
||||
"-",
|
||||
plistPath,
|
||||
])
|
||||
).stdout;
|
||||
/** Decode a binary plist in-process, so no host plist tool is required. */
|
||||
const parseBinaryPlistExecutable = (plist: Buffer): string => {
|
||||
preflightPropertyListDecode(plist, BINARY_PLIST_DECODE_BYTES);
|
||||
// `plist` assigns a `__proto__` key as the prototype; copying drops it, as
|
||||
// the XML path does.
|
||||
return executableName(omitPrototypeKeys(parseBinary(plist)).value);
|
||||
};
|
||||
|
||||
const isSafeExecutableName = (name: string): boolean =>
|
||||
name.length > 0 &&
|
||||
|
||||
@@ -12,7 +12,7 @@ import type {
|
||||
CompareWebScreenshotsInput,
|
||||
} from "../domain/webScreenshot.js";
|
||||
import { AnalysisCapabilityUnavailableError } from "../domain/analysisErrorCore.js";
|
||||
import { type AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import type { AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import type { ExecutionOptions } from "./AnalysisProvider.js";
|
||||
import type { BrowserObservationPort } from "./BrowserObservationPort.js";
|
||||
|
||||
@@ -2,7 +2,7 @@ import type { ExecutionOptions } from "./AnalysisProvider.js";
|
||||
import type { BrowserScenarioCapturePort } from "./BrowserScenarioCapturePort.js";
|
||||
import { createBrowserScenarioEvidence } from "./BrowserScenarioEvidence.js";
|
||||
import { AnalysisCapabilityUnavailableError } from "../domain/analysisErrorCore.js";
|
||||
import { type AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import type { AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import type { Evidence } from "../domain/evidence.js";
|
||||
import type { BrowserScenario } from "../domain/browserScenario.js";
|
||||
import { err, type Result } from "../domain/result.js";
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import type { AnalysisOperation } from "./AnalysisProvider.js";
|
||||
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
|
||||
import { type AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import type { AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import { parseRelatedAddresses } from "../domain/hopperValues.js";
|
||||
import type { JsonValue } from "../domain/jsonValue.js";
|
||||
import { ok, type Result } from "../domain/result.js";
|
||||
|
||||
@@ -1,7 +1,5 @@
|
||||
import type { ExecutableFormatHint } from "../domain/dosCom.js";
|
||||
import type { JsonValue } from "../domain/jsonValue.js";
|
||||
import { EnhancedTools } from "./EnhancedTools.js";
|
||||
import { executeFunctionAnalysisEvidence } from "./FunctionAnalysisEvidence.js";
|
||||
import type { DirectAnalysisDependencies } from "./DirectAnalysisDependencies.js";
|
||||
import type { BinarySession } from "./binary/BinarySession.js";
|
||||
import type { SessionProviderRoute } from "./binary/SessionProviderRouter.js";
|
||||
@@ -18,7 +16,7 @@ import {
|
||||
} from "../contracts/managed/managedToolContracts.js";
|
||||
import { EvidenceIntegrityError } from "../domain/evidenceErrors.js";
|
||||
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
|
||||
import { type AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import type { AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import { access } from "node:fs/promises";
|
||||
import {
|
||||
readAnalysisSnapshot,
|
||||
@@ -138,7 +136,7 @@ export const runManagedProviderExecution = async (
|
||||
signal?: AbortSignal,
|
||||
): Promise<Result<AnalysisExecution, AnalysisError>> =>
|
||||
withProcessCancellation(signal, async (operationSignal) => {
|
||||
const session = dependencies.createManagedBinarySession();
|
||||
const session = await dependencies.createManagedBinarySession();
|
||||
return withSessionCleanup(
|
||||
session,
|
||||
async () => {
|
||||
@@ -194,7 +192,7 @@ const runAnalysis = async (
|
||||
const { logger, signal, snapshotPath } = options;
|
||||
const config = dependencies.readConfiguration();
|
||||
if (!config.ok) return cliError(config.error);
|
||||
const session = dependencies.createBinarySession(config.value, logger);
|
||||
const session = await dependencies.createBinarySession(config.value, logger);
|
||||
return withSessionCleanup(
|
||||
session,
|
||||
async () => {
|
||||
@@ -332,6 +330,8 @@ const executeAnalysisTool = async (input: {
|
||||
}): Promise<{ readonly output: JsonValue; readonly evidence?: Evidence }> => {
|
||||
const { session, tool, signal, evidenceProfile } = input;
|
||||
if (tool === "analyze_function") {
|
||||
const { executeFunctionAnalysisEvidence } =
|
||||
await import("./FunctionAnalysisEvidence.js");
|
||||
const result = await executeFunctionAnalysisEvidence(
|
||||
session,
|
||||
input.arguments,
|
||||
@@ -350,6 +350,7 @@ const executeAnalysisTool = async (input: {
|
||||
tool === "trace_native_values" ||
|
||||
tool === "trace_native_ui_action"
|
||||
) {
|
||||
const { EnhancedTools } = await import("./EnhancedTools.js");
|
||||
const result = await new EnhancedTools(session).execute(
|
||||
tool,
|
||||
input.arguments,
|
||||
@@ -450,7 +451,7 @@ const prepareSnapshot = async (options: {
|
||||
const { path, snapshotPath } = options;
|
||||
if (snapshotPath === undefined || !(await fileExists(snapshotPath)))
|
||||
return ok({});
|
||||
const loaded = await readAnalysisSnapshot(snapshotPath);
|
||||
const loaded = await readAnalysisSnapshot(snapshotPath, options.signal);
|
||||
if (!loaded.ok) return loaded;
|
||||
const target = await parseBinaryTarget(path, {
|
||||
signal: options.signal,
|
||||
|
||||
@@ -10,6 +10,8 @@ export interface DirectAnalysisDependencies {
|
||||
readonly createBinarySession: (
|
||||
config: AppConfig,
|
||||
logger: Logger,
|
||||
) => BinarySession;
|
||||
readonly createManagedBinarySession: () => BinarySession;
|
||||
) => BinarySession | Promise<BinarySession>;
|
||||
readonly createManagedBinarySession: () =>
|
||||
| BinarySession
|
||||
| Promise<BinarySession>;
|
||||
}
|
||||
|
||||
@@ -15,7 +15,7 @@ export const runSessionStatus = async (
|
||||
): Promise<JsonValue> => {
|
||||
const config = dependencies.readConfiguration();
|
||||
if (!config.ok) return { error: projectAnalysisError(config.error) };
|
||||
const session = dependencies.createBinarySession(config.value, logger);
|
||||
const session = await dependencies.createBinarySession(config.value, logger);
|
||||
try {
|
||||
return {
|
||||
...jsonObjectSchema.parse(session.status()),
|
||||
|
||||
@@ -22,7 +22,7 @@ describe("doctor", () => {
|
||||
),
|
||||
).toEqual({
|
||||
node: `Install ${SUPPORTED_NODE_VERSION_PROSE}.`,
|
||||
host: "REA supports macOS 12+, Ubuntu 24.04+, Fedora 41+, Nobara 44+, 64-bit Arch Linux, CachyOS, and the experimental Windows x64 Ghidra P0 boundary.",
|
||||
host: "REA supports macOS 12+, Ubuntu 24.04+, Fedora 41+, Nobara 44+, 64-bit Arch Linux, CachyOS, Omarchy, and the experimental Windows x64 Ghidra P0 boundary.",
|
||||
hopper: "Run rea setup to install Hopper, or set HOPPER_LAUNCHER_PATH.",
|
||||
target: "Supply a readable local app or program path.",
|
||||
});
|
||||
|
||||
@@ -36,8 +36,6 @@ import {
|
||||
type DoctorScopeReport,
|
||||
} from "./DoctorScope.js";
|
||||
|
||||
/** Deep provider IDs whose readiness can be selected explicitly by doctor. */
|
||||
export const DOCTOR_PROVIDER_IDS = ["hopper", "ghidra", "ida"] as const;
|
||||
/** One actionable environment diagnostic. */
|
||||
export interface DoctorCheck {
|
||||
readonly name: string;
|
||||
|
||||
@@ -113,7 +113,7 @@ const hostCheck = async (host: DoctorHost): Promise<DoctorCheck> => {
|
||||
: `${linuxDistribution.id} ${linuxDistribution.versionId ?? "unknown"}`);
|
||||
return check("host", supported, detail, {
|
||||
remediation:
|
||||
"REA supports macOS 12+, Ubuntu 24.04+, Fedora 41+, Nobara 44+, 64-bit Arch Linux, CachyOS, and the experimental Windows x64 Ghidra P0 boundary.",
|
||||
"REA supports macOS 12+, Ubuntu 24.04+, Fedora 41+, Nobara 44+, 64-bit Arch Linux, CachyOS, Omarchy, and the experimental Windows x64 Ghidra P0 boundary.",
|
||||
classification: "unsupported_host",
|
||||
});
|
||||
};
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
/** Deep-analysis providers whose readiness can be selected explicitly by doctor. */
|
||||
export const DOCTOR_PROVIDER_IDS = ["hopper", "ghidra", "ida"] as const;
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user