Compare commits

...
Author SHA1 Message Date
morluto 4c99912e8d chore(main): merge pinned CI checkpoint 2895a18 2026-10-14 03:12:47 +08:00
morluto b3a60b6b03 test(ci): provide proc metadata for Linux hopper fixtures 2026-10-14 03:02:01 +08:00
morlutoandGitHub 2895a18ca9 fix(ci): skip portable report merge when shards are skipped (#2943)
* fix(ci): skip portable report merge when shards are skipped

The portable consumer can still run after an upstream build failure, even though the test-shard matrix is skipped and no Vitest blob artifacts exist. Guard artifact download and both report merge variants on successful shards; retain the required-shard gate so skipped or failed shards remain failures.

This applies the report-consumer condition from ee0256c740c9be18baae6c56d90285df8069f5b4 to current pinned main without changing action pins or required gates.

* fix(ci): preserve failed shard diagnostics

* docs: clarify current revision CI policy
2026-10-14 02:30:44 +08:00
morluto d5b5441a3a fix(hopper): refresh runtime cleanup against main 2026-10-14 02:27:23 +08:00
morlutoandGitHub 7a6defab18 test(reconstruction): declare complete comparison fixture facets (#2964) 2026-10-14 02:17:57 +08:00
morluto 4f83c64da8 fix(hopper): merge current main into runtime cleanup 2026-10-14 02:12:58 +08:00
Morluto 780191226c test(hopper): run Python-named fixture through Node eval 2026-10-14 02:03:25 +08:00
morlutoandGitHub 8fef2aa64e fix(ci): enforce merge verification and unblock superseded runs (#2938)
* fix(ci): repair inherited formatting failures

* fix(ci): release superseded PR concurrency slots

* docs: require current-base checks before maintainer merges

* docs: preserve administrator merge override
2026-10-14 01:52:57 +08:00
morluto 1738989590 test: isolate hopper scanner process fixtures 2026-10-14 01:26:30 +08:00
morluto 3f18439acd fix(types): declare hopper inventory proc root 2026-10-14 01:24:48 +08:00
morluto dd401a2804 test(process): isolate hopper inventory proc root 2026-10-14 01:06:02 +08:00
morluto 2bc898fae2 chore: merge captured main into Hopper runtime cleanup 2026-10-13 21:45:06 +08:00
N0zoM1z0andKaoru0822 b2e743ef24 test(hopper): reap orphan fixtures independently of container init
Keep a test-owned Linux subreaper until token-verified process-group cleanup completes, then wait for each adopted child. Observe from a separate verifier so the orphan remains outside its lineage. Assert disappearance from procfs after reaping.

Reproduced two failures and eight retained zombies without an init reaper. The corrected fourteen process cases pass with no residual children; typecheck and lint pass.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
2026-10-13 12:25:38 +00:00
morluto 51b3418f2b chore: merge upstream CI baseline fixes 2026-10-13 20:13:27 +08:00
N0zoM1z0andKaoru0822 3bb09b8d2d fix(verification): retain command-less runtime observations
Use the kernel command name for processes without argv or an executable link. Preserve uncertainty for command-less Python interpreters and detect Hopper/Xvfb zombies without signaling. Real unreaped-child regressions keep cleanup with their waiting parent.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
2026-10-13 10:26:57 +00:00
N0zoM1z0andKaoru0822 3deda4781d fix(hopper): parse Python helper interpreter options
Resolve the script operand using CPython option boundaries, including combined flags, separate or attached option values, and --. Keep command/module arguments distinct and defer unknown invocation failures rather than declaring them clean.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
2026-10-13 10:26:57 +00:00
N0zoM1z0andKaoru0822 2ac4795df5 test(hopper): cover a reparented Python helper command
Launch a real Python child with the helper filename, wait for readiness before the Node parent exits, bind its procfs arguments, require non-signaling detection, and clean it through the existing run-token/group identity primitive.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
2026-10-13 10:26:57 +00:00
N0zoM1z0andKaoru0822 a588dc6376 fix(hopper): distinguish runtime processes from argument text
Parse procfs argument boundaries rather than searching rendered ps command text. Keep executable identity checks and deferred permission failures, eliminate the extra ps process, and put shared process observations in the serial process-boundary lane.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
2026-10-13 10:26:57 +00:00
11 changed files with 564 additions and 54 deletions
+15 -6
View File
@@ -33,7 +33,7 @@ env:
jobs:
changes:
name: Classify changes
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
timeout-minutes: 5
permissions:
contents: read
@@ -290,14 +290,19 @@ jobs:
- if: needs.changes.outputs.code == 'true' && success()
run: npm ci
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
if: needs.changes.outputs.code == 'true' && success()
if: >-
!cancelled() && needs.changes.outputs.code == 'true' &&
(needs.test-shard.result == 'success' || needs.test-shard.result == 'failure')
with:
path: .vitest/blob
pattern: vitest-report-*
merge-multiple: true
- name: Merge coverage, JUnit, and timing reports
id: merge
if: needs.changes.outputs.code == 'true' && needs.changes.outputs.tier == 'full' && success()
if: >-
!cancelled() && needs.changes.outputs.code == 'true' &&
needs.changes.outputs.tier == 'full' &&
(needs.test-shard.result == 'success' || needs.test-shard.result == 'failure')
run: >-
npm run test:ci:merge --
--coverage.reportsDirectory=coverage
@@ -312,7 +317,10 @@ jobs:
--outputFile.json=test-results.json
- name: Merge fast-suite JUnit and timing reports
id: merge-fast
if: needs.changes.outputs.code == 'true' && needs.changes.outputs.tier == 'fast' && success()
if: >-
!cancelled() && needs.changes.outputs.code == 'true' &&
needs.changes.outputs.tier == 'fast' &&
(needs.test-shard.result == 'success' || needs.test-shard.result == 'failure')
run: >-
npm exec -- vitest --merge-reports
--reporter=default
@@ -777,7 +785,8 @@ jobs:
required:
name: CI required
if: ${{ always() }}
# Report failed dependencies, but release superseded PR concurrency slots.
if: ${{ !cancelled() }}
needs:
[
changes,
@@ -804,7 +813,7 @@ jobs:
nativeaot,
website,
]
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
timeout-minutes: 5
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
+1 -1
View File
@@ -14,7 +14,7 @@ concurrency:
jobs:
pr-title:
name: Conventional PR title
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
timeout-minutes: 2
steps:
- name: Validate title for Release Please
+4
View File
@@ -181,6 +181,10 @@ The separate GitHub Pages host uses only the manual
## Development feedback and PR verification
For routine merges, wait for the required checks to pass for the current PR
revision. Administrators retain an override for deliberate maintainer recovery.
If `main` is broken, prioritize a focused repair before updating affected PRs.
Select checks for the behavior changed. For a source edit:
```bash
+23 -23
View File
@@ -132,30 +132,30 @@ Ajanınız, hedefi incelemek ve ilgili kodu izlemek için MCP üzerinden REA'yı
REA; Node.js 22.x (>=22.19), 24.x (>=24.11) veya 26+ ve npm gerektirir. Ek araçlar ve desteklenen ana bilgisayar sistemleri hedefe bağlıdır:
| Hedef | REA'nın döndürdükleri | Gereksinimler ve rehber |
| --------------------------------------- | ----------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| Yerel makine kodu içeren ikili dosyalar | Sözde kod, assembly kodu, dizeler, semboller, çağrılar ve referanslar | Hopper, Ghidra veya IDA; [yerel kod analizi](https://rea.tools/tr/guides/native/) |
| Çevrimdışı ELF yapısı | Bölümler, segmentler, özgün semboller/yeniden konumlandırmalar ve statik analizden olası güvenlik önlemleri | Linux x64 üzerinde çağıranın sağladığı pwntools; [ikili dosya tanılama](../binary-diagnostics.md) |
| Go ikili dosyaları | Gömülü compiler sürümü, modüller, replacement'lar ve build ayarları | Yerleşik statik ELF/PE/thin Mach-O incelemesi; Go kurulumu gerekmez |
| WASM artefaktı | Doğrulama, bölümler, tam import/export WAT biçimleri ve decode edilmiş WAT | Çağıranın sağladığı WABT 1.0.42; `inspect_wasm_artifact` |
| EVM bayt kodu | Yönlendirme seçicileri, bayt ofsetleri, çıkarımla belirlenen argümanlar ve durum değiştirilebilirliği | Linux x64 üzerinde yerel ham bayt/onaltılık girdi; [çevrimdışı EVM rehberi](../evm-bytecode.md) |
| SQLite veritabanları | Tablolar, sütunlar, index'ler, view'lar, trigger'lar ve seçilen sıradan tablo satırları | Yerel database/WAL snapshot'ı; native SQLite sınırlarıyla Node.js 24.x >=24.15 veya 26+ |
| Kaydedilmiş Linux çökmeleri | Ham note kayıtları, kaydedilen her iş parçacığının yazmaçları/sinyalleri ve isteğe bağlı eşleme adayları | Linux x64 üzerinde çağıranın sağladığı pwntools; isteğe bağlı GDB/pwndbg; [kaydedilmiş çökmeler](../recorded-crashes.md) |
| JavaScript / Electron | Modüller, içe aktarmalar, kaynak haritaları, rotalar, IPC ve yerel eklenti ilişkileri | Node.js ve npm; [uygulama analizi](https://rea.tools/tr/guides/javascript/) |
| Web siteleri | Sayfa yapısı, betikler, ağ gözlemleri ve istenen ekran görüntüleri | Chrome ailesinden bir tarayıcı; [tarayıcı analizi](https://rea.tools/tr/guides/browser/) |
| Kaydedilmiş ağ yakalamaları | İstekler, yanıtlar, erişilebilir yük içerikleri ve kaynak konumları | HAR veya Fiddler Everywhere SAZ; Linux'ta native mitmproxy için mitmdump; [ağ yakalama rehberi](../web-network-captures.md) |
| .NET derlemeleri | Meta veriler, CIL talimatları, bildirilmiş yerel bağımlılıklar ve derleme karşılaştırmaları | Statik inceleme; [yönetilen kod rehberi](../managed-code-analysis.md) |
| Android APK'leri | Manifest bildirimleri, sınıflar, tersine derlenmiş metotlar ve referanslar | Linux/macOS/Windows x64 üzerinde arayüzsüz JADX ve tam bir JDK; [Android rehberi](../android-analysis.md) |
| Android cihazları | Cihazlar, paketler, süreçler, günlükler, dökümler, ekran görüntüleri ve APK/dosya aktarımları | Çağıranın sağladığı adb (emülatör veya cihaz); [ADB rehberi](../adb-device-analysis.md) |
| JEB projeleri | Motor destekli proje birimleri, artefakt özetleri ve tür/metot sözdekodu | Çağıranın başlattığı, MCP sunan bir JEB istemcisi, REA_JEB_MCP_URL ile seçilir; [JEB rehberi](../jeb-analysis.md) |
| Android kaynakları | Çözülmüş manifest, dize tabloları, diller ve sürüm bilgileri | Çağıranın sağladığı Apktool başlatıcısı; [kaynak rehberi](../apktool-resource-analysis.md) |
| Flutter uygulamaları | Dart anlık görüntü karması, ABI başına kitaplık özetleri ve motor derleme kökeni | Yok (statik APK çözümleme); [Flutter rehberi](../flutter-analysis.md) |
| Donanım yazılımı | Bölgeler, çıkarma sonuçları ve yerel kod analizine devirler | Linux üzerinde Binwalk / Unblob; [donanım yazılımı rehberi](../firmware-analysis.md) |
| Kaydedilmiş paketler | Seçilen alanlar, yinelenen değerler, filtreler ve çözümleme kuralları | Linux: TShark 4.x + prlimit; [yakalama kılavuzu](../packet-captures.md) |
| Paket özetleri | Uç noktalar, DNS, TLS SNI, protokol dağılımı ve HTTP istekleri | Çağıranın sağladığı tshark ve capinfos (Linux üzerinde doğrulandı); [pcap rehberi](../pcap-analysis.md) |
| Kaydedilmiş mantık sinyalleri | Kod çözücü açıklamaları, örnek aralıkları ve tanılama | Linux: sigrok-cli 0.7.2 + prlimit; [sinyal kılavuzu](../signal-recordings.md) |
| Hedef | REA'nın döndürdükleri | Gereksinimler ve rehber |
| --------------------------------------- | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| Yerel makine kodu içeren ikili dosyalar | Sözde kod, assembly kodu, dizeler, semboller, çağrılar ve referanslar | Hopper, Ghidra veya IDA; [yerel kod analizi](https://rea.tools/tr/guides/native/) |
| Çevrimdışı ELF yapısı | Bölümler, segmentler, özgün semboller/yeniden konumlandırmalar ve statik analizden olası güvenlik önlemleri | Linux x64 üzerinde çağıranın sağladığı pwntools; [ikili dosya tanılama](../binary-diagnostics.md) |
| Go ikili dosyaları | Gömülü compiler sürümü, modüller, replacement'lar ve build ayarları | Yerleşik statik ELF/PE/thin Mach-O incelemesi; Go kurulumu gerekmez |
| WASM artefaktı | Doğrulama, bölümler, tam import/export WAT biçimleri ve decode edilmiş WAT | Çağıranın sağladığı WABT 1.0.42; `inspect_wasm_artifact` |
| EVM bayt kodu | Yönlendirme seçicileri, bayt ofsetleri, çıkarımla belirlenen argümanlar ve durum değiştirilebilirliği | Linux x64 üzerinde yerel ham bayt/onaltılık girdi; [çevrimdışı EVM rehberi](../evm-bytecode.md) |
| SQLite veritabanları | Tablolar, sütunlar, index'ler, view'lar, trigger'lar ve seçilen sıradan tablo satırları | Yerel database/WAL snapshot'ı; native SQLite sınırlarıyla Node.js 24.x >=24.15 veya 26+ |
| Kaydedilmiş Linux çökmeleri | Ham note kayıtları, kaydedilen her iş parçacığının yazmaçları/sinyalleri ve isteğe bağlı eşleme adayları | Linux x64 üzerinde çağıranın sağladığı pwntools; isteğe bağlı GDB/pwndbg; [kaydedilmiş çökmeler](../recorded-crashes.md) |
| JavaScript / Electron | Modüller, içe aktarmalar, kaynak haritaları, rotalar, IPC ve yerel eklenti ilişkileri | Node.js ve npm; [uygulama analizi](https://rea.tools/tr/guides/javascript/) |
| Web siteleri | Sayfa yapısı, betikler, ağ gözlemleri ve istenen ekran görüntüleri | Chrome ailesinden bir tarayıcı; [tarayıcı analizi](https://rea.tools/tr/guides/browser/) |
| Kaydedilmiş ağ yakalamaları | İstekler, yanıtlar, erişilebilir yük içerikleri ve kaynak konumları | HAR veya Fiddler Everywhere SAZ; Linux'ta native mitmproxy için mitmdump; [ağ yakalama rehberi](../web-network-captures.md) |
| .NET derlemeleri | Meta veriler, CIL talimatları, bildirilmiş yerel bağımlılıklar ve derleme karşılaştırmaları | Statik inceleme; [yönetilen kod rehberi](../managed-code-analysis.md) |
| Android APK'leri | Manifest bildirimleri, sınıflar, tersine derlenmiş metotlar ve referanslar | Linux/macOS/Windows x64 üzerinde arayüzsüz JADX ve tam bir JDK; [Android rehberi](../android-analysis.md) |
| Android cihazları | Cihazlar, paketler, süreçler, günlükler, dökümler, ekran görüntüleri ve APK/dosya aktarımları | Çağıranın sağladığı adb (emülatör veya cihaz); [ADB rehberi](../adb-device-analysis.md) |
| JEB projeleri | Motor destekli proje birimleri, artefakt özetleri ve tür/metot sözdekodu | Çağıranın başlattığı, MCP sunan bir JEB istemcisi, REA_JEB_MCP_URL ile seçilir; [JEB rehberi](../jeb-analysis.md) |
| Android kaynakları | Çözülmüş manifest, dize tabloları, diller ve sürüm bilgileri | Çağıranın sağladığı Apktool başlatıcısı; [kaynak rehberi](../apktool-resource-analysis.md) |
| Flutter uygulamaları | Dart anlık görüntü karması, ABI başına kitaplık özetleri ve motor derleme kökeni | Yok (statik APK çözümleme); [Flutter rehberi](../flutter-analysis.md) |
| Donanım yazılımı | Bölgeler, çıkarma sonuçları ve yerel kod analizine devirler | Linux üzerinde Binwalk / Unblob; [donanım yazılımı rehberi](../firmware-analysis.md) |
| Kaydedilmiş paketler | Seçilen alanlar, yinelenen değerler, filtreler ve çözümleme kuralları | Linux: TShark 4.x + prlimit; [yakalama kılavuzu](../packet-captures.md) |
| Paket özetleri | Uç noktalar, DNS, TLS SNI, protokol dağılımı ve HTTP istekleri | Çağıranın sağladığı tshark ve capinfos (Linux üzerinde doğrulandı); [pcap rehberi](../pcap-analysis.md) |
| Kaydedilmiş mantık sinyalleri | Kod çözücü açıklamaları, örnek aralıkları ve tanılama | Linux: sigrok-cli 0.7.2 + prlimit; [sinyal kılavuzu](../signal-recordings.md) |
| Paketler ve kaynaklar | Dosya envanterleri, özet değerleri, plist'ler, Apple paket yapısı ve çıkarılmış kaynaklar | [Artefakt ve JavaScript rehberi](../javascript-artifact-reconstruction.md), [Apple uygulamaları](../apple-application-analysis.md) |
| Süreç davranışı | Terminal çıktısı, etkileşimler, sonlanma ve dosya sistemi gözlemleri, çalıştırma karşılaştırmaları | Yerel PTY ile Linux/macOS; [süreç yakalama](../process-capture.md) |
| Süreç davranışı | Terminal çıktısı, etkileşimler, sonlanma ve dosya sistemi gözlemleri, çalıştırma karşılaştırmaları | Yerel PTY ile Linux/macOS; [süreç yakalama](../process-capture.md) |
Statik JavaScript ve .NET incelemesi, uygulamayı çalıştırmadan sağlanan dosyaları okur. Çalışma zamanı yakalaması, seçilen hedefi kullanıcı izinlerinizle çalıştırır veya onunla etkileşime girer; her çalışma zamanı rehberi etkilerini açıklar.
+74 -16
View File
@@ -1,8 +1,36 @@
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { readdir, stat } from "node:fs/promises";
import { readFile, readdir, stat } from "node:fs/promises";
import { basename } from "node:path";
const execFileAsync = promisify(execFile);
// CPython's interface option ends interpreter parsing; following values are
// script arguments. -W/-X consume either an attached value or the next operand.
// https://docs.python.org/3.12/using/cmdline.html
const pythonScriptOperand = (arguments_) => {
for (let index = 1; index < arguments_.length; index++) {
const option = arguments_[index];
if (option === "--") return arguments_[index + 1];
if (option === "-" || option === "") return undefined;
if (!option.startsWith("-")) return option;
if (option === "--check-hash-based-pycs") {
index++;
continue;
}
if (/^--(?:help(?:-env|-xoptions|-all)?|version)$/u.test(option))
return undefined;
if (option.startsWith("--"))
throw new Error(`Unknown Python helper invocation option: ${option}`);
for (let character = 1; character < option.length; character++) {
const flag = option[character];
if ("cmh?V".includes(flag)) return undefined;
if (flag === "W" || flag === "X") {
if (character === option.length - 1) index++;
break;
}
if (!"bBdEiIOPqRsSuvx".includes(flag))
throw new Error(`Unknown Python helper invocation option: ${option}`);
}
}
return undefined;
};
/**
* Observe leftover demo processes on a dedicated clean Linux runner, including
@@ -18,9 +46,6 @@ export const assertCleanLinuxHopperRuntime = async (
],
procRoot = "/proc",
) => {
const { stdout } = await execFileAsync("ps", ["-ax", "-o", "command="]);
if (/\/opt\/hopper\/bin\/Hopper|\bXvfb\b|hopper-demo-x11\.py/u.test(stdout))
throw new Error("Clean Linux runner retained a Hopper demo process");
const launchers = [];
for (const path of launcherPaths) {
try {
@@ -29,18 +54,55 @@ export const assertCleanLinuxHopperRuntime = async (
if (error?.code !== "ENOENT") throw error;
}
}
if (launchers.length === 0) return;
// /proc exposes the executed file even when argv[0] names a removed symlink.
// Compare file identity, not a display path or a process name.
let inspectionError;
for (const entry of await readdir(procRoot, { withFileTypes: true })) {
if (!entry.isDirectory() || !/^\d+$/u.test(entry.name)) continue;
let executable;
let retainedRuntime;
try {
executable = await stat(`${procRoot}/${entry.name}/exe`, { bigint: true });
// procfs preserves argument boundaries. A caller merely mentioning a
// runtime in its script text or other arguments is not that runtime.
const commandLine = await readFile(
`${procRoot}/${entry.name}/cmdline`,
"utf8",
);
const arguments_ = commandLine.split("\0");
// The kernel command field identifies renamed runtime processes such
// as Xvfb. Keep it separate from arguments so ordinary script text does
// not count as a retained runtime.
const kernelCommand = (
await readFile(`${procRoot}/${entry.name}/comm`, "utf8")
).replace(/\n$/u, "");
if (
commandLine.length === 0 &&
/^python3(?:\.\d+)?$/u.test(kernelCommand)
)
throw new Error(
"Cannot identify a command-less Python process on the clean Linux runner",
);
retainedRuntime =
kernelCommand === "Hopper" ||
kernelCommand === "Xvfb" ||
arguments_[0] === "/opt/hopper/bin/Hopper" ||
basename(arguments_[0] ?? "") === "Xvfb" ||
(/^python3(?:\.\d+)?$/u.test(basename(arguments_[0] ?? "")) &&
arguments_.some(
(value) => basename(value) === "hopper-demo-x11.py",
) &&
basename(pythonScriptOperand(arguments_) ?? "") ===
"hopper-demo-x11.py");
if (!retainedRuntime && launchers.length > 0) {
const executable = await stat(`${procRoot}/${entry.name}/exe`, {
bigint: true,
});
retainedRuntime = launchers.some(
({ dev, ino }) => dev === executable.dev && ino === executable.ino,
);
}
} catch (error) {
if (error?.code === "ENOENT" || error?.code === "ESRCH") continue;
// Other users' executables may be unreadable. The setup lane launches
// Other users' process facts may be unreadable. The setup lane launches
// providers as the caller; unreadable caller-owned processes fail closed.
if (error?.code === "EACCES") {
const owner = await stat(`${procRoot}/${entry.name}`).catch((cause) => {
@@ -54,11 +116,7 @@ export const assertCleanLinuxHopperRuntime = async (
inspectionError ??= error;
continue;
}
if (
launchers.some(
({ dev, ino }) => dev === executable.dev && ino === executable.ino,
)
)
if (retainedRuntime)
throw new Error("Clean Linux runner retained a Hopper demo process");
}
if (inspectionError !== undefined) throw inspectionError;
+4 -3
View File
@@ -1,3 +1,7 @@
import type { z } from "zod";
import type { analysisStringSchema } from "../../src/domain/hopperValues.js";
type VerifierCall = (
name: string,
arguments_?: Record<string, unknown>,
@@ -25,6 +29,3 @@ export function verifyHopperStringObjects(
}[];
readonly displayEscapesExcludedFromMatching: true;
}>;
import type { z } from "zod";
import type { analysisStringSchema } from "../../src/domain/hopperValues.js";
@@ -44,6 +44,16 @@ const dossier = jsonValueSchema.parse(
referenced_names: [],
native_api: null,
native_value_flow: null,
facet_availability: {
pseudocode: { available: true },
assembly: { available: true },
comments: { available: true },
callers: { available: true },
callees: { available: true },
references: { available: true },
strings_names: { available: true },
cfg: { available: true },
},
basic_blocks: [{ start: "0x1000", end: "0x1001", successors: [] }],
limitations: [],
}),
@@ -365,7 +365,9 @@ it("keeps fast feedback separate from complete qualification and release proof",
expect(ci.concurrency["cancel-in-progress"]).toBe(
"${{ github.event_name == 'pull_request' && inputs.source_sha == '' && !contains(github.event.pull_request.labels.*.name, 'ci:full') }}",
);
expect(ci.jobs.required?.if).toBe("${{ always() }}");
// Failed dependencies still reach the gate, but superseded PR runs must
// release their concurrency slot without waiting for another runner.
expect(ci.jobs.required?.if).toBe("${{ !cancelled() }}");
const shards = ci.jobs["test-shard"]?.steps ?? [];
const fast = shards.find((step) => step.run?.includes("npm run test:run"));
const full = shards.find((step) => step.run?.includes("test:ci:shard:run"));
@@ -45,23 +45,34 @@ const workflow = async () =>
),
).jobs;
it("collects failed-shard diagnostics before enforcing the shard result", async () => {
it("merges available failed-shard diagnostics and skips unavailable shard reports", async () => {
const steps = (await workflow()).test!.steps;
const download = steps.find(
(step) =>
step.uses?.includes("actions/download-artifact") &&
step.with?.pattern === "vitest-report-*",
);
const merge = steps.findIndex((step) => step.run?.includes("test:ci:merge"));
const upload = steps.findIndex((step) => step.with?.name === "test-results");
const gate = steps.findIndex(
(step) => step.env?.SHARD_RESULT && step.run?.includes("test "),
);
expect(merge).toBeGreaterThan(0);
expect(download?.if).toContain("!cancelled()");
expect(download?.if).toContain("needs.test-shard.result == 'success'");
expect(download?.if).toContain("needs.test-shard.result == 'failure'");
expect(upload).toBeGreaterThan(merge);
expect(gate).toBeGreaterThan(upload);
expect(steps[merge]?.if).not.toContain("needs.test-shard.result");
const merges = steps.filter((step) => step.run?.includes("--reporter=json"));
expect(merges).toHaveLength(2);
for (const step of merges)
for (const step of merges) {
expect(step.run).toContain(
"--reporter=./scripts/ci/test-timing-reporter.mjs",
);
expect(step.if).toContain("!cancelled()");
expect(step.if).toContain("needs.test-shard.result == 'success'");
expect(step.if).toContain("needs.test-shard.result == 'failure'");
}
expect(steps[upload]?.with?.path).toContain("test-timings.json");
expect(steps[upload]?.with).toMatchObject({ "include-hidden-files": true });
expect(steps[upload]?.with?.path).toContain(
@@ -0,0 +1,411 @@
import { execFile } from "node:child_process";
import { randomUUID } from "node:crypto";
import {
chmod,
copyFile,
mkdir,
readFile,
symlink,
unlink,
writeFile,
} from "node:fs/promises";
import { join } from "node:path";
import { promisify } from "node:util";
import { expect, it } from "vitest";
import { assertCleanLinuxHopperRuntime } from "../../../scripts/lib/clean-linux-hopper-runtime.mjs";
import { cleanupOwnedProcessGroup } from "../../../src/process/ProcessOwnership.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
const execFileAsync = promisify(execFile);
it.runIf(process.platform === "linux").each([
["Xvfb", "Clean Linux runner retained a Hopper demo process"],
["Hopper", "Clean Linux runner retained a Hopper demo process"],
["python3", "Cannot identify a command-less Python process"],
["unrelated", undefined],
] as const)(
"observes an unreaped %s without signaling or reaping it",
async (name, refusal) => {
const directory = await createTestTempDirectory("rea-runtime-zombie-");
const procRoot = join(directory, "proc");
await mkdir(procRoot);
const executable = join(directory, name);
await copyFile("/usr/bin/true", executable);
await chmod(executable, 0o700);
// WNOWAIT observes exit without reaping. The parent keeps the zombie until
// the test closes its stdin, then waitpid performs the actual cleanup.
const parent = execFileAsync(
"/usr/bin/python3",
[
"-c",
`
import os, sys
pid = os.fork()
if pid == 0:
os.execl(sys.argv[1], sys.argv[1])
try:
os.waitid(os.P_PID, pid, os.WEXITED | os.WNOWAIT)
print(pid, flush=True)
sys.stdin.buffer.read(1)
finally:
os.waitpid(pid, 0)
`,
executable,
],
{ timeout: 10_000 },
);
const ready = new Promise<number>((resolve, reject) => {
parent.child.stdout?.once("data", (data: Buffer) =>
resolve(Number(data.toString().trim())),
);
parent.catch(reject);
});
let zombiePid: number | undefined;
try {
zombiePid = await ready;
expect(Number.isSafeInteger(zombiePid)).toBe(true);
const prefix = `/proc/${String(zombiePid)}`;
const procEntry = join(procRoot, String(zombiePid));
await mkdir(procEntry);
for (const name of ["cmdline", "comm"])
await symlink(`${prefix}/${name}`, join(procEntry, name));
expect(await readFile(`${prefix}/cmdline`, "utf8")).toBe("");
expect((await readFile(`${prefix}/comm`, "utf8")).trim()).toBe(name);
const observation = assertCleanLinuxHopperRuntime([], procRoot);
if (refusal === undefined)
await expect(observation).resolves.toBeUndefined();
else await expect(observation).rejects.toThrow(refusal);
expect(await readFile(`${prefix}/stat`, "utf8")).toContain(
`(${name}) Z `,
);
} finally {
parent.child.stdin?.end("reap");
await parent;
if (zombiePid !== undefined)
await expect(
readFile(`/proc/${String(zombiePid)}/stat`, "utf8"),
).rejects.toMatchObject({ code: "ENOENT" });
}
},
);
it.runIf(process.platform === "linux")(
"does not treat runtime names in another process's arguments as a runtime",
async () => {
const directory = await createTestTempDirectory("rea-runtime-arguments-");
const procRoot = join(directory, "proc");
await mkdir(procRoot);
const observer = new URL(
"../../../scripts/lib/clean-linux-hopper-runtime.mjs",
import.meta.url,
).href;
const code = `
import { mkdir, symlink } from "node:fs/promises";
import { join } from "node:path";
const { assertCleanLinuxHopperRuntime } = await import(${JSON.stringify(observer)});
const entry = join(${JSON.stringify(procRoot)}, String(process.pid));
await mkdir(entry);
await symlink("/proc/" + process.pid + "/cmdline", join(entry, "cmdline"));
await symlink("/proc/" + process.pid + "/comm", join(entry, "comm"));
await assertCleanLinuxHopperRuntime([], ${JSON.stringify(procRoot)});
console.log('clean');
`;
const { stdout } = await execFileAsync(
process.execPath,
[
"--input-type=module",
"-e",
code,
procRoot,
"Xvfb",
"/opt/hopper/bin/Hopper",
"hopper-demo-x11.py",
],
{ timeout: 10_000 },
);
expect(stdout.trim()).toBe("clean");
},
);
it.runIf(process.platform === "linux")(
"distinguishes an ordinary Node script with the helper's filename",
async () => {
const directory = await createTestTempDirectory("rea-runtime-arguments-");
const procRoot = join(directory, "proc");
await mkdir(procRoot);
const script = join(directory, "hopper-demo-x11.py");
const observer = new URL(
"../../../scripts/lib/clean-linux-hopper-runtime.mjs",
import.meta.url,
).href;
const code = `
import { mkdir, symlink } from "node:fs/promises";
import { join } from "node:path";
const entry = join(${JSON.stringify(procRoot)}, String(process.pid));
await mkdir(entry);
await symlink("/proc/" + process.pid + "/cmdline", join(entry, "cmdline"));
await symlink("/proc/" + process.pid + "/comm", join(entry, "comm"));
const { assertCleanLinuxHopperRuntime } = await import(${JSON.stringify(observer)});
await assertCleanLinuxHopperRuntime([], ${JSON.stringify(procRoot)});
console.log('clean');
`;
const { stdout } = await execFileAsync(
process.execPath,
["--input-type=module", "--eval", code, script],
{ timeout: 10_000 },
);
expect(stdout.trim()).toBe("clean");
},
);
const launchReapedOrphan = (runId: string, parentCode: string) => {
// The short-lived Node launcher still exits before observation. A retained
// subreaper adopts its child and waits after owned cleanup, even when the
// container's PID 1 does not reap. It is a sibling of the verifier below.
const reaper = execFileAsync(
"/usr/bin/python3",
[
"-c",
`
import ctypes, json, os, subprocess, sys
libc = ctypes.CDLL(None, use_errno=True)
if libc.prctl(36, 1, 0, 0, 0) != 0: # PR_SET_CHILD_SUBREAPER
raise OSError(ctypes.get_errno(), "PR_SET_CHILD_SUBREAPER")
launcher = subprocess.run(sys.argv[1:], capture_output=True, text=True, check=True)
child = json.loads(launcher.stdout)
try:
print(json.dumps(child), flush=True)
sys.stdin.buffer.read(1)
finally:
os.waitpid(child["pid"], 0)
`,
process.execPath,
"-e",
parentCode,
],
{
env: { ...process.env, REA_PROCESS_RUN_ID: runId },
timeout: 40_000,
},
);
const ready = new Promise<{ pid: number; parentPid: number }>(
(resolve, reject) => {
reaper.child.stdout?.once("data", (data: Buffer) => {
try {
resolve(JSON.parse(data.toString()));
} catch (error) {
reject(error);
}
});
reaper.then(
() => reject(new Error("Fixture reaper exited before readiness")),
reject,
);
},
);
return { reaper, ready };
};
const observeFromSeparateVerifier = async (
runId: string,
launcherPaths: string[],
procRoot: string,
) => {
const observer = new URL(
"../../../scripts/lib/clean-linux-hopper-runtime.mjs",
import.meta.url,
).href;
const verifierRun = new URL(
"../../../scripts/lib/verifier-run.mjs",
import.meta.url,
).href;
const { stdout: observed } = await execFileAsync(
process.execPath,
[
"--input-type=module",
"-e",
`
const { assertCleanLinuxHopperRuntime } = await import(${JSON.stringify(observer)});
const { completeVerifierRun } = await import(${JSON.stringify(verifierRun)});
const { process_lineage } = await completeVerifierRun({
run_id: ${JSON.stringify(runId)}, verifier_pid: process.pid, parent_pid: process.ppid,
});
let refusal;
try { await assertCleanLinuxHopperRuntime(${JSON.stringify(launcherPaths)}, ${JSON.stringify(procRoot)}); }
catch (error) { refusal = error.message; }
console.log(JSON.stringify({ process_lineage, refusal }));
`,
],
{ timeout: 10_000 },
);
return JSON.parse(observed) as {
process_lineage: {
status: string;
descendants: { pid: number }[];
};
refusal?: string;
};
};
it.runIf(process.platform === "linux").each([
["Xvfb-named Node", null, true],
["Python helper", [], true],
["Python helper with -u", ["-u"], true],
[
"Python helper with combined/value options",
["-Bu", "-W", "ignore", "-X", "dev", "--check-hash-based-pycs", "always"],
true,
],
["Python helper after --", ["--"], true],
[
"Python -c script argument",
["-c", "import time; print('ready', flush=True); time.sleep(30)"],
false,
],
["Python -m script argument", ["-m", "unrelated"], false],
["alternate launcher", null, true],
] as const)(
"classifies a reparented %s without signaling it",
async (mode, pythonOptions, retained) => {
const runId = randomUUID();
const childCode =
"process.title = 'Xvfb'; setInterval(() => {}, 1000); process.send('ready');";
const directory = await createTestTempDirectory("rea-clean-runtime-test-");
const procRoot = join(directory, "proc");
await mkdir(procRoot);
const launcher = join(directory, "Hopper");
const alternate = join(directory, "Hopper-alternate");
const helper = join(directory, "hopper-demo-x11.py");
const alternateMode = mode === "alternate launcher";
const pythonMode = pythonOptions !== null;
if (alternateMode) {
await copyFile("/usr/bin/sleep", launcher);
await chmod(launcher, 0o700);
await symlink(launcher, alternate);
}
if (pythonMode) {
await writeFile(
helper,
"import time\nprint('ready', flush=True)\ntime.sleep(30)\n",
);
await copyFile(helper, join(directory, "unrelated.py"));
}
// The fixture must be the cause of a refusal; a retained host runtime
// would otherwise mask a broken classification or executable-identity check.
await assertCleanLinuxHopperRuntime(
alternateMode ? [launcher] : [],
procRoot,
);
const command = alternateMode
? alternate
: pythonMode
? "/usr/bin/python3"
: process.execPath;
const arguments_ = alternateMode
? ["30"]
: pythonMode
? [...pythonOptions, helper]
: ["-e", childCode];
const stdio = alternateMode
? "ignore"
: pythonMode
? ["ignore", "pipe", "ignore"]
: ["ignore", "ignore", "ignore", "ipc"];
const parentCode = `
const { spawn } = require('node:child_process');
const child = spawn(${JSON.stringify(command)}, ${JSON.stringify(arguments_)}, {
detached: true, stdio: ${JSON.stringify(stdio)}, cwd: ${JSON.stringify(directory)}
});
child.once('error', () => process.exit(1));
const ready = () => {
process.stdout.write(JSON.stringify({ pid: child.pid, parentPid: process.pid }));
if (child.connected) child.disconnect();
child.stdout?.destroy();
child.unref();
};
${pythonMode ? "child.stdout.once('data', ready);" : `child.once(${JSON.stringify(alternateMode ? "spawn" : "message")}, ready);`}
`;
const { reaper, ready } = launchReapedOrphan(runId, parentCode);
let orphanPid: number | undefined;
try {
const { pid, parentPid } = await ready;
orphanPid = pid;
const procEntry = join(procRoot, String(pid));
await mkdir(procEntry);
for (const name of ["cmdline", "comm", "exe"])
await symlink(`/proc/${String(pid)}/${name}`, join(procEntry, name));
const { stdout: coordinates } = await execFileAsync("ps", [
"-p",
String(pid),
"-o",
"ppid=,pgid=",
]);
const [currentParent, group] = coordinates
.trim()
.split(/\s+/u)
.map(Number);
expect(currentParent).not.toBe(parentPid);
expect(currentParent).toBe(reaper.child.pid);
expect(group).toBe(pid);
if (alternateMode) {
const commandLine = await readFile(
`/proc/${String(pid)}/cmdline`,
"utf8",
);
expect(commandLine.split("\0")[0]).toBe(alternate);
await unlink(alternate);
}
if (pythonMode) {
const commandLine = await readFile(
`/proc/${String(pid)}/cmdline`,
"utf8",
);
expect(
commandLine.split("\0").slice(0, pythonOptions.length + 2),
).toEqual([command, ...pythonOptions, helper]);
}
const { process_lineage: lineage, refusal } =
await observeFromSeparateVerifier(
runId,
alternateMode ? [launcher] : [],
procRoot,
);
expect(lineage.status).toBe("verified");
expect(lineage.descendants.some((p) => p.pid === pid)).toBe(false);
if (retained)
expect(refusal).toBe(
"Clean Linux runner retained a Hopper demo process",
);
else expect(refusal).toBeUndefined();
// Observation must not terminate the fixture, even though it resembles Xvfb.
const { stdout: alive } = await execFileAsync("ps", [
"-p",
String(pid),
"-o",
"stat=",
]);
expect(alive.trim()).not.toMatch(/^Z/u);
} finally {
try {
if (orphanPid !== undefined) {
const cleanup = await cleanupOwnedProcessGroup({
runId,
leaderPid: orphanPid,
processGroupId: orphanPid,
});
expect(cleanup.cleaned, JSON.stringify(cleanup)).toBe(true);
}
} finally {
reaper.child.stdin?.end("reap");
await reaper;
if (orphanPid !== undefined)
await expect(
readFile(`/proc/${String(orphanPid)}/stat`, "utf8"),
).rejects.toMatchObject({ code: "ENOENT" });
}
}
},
);
@@ -74,7 +74,11 @@ it
if (alternateMode) {
const processEntry = join(procRoot, String(pid));
await mkdir(processEntry);
await symlink(`/proc/${String(pid)}/exe`, join(processEntry, "exe"));
for (const name of ["cmdline", "comm", "exe"])
await symlink(
`/proc/${String(pid)}/${name}`,
join(processEntry, name),
);
}
const { stdout: coordinates } = await execFileAsync("ps", [
"-p",