From 2cb076e9b5de5e26b6e864ae85cfb147a3a510e6 Mon Sep 17 00:00:00 2001 From: N0zoM1z0 <161784452+N0zoM1z0@users.noreply.github.com> Date: Tue, 13 Oct 2026 07:33:43 +0000 Subject: [PATCH] docs(skills): add CI and test correctness auditing Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> --- .agents/skills/rea-ci-tests-audit/SKILL.md | 112 ++++++++++++++++++ .../rea-ci-tests-audit/agents/openai.yaml | 4 + .../references/measurement-and-ci.md | 65 ++++++++++ CONTRIBUTING.md | 4 + 4 files changed, 185 insertions(+) create mode 100644 .agents/skills/rea-ci-tests-audit/SKILL.md create mode 100644 .agents/skills/rea-ci-tests-audit/agents/openai.yaml create mode 100644 .agents/skills/rea-ci-tests-audit/references/measurement-and-ci.md diff --git a/.agents/skills/rea-ci-tests-audit/SKILL.md b/.agents/skills/rea-ci-tests-audit/SKILL.md new file mode 100644 index 000000000..4d7b44bf5 --- /dev/null +++ b/.agents/skills/rea-ci-tests-audit/SKILL.md @@ -0,0 +1,112 @@ +--- +name: rea-ci-tests-audit +description: Audit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. +--- + +# REA CI and tests audit + +Establish which checks still catch real regressions and which costs can be +removed without losing that protection. The outcome may be a verified fix, +an evidence-backed recommendation, or an explicit decision to stop. + +Follow the checkout's `AGENTS.md`, [testing guide](../../../docs/testing.md), +and [contribution rules](../../../CONTRIBUTING.md). Use their current commands +and pinned toolchain, rather than commands remembered from an earlier audit. +Preserve the requested scope: investigation alone does not authorize edits; +this skill does not grant permission to publish issues, push, or merge. Existing +user authorization still applies. Keep audit notes and raw results outside the +tracked tree, normally in a task-specific `/tmp` directory. + +## Establish the executed baseline + +Fetch when the user asks for latest main. Pin the baseline SHA and inspected +workflow revision; use an isolated worktree when changing or comparing code. +Do not modify another task's checkout. Record dirty changes and the exact +command, toolchain, host, selected tests, instrumentation, and prerequisites. +An old run is evidence for its executed revision, even if main advances. + +Trace workflow triggers, changed-path selection, job dependencies, reusable +actions, package scripts, test discovery, artifact producers, and report gates +through to the commands that actually execute. Include scripts, generated data, +runtime registration, and subprocess boundaries missing from import graphs. +Check positive and negative selection paths: required files must be discovered +once per intended test configuration; deliberate platform/configuration repeats +are separate coverage. A missing explicit test or missing required report must +not turn green. Optional lanes may skip; preserve their stated prerequisites. + +Use existing timing and failure artifacts to select suspicious families. Run +relevant checks to establish a baseline; reserve a full local suite for broad +CI diagnosis or uncertainty that focused checks cannot resolve. Do not rebuild +or rerun an unchanged complete suite after every documentation edit. + +## Prove the failure or protection + +For each candidate, state the expected observable behavior and its owner, +then construct the smallest reproduction against production behavior. Follow +the producer and affected consumers before deciding where to fix it. + +- For drift, compare fixtures with current admission requirements, protocol + replies, named contracts, tool versions, and actual public entrypoints. A + plausible-looking mock can fail before the behavior its test claims to cover. + Search the whole affected fixture family, including other version strings. + Preserve intentionally malformed fixtures and test their intended rejection. +- For a suspicious green test, perturb the relevant behavior in an isolated + copy or inject a controlled failure. Verify that the existing assertion or + gate detects it. A test may merely repeat its mock, import abandoned code, + or hide a subprocess/checker failure. Restore mutations after the experiment. +- For shared setup or pruning, map distinct assertions to their retained + scenarios. Preserve malformed inputs, cancellation, cleanup failure, + permissions, capacity, identity, and meaningful concurrency interactions. + A faster run that skips a case or weakens its oracle is not an improvement. +- For provider coverage, separate fake-producer, boundary, and actual-engine + evidence. Inspect named skips and prerequisites; fixtures do not establish + real Ghidra, LLDB, browser, or other provider behavior. + +Classify findings as reproduced defects, supported costs, hypotheses, or +unavailable evidence. Record the baseline command, observed output/exit status, +expected result, affected scope, and competing explanation. A failure on main +may be pre-existing; that does not by itself make it irrelevant to a CI audit. +Attribute PR failures using the executed revision and a clean baseline, not +just their resemblance to a previous failure. Retry only to test a specific +transient explanation; retain the original failure and stop when repetition +adds no evidence. A passing retry does not establish a root cause or a fix. + +## Assess efficiency without losing the gate + +Read [measurement and CI checks](references/measurement-and-ci.md) when +investigating slowness, caching, runner waits, shared artifacts, or log noise. +Compare equivalent completed work before claiming a speedup. Distinguish +setup/build time, test execution, coverage instrumentation, dependency waits, +and runner scheduling; state which costs remain unmeasured. + +Prefer removing proven duplicate preparation, reusing immutable setup, and +reducing repeated subprocess work before shrinking correctness coverage. +Preserve independent sessions where lifecycle or concurrency is the oracle. +Keep live tests fresh; a cached previous pass cannot validate current host, +installed provider contents, permissions, or external state. + +## Fix, verify, and hand off within scope + +If implementation is authorized, make the smallest structural change at the +owning boundary and add a regression that fails on the baseline for the right +reason. Retest affected consumers and meaningful negative paths; verify cache +invalidation or discovery when those mechanisms change. Read the installed +Turbo documentation before changing its configuration or commands. + +Run the repository's applicable checks and inspect complete current-head CI +results before an authorized merge. Preserve required coverage and discovery +gates. Do not suppress a failure, add blanket retries, raise timeouts, or delete +distinct assertions solely to make CI green. Changes to these controls need +their own demonstrated reason and retained failure detection. + +Report each confirmed issue with exact revision, reproduction, consequence, +proposed correction, and validation limits. Report performance with workload +and measurements, not just a percentage. Use English issue/PR records when +requested; follow the user's assignment, commit identity, review, and merge +preferences without assuming them for other users. Open ready-for-review PRs +only when that lifecycle is authorized. Resolve necessary review findings and +check final main against the tested tree after merging. + +Stop when the authorized work is verified, remaining benefits are unproven, +or further changes would trade correctness for speculative speed. List genuine +uncertainties without turning them into additional optimization rounds. diff --git a/.agents/skills/rea-ci-tests-audit/agents/openai.yaml b/.agents/skills/rea-ci-tests-audit/agents/openai.yaml new file mode 100644 index 000000000..366dd1229 --- /dev/null +++ b/.agents/skills/rea-ci-tests-audit/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "REA CI and tests audit" + short_description: "Audit CI/test correctness and measured performance" + default_prompt: "Use $rea-ci-tests-audit to investigate CI and test correctness, drift, and performance before proposing changes." diff --git a/.agents/skills/rea-ci-tests-audit/references/measurement-and-ci.md b/.agents/skills/rea-ci-tests-audit/references/measurement-and-ci.md new file mode 100644 index 000000000..e86ba9a4d --- /dev/null +++ b/.agents/skills/rea-ci-tests-audit/references/measurement-and-ci.md @@ -0,0 +1,65 @@ +# Measurement and CI checks + +## Equivalent completed work + +Record the source/workflow SHA, command, test identities and counts, skips, +assertions or output contract, exit status, coverage scope and thresholds, +toolchain/provider versions, host resources, and prepared artifacts. Inspect +both the child result and its report: a profiling wrapper may exit successfully +after the measured command failed. Focused tests can pass while global coverage +thresholds fail; describe both rather than calling the command green. + +Measure wall time, user/system CPU, and peak process-family memory for relevant +completed tasks. Parent-only RSS misses workers and their children. On Linux, +summed RSS can double-count shared pages; report RSS/PSS semantics, sampling +interval, missing samples, and observed cleanup limits. Use equivalent host +tools elsewhere. State unknown costs, including agent tokens; fewer log bytes +alone prove neither fewer tokens nor faster tests. + +Separate cold execution, warm build/file caches, and warm runtime caches. +Use repeated or alternating runs when variance affects the decision, with +enough observations to distinguish the proposed benefit from noise. Keep +workload, instrumentation, and host conditions comparable; record interference. +A run with fewer tests, early rejection, lost coverage, or a different runner +does not prove suite acceleration. CI-wide claims need end-to-end measurements, +not an extrapolation from one improved family. + +## Workflow, caches, and shared artifacts + +Trace each reused result from producer to consumers and identify its lifetime. +Check key/input coverage against the actual file inventory and result-affecting +configuration, environment, generated data, tool versions, and host state. +Exercise an affected input change to prove invalidation; exercise an irrelevant +change to support reuse where useful. Cache deterministic preparation and +static analysis only when their inputs fully describe the result. + +Verify downloaded artifacts match the expected revision, platform, runtime, +and schema. Check restore misses, failed/partial producers, stale reports, +removed output files, and consumers after producer failures. Shared writers +need output ownership or locks; tests need isolated state, not an unnecessary +global lock. Worker counts do not bound children spawned within each test. +Preserve intentionally concurrent scenarios while bounding unrelated batches. + +Check matrix expansion and all job pages before saying CI passed. Inspect +required gates after failed, cancelled, and skipped dependencies. `always()` +alone does not preserve failures; inspect the aggregation logic. Test/report +uploads should retain available diagnostics after failure, while absent or +malformed required results must fail closed. Do not demand an optional report +from a lane intentionally excluded by the selector. + +## Runner waits and console noise + +Use per-job and per-step timestamps where available. Workflow creation to job +start includes dependencies and other delays; it is not necessarily runner +queue time. Identify when the job became eligible before attributing waits to +runner capacity. If eligibility/queue timestamps are unavailable, report the +combined delay and unknown split. Compare equivalent lanes and runner types +across enough runs; a single busy runner is not a workflow regression. + +For console reductions, retain checker exit codes, actionable errors, useful +summary counts, and complete machine-readable artifacts. Verify authored code +remains in the actual checker inventory when excluding generated outputs. +Introduce an authored error as a control; it must still fail. An ignore pattern +can hide real source as well as noise. Show that cached reports are restored, +stale successful reports cannot mask current failures, and report parse errors +remain failures. Measure readability or byte reduction separately from runtime. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a8d5f3635..f32226263 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -187,6 +187,10 @@ CI diagnosis. Packaging/setup/distribution changes additionally require `npm run verify:package` and `npm pack --dry-run`; provider changes require the matching real-provider lane. State which host/provider workflows ran. +For CI/test correctness, drift, or performance investigations, use the +[`rea-ci-tests-audit` contributor skill](.agents/skills/rea-ci-tests-audit/SKILL.md). +It guides reproduction and comparable measurements before changing checks. + `check:pr` reuses build and static-check outputs while executing the test suite fresh, including the fast Python producer checks. `docs:check` also runs the onboarding example consistency suite for README-only changes.