21aa877e5a
* docs(skills): add CI and test correctness auditing Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> * fix(process): report no remaining time after a provider deadline fires Node timers can fire while performance.now() is still a fraction of a millisecond short of the monotonic deadline. ProviderStartupDeadline then reported interruption "timeout" and an aborted signal while remainingMs() returned ceil(0.x) = 1, so polling loops guarded by remainingMs() > 0 (Hopper and Ghidra startup) could run past an elapsed deadline, and the real-timer test "aborts a bounded wait when its real monotonic deadline expires" failed intermittently in CI (expected 1 to be 0). Once the deadline's own timer has classified a timeout, report zero remaining time. A deterministic regression test fires the timer with the monotonic clock 0.4 ms short of the deadline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(process): retain the monotonic startup deadline budget Recheck the absolute monotonic deadline when an early timer fires, and keep polling waits pending until their own target or the shared deadline arrives. Preserve caller cancellation, disposal and the unchanged real-timer assertion. This replaces the earlier remainingMs workaround: an early callback must not classify an unexpired deadline as a timeout. Adapt the production fix and regression cases from nosmile99 PR #2793 at 546001693636f13700d86c6e93b6086abba2b7d6, including the independently reported both-timers-early reproduction. Preserve the prior contributor's commit. Fixes #2760 Co-authored-by: nosmile99 <147114285+nosmile99@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> * style(tests): format the Java class fixture signature The captured main baseline has an unformatted helper signature. Apply the repository formatter so the required style gate can run to completion. Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> * test(ghidra): group expanded suites by their contracts Split result-envelope and completeness checks, and Windows invocation and runtime checks, into named describe groups within the existing lint limit. Preserve every test, fixture, assertion and platform prerequisite. Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> --------- Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> Co-authored-by: morluto <76467478+morluto@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: nosmile99 <147114285+nosmile99@users.noreply.github.com>
3.8 KiB
Measurement and CI checks
Equivalent completed work
Record the source/workflow SHA, command, test identities and counts, skips, assertions or output contract, exit status, coverage scope and thresholds, toolchain/provider versions, host resources, and prepared artifacts. Inspect both the child result and its report: a profiling wrapper may exit successfully after the measured command failed. Focused tests can pass while global coverage thresholds fail; describe both rather than calling the command green.
Measure wall time, user/system CPU, and peak process-family memory for relevant completed tasks. Parent-only RSS misses workers and their children. On Linux, summed RSS can double-count shared pages; report RSS/PSS semantics, sampling interval, missing samples, and observed cleanup limits. Use equivalent host tools elsewhere. State unknown costs, including agent tokens; fewer log bytes alone prove neither fewer tokens nor faster tests.
Separate cold execution, warm build/file caches, and warm runtime caches. Use repeated or alternating runs when variance affects the decision, with enough observations to distinguish the proposed benefit from noise. Keep workload, instrumentation, and host conditions comparable; record interference. A run with fewer tests, early rejection, lost coverage, or a different runner does not prove suite acceleration. CI-wide claims need end-to-end measurements, not an extrapolation from one improved family.
Workflow, caches, and shared artifacts
Trace each reused result from producer to consumers and identify its lifetime. Check key/input coverage against the actual file inventory and result-affecting configuration, environment, generated data, tool versions, and host state. Exercise an affected input change to prove invalidation; exercise an irrelevant change to support reuse where useful. Cache deterministic preparation and static analysis only when their inputs fully describe the result.
Verify downloaded artifacts match the expected revision, platform, runtime, and schema. Check restore misses, failed/partial producers, stale reports, removed output files, and consumers after producer failures. Shared writers need output ownership or locks; tests need isolated state, not an unnecessary global lock. Worker counts do not bound children spawned within each test. Preserve intentionally concurrent scenarios while bounding unrelated batches.
Check matrix expansion and all job pages before saying CI passed. Inspect
required gates after failed, cancelled, and skipped dependencies. always()
alone does not preserve failures; inspect the aggregation logic. Test/report
uploads should retain available diagnostics after failure, while absent or
malformed required results must fail closed. Do not demand an optional report
from a lane intentionally excluded by the selector.
Runner waits and console noise
Use per-job and per-step timestamps where available. Workflow creation to job start includes dependencies and other delays; it is not necessarily runner queue time. Identify when the job became eligible before attributing waits to runner capacity. If eligibility/queue timestamps are unavailable, report the combined delay and unknown split. Compare equivalent lanes and runner types across enough runs; a single busy runner is not a workflow regression.
For console reductions, retain checker exit codes, actionable errors, useful summary counts, and complete machine-readable artifacts. Verify authored code remains in the actual checker inventory when excluding generated outputs. Introduce an authored error as a control; it must still fail. An ignore pattern can hide real source as well as noise. Show that cached reports are restored, stale successful reports cannot mask current failures, and report parse errors remain failures. Measure readability or byte reduction separately from runtime.