From 78b8e28b67f091bfaf8809ef3aa6498d0f6c4df4 Mon Sep 17 00:00:00 2001 From: N0zoM1z0 <161784452+N0zoM1z0@users.noreply.github.com> Date: Tue, 13 Oct 2026 06:34:01 +0000 Subject: [PATCH] docs(perf): preserve negative small-capture token and timing results Record repeated public SDK/Codex comparisons, real CLI parity, independent raw-frame validation and grading regressions. Keep whole-resource counters and broader capture/platform coverage explicitly unknown; no production change is justified. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> --- experiments/analysis-efficiency/README.md | 4 + .../pcap-small-captures.md | 143 ++++++++++++++++++ 2 files changed, 147 insertions(+) create mode 100644 experiments/analysis-efficiency/pcap-small-captures.md diff --git a/experiments/analysis-efficiency/README.md b/experiments/analysis-efficiency/README.md index 5f656a02c..cdb6c3c9b 100644 --- a/experiments/analysis-efficiency/README.md +++ b/experiments/analysis-efficiency/README.md @@ -19,6 +19,10 @@ Follow-up: [SQLite existing-control measurements](sqlite-existing-controls.md) tests QGIS's shipped GeoPackage through the existing column selection, preserving full selected-table schema and typed cells before considering another API. +Follow-up: [small real PCAP recordings](pcap-small-captures.md) compares existing +complete delivery with summary/pages. Preserve the negative SDK/model input +results: compact captures do not justify another production mechanism. + Follow-up: [native PE/DOS entry views and real model verification](native-entry-views.md) records the post-merge native continuation, including a small-function case where four selected replies cost more bytes than the complete dossier. diff --git a/experiments/analysis-efficiency/pcap-small-captures.md b/experiments/analysis-efficiency/pcap-small-captures.md new file mode 100644 index 000000000..5d9c566c2 --- /dev/null +++ b/experiments/analysis-efficiency/pcap-small-captures.md @@ -0,0 +1,143 @@ +# Small real PCAP recordings: keep complete delivery + +Continues [#2016](https://github.com/morluto/rea/issues/2016) after the QGIS +SQLite round. Measured main is pinned to +`3ae826ceba6f25b08bce50bcb51001209bc16b57`, with 23 intervening commits. +These include bounded PCAP summaries/pages, authenticated artifact subjects, +unknown-snapshot reuse and shared transcript grading. This round measures existing +PCAP delivery choices; it adds no production, API, default, cache or skill change. + +## Corpus and question + +Use two unmodified recordings published on the official +[Wireshark sample-capture page](https://wiki.wireshark.org/SampleCaptures): + +| Recording | Bytes | SHA-256 | +| --------------------------------------- | -----: | ------------------------------------------------------------------ | +| HTTP request/response, `http.cap` | 25,803 | `25a72bdf10339f2c29916920c8b9501d294923108de8f29b19aba7cc001ab60d` | +| HTTP redirects, `http_redirects.pcapng` | 47,804 | `ff432a6b71d06cd20720f7763e584014edd0a43c3b55981c682d1b7640207abc` | + +`corpus.json` preserves original/resolved URLs. Local digests pin downloaded +bytes; no independently published upstream digest or immutable source commit is +claimed. These are upstream saved application-protocol recordings, not generated +REA fixtures. Recorded clients and current deployments are unknown. No new +traffic is captured and no recorded target/URL is executed or fetched. + +The question asks for complete returned identity/client/scalar metadata, every +observed count and source coverage, six retained inventory counts, explicit nulls, +diagnostic UTF-16 lengths, and the first two retained HTTP URIs in original order. +Preserve each used receipt's limitations and identity. Summary/page scope and +unknowns remain distinct from source coverage; exhaustion does not establish +network completeness. Other inventory lists and diagnostic text are unrequested. + +Compare default `{path}` with existing `{path, detail: "summary"}` plus a retained +`http-uris` page, offset 0, limit 2. Ordinary selected-field +`inspect_packet_capture` remains a separate boundary, not a substitute claim +about complete inventories. `questions/pcap-http.txt` contains the fixed question. + +## SDK measurements + +Frozen dist/bridge file-table SHA-256: +`db050545e94dc305a48042e06df96ac6c64b21ee6c5c4707c787d5e46df79e47`. +Linux x64, Node 24.18.0, SDK client/server 2.3.1, installed Wireshark 4.0.17. +Explicit commands are `/usr/bin/tshark`, `/usr/bin/capinfos`, `/usr/bin/prlimit`; +private receipts retain executable hashes. Each workflow starts a fresh server +and uses actual named advertised definitions/schema validation. Host page caches +are not flushed. Both tools read the owned snapshot. + +Eight trials run complete/summary/summary/complete per recording. Both pairs +match selected capture facts, URI prefixes, settings and original source bytes. +Each delivery's exact limitations, identities and coverage remain separately +recorded, rather than equating source coverage with page exhaustion. + +| Trial | Reply bytes | SDK seconds | Payload CPU seconds | Kernel peak bytes | +| -------------------- | ----------: | ----------: | ------------------: | ----------------: | +| HTTP complete-1 | 7,601 | 4.349 | 5.336 | 611,778,560 | +| HTTP summary-1 | 17,540 | 4.972 | 5.973 | 601,661,440 | +| HTTP summary-2 | 17,540 | 5.101 | 6.092 | 611,438,592 | +| HTTP complete-2 | 7,601 | 4.463 | 5.503 | 617,148,416 | +| Redirects complete-1 | 7,535 | 4.670 | 5.641 | 610,627,584 | +| Redirects summary-1 | 17,240 | 5.099 | 6.104 | 615,522,304 | +| Redirects summary-2 | 17,240 | 5.060 | 6.056 | 610,385,920 | +| Redirects complete-2 | 7,535 | 4.466 | 5.458 | 608,141,312 | + +Bytes are compact client-result JSON summed over one complete or two summary/page +calls, not model tokens or raw wire bytes. Summary/page expands replies to more +than twice the complete result on both recordings. All four SDK pairs are slower +and use more payload CPU; memory differences are small/inconsistent. CPU includes +driver/catalog work outside the SDK workflow timer. + +Reproduce with `measure.mjs --artifact wireshark-http.cap` or +`wireshark-http_redirects.pcapng`, pinned path/runtime/revision, explicit provider +JSON, `--schemas selected`, and `--delivery complete|summary`. Use `compare.mjs` +on completed output directories. Original replies, catalogs, comparison receipts +and exact measured driver source stay private. + +## Actual model tasks + +Four actual Codex CLI 0.161.0 tasks use requested `gpt-5.6-terra` high, the same +redirects question/runtime/artifact/provider settings, direct mode, two exact +enabled tools and equal context/discovery capture flags. The requested alias is +not verified server-resolved model identity. Equal cohort instructions select +complete or summary plus one URI page. The experiment driver applies its existing +private per-tool approval policy to `inspect_pcap_capture`; global configuration +and REA authority are unchanged. + +| Trial | Input | Cached | Uncached | Output | CLI wall seconds | MCP calls | Facts | +| ---------- | -----: | -----: | -------: | -----: | ---------------: | --------: | ----- | +| complete-1 | 38,947 | 9,984 | 28,963 | 2,119 | 69.469 | 1 | pass | +| summary-1 | 58,779 | 38,144 | 20,635 | 3,241 | 141.304 | 2 | pass | +| summary-2 | 58,584 | 38,144 | 20,440 | 4,261 | 50.916 | 2 | pass | +| complete-2 | 38,917 | 23,040 | 15,877 | 2,813 | 106.896 | 1 | pass | + +Summary/page total input **increases 50.92% / 50.54%** and output increases +52.95% / 51.48%. Uncached input is mixed: 28.75% lower / 28.74% higher. Time is +mixed: 103.41% slower / 52.37% faster. No cause is inferred for caching or timing +variation. Cached input is included in total; each CLI reports one final aggregate +usage. Output includes 1,149 / 1,162 / 2,196 / 1,837 reasoning tokens. + +All calls succeed without extras/retries. Events contain only the selected MCP +tools and agent messages. All source-digest and private-profile cleanup checks +pass. Model payload CPU seconds/kernel peaks: 7.818/791,060,480; +9.793/752,705,536; 7.685/757,891,072; 7.999/750,993,408. These do not establish +a general CPU/memory gain. + +The first summary initially fails the grader because it uses explicit equivalent +labels and a URI object retaining page/source coverage. Its facts are correct. +The grading boundary accepts these known forms while checking exact types, +null presence, original limitations and extra scopes against actual receipts. +Conflicting duplicates and unsupported claims remain failures. Each cited +receipt's own ID must be reported; its parent lookup ID is checked separately. +Original answers/reviews/grader source remain intact. No model is rerun. A grader +review pause separates the pairs; it is not task time or a cold-cache claim. + +## Verification and decision + +An independent stdlib raw PCAP/PCAPNG/Ethernet/IP/TCP/complete-HTTP-header parser +agrees on 43/271 packets, per-packet endpoint counts, first URI prefixes and 2/23 +distinct HTTP URIs. It invokes no Wireshark. DNS associations, protocol-stack +dissection, nullable timestamps and live-network semantics are not independently +validated. + +Both real CLI complete results match full MCP normalized results, subjects, +parameters, IDs and limitations. Portable CLI summary/URI views match MCP retained +normalized facts and subjects; source forms/caller parameters remain distinct. +Source bytes stay unchanged. Nine context/grading regressions pass. Four real-answer +mutations—omitted null, false page exhaustion, contradictory source scope, +unsupported absence claim—fail correctly. JS syntax, Python compilation and +relevant standard lint pass. + +**Decision: keep existing complete defaults; add no production optimization.** +For these compact recordings and this question, summary/page costs more total +input in both real-model pairs. No stable speed or CPU/memory benefit justifies +another API, automatic delivery switch, cache or plugin layer. Existing pages +remain available; their benefit needs evidence for the actual question/inventory. +Large captures, other protocols, Windows/macOS/Bun Wireshark and other model +workloads remain unverified here. + +Heavy work stays serial with one CPU, 3 GiB early stop/4 GiB hard memory and no +swap. Build kernel peak: 2,575,732,736 bytes. Payload cleanup and zero phase limit +events are verified; missing terminal whole-scope counters remain unknown, so +strict whole-resource grades stay false. No raw capture, trace, credential, +runtime or result JSON is committed. No unrelated REA product bug is identified +or fixed, and no release is performed.