docs(perf): preserve negative small-capture token and timing results
Record repeated public SDK/Codex comparisons, real CLI parity, independent raw-frame validation and grading regressions. Keep whole-resource counters and broader capture/platform coverage explicitly unknown; no production change is justified. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
This commit is contained in:
committed by
morluto
co-authored by
factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Kaoru0822
parent
fb10bdc546
commit
78b8e28b67
@@ -19,6 +19,10 @@ Follow-up: [SQLite existing-control measurements](sqlite-existing-controls.md)
|
||||
tests QGIS's shipped GeoPackage through the existing column selection, preserving
|
||||
full selected-table schema and typed cells before considering another API.
|
||||
|
||||
Follow-up: [small real PCAP recordings](pcap-small-captures.md) compares existing
|
||||
complete delivery with summary/pages. Preserve the negative SDK/model input
|
||||
results: compact captures do not justify another production mechanism.
|
||||
|
||||
Follow-up: [native PE/DOS entry views and real model verification](native-entry-views.md)
|
||||
records the post-merge native continuation, including a small-function case where
|
||||
four selected replies cost more bytes than the complete dossier.
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
# Small real PCAP recordings: keep complete delivery
|
||||
|
||||
Continues [#2016](https://github.com/morluto/rea/issues/2016) after the QGIS
|
||||
SQLite round. Measured main is pinned to
|
||||
`3ae826ceba6f25b08bce50bcb51001209bc16b57`, with 23 intervening commits.
|
||||
These include bounded PCAP summaries/pages, authenticated artifact subjects,
|
||||
unknown-snapshot reuse and shared transcript grading. This round measures existing
|
||||
PCAP delivery choices; it adds no production, API, default, cache or skill change.
|
||||
|
||||
## Corpus and question
|
||||
|
||||
Use two unmodified recordings published on the official
|
||||
[Wireshark sample-capture page](https://wiki.wireshark.org/SampleCaptures):
|
||||
|
||||
| Recording | Bytes | SHA-256 |
|
||||
| --------------------------------------- | -----: | ------------------------------------------------------------------ |
|
||||
| HTTP request/response, `http.cap` | 25,803 | `25a72bdf10339f2c29916920c8b9501d294923108de8f29b19aba7cc001ab60d` |
|
||||
| HTTP redirects, `http_redirects.pcapng` | 47,804 | `ff432a6b71d06cd20720f7763e584014edd0a43c3b55981c682d1b7640207abc` |
|
||||
|
||||
`corpus.json` preserves original/resolved URLs. Local digests pin downloaded
|
||||
bytes; no independently published upstream digest or immutable source commit is
|
||||
claimed. These are upstream saved application-protocol recordings, not generated
|
||||
REA fixtures. Recorded clients and current deployments are unknown. No new
|
||||
traffic is captured and no recorded target/URL is executed or fetched.
|
||||
|
||||
The question asks for complete returned identity/client/scalar metadata, every
|
||||
observed count and source coverage, six retained inventory counts, explicit nulls,
|
||||
diagnostic UTF-16 lengths, and the first two retained HTTP URIs in original order.
|
||||
Preserve each used receipt's limitations and identity. Summary/page scope and
|
||||
unknowns remain distinct from source coverage; exhaustion does not establish
|
||||
network completeness. Other inventory lists and diagnostic text are unrequested.
|
||||
|
||||
Compare default `{path}` with existing `{path, detail: "summary"}` plus a retained
|
||||
`http-uris` page, offset 0, limit 2. Ordinary selected-field
|
||||
`inspect_packet_capture` remains a separate boundary, not a substitute claim
|
||||
about complete inventories. `questions/pcap-http.txt` contains the fixed question.
|
||||
|
||||
## SDK measurements
|
||||
|
||||
Frozen dist/bridge file-table SHA-256:
|
||||
`db050545e94dc305a48042e06df96ac6c64b21ee6c5c4707c787d5e46df79e47`.
|
||||
Linux x64, Node 24.18.0, SDK client/server 2.3.1, installed Wireshark 4.0.17.
|
||||
Explicit commands are `/usr/bin/tshark`, `/usr/bin/capinfos`, `/usr/bin/prlimit`;
|
||||
private receipts retain executable hashes. Each workflow starts a fresh server
|
||||
and uses actual named advertised definitions/schema validation. Host page caches
|
||||
are not flushed. Both tools read the owned snapshot.
|
||||
|
||||
Eight trials run complete/summary/summary/complete per recording. Both pairs
|
||||
match selected capture facts, URI prefixes, settings and original source bytes.
|
||||
Each delivery's exact limitations, identities and coverage remain separately
|
||||
recorded, rather than equating source coverage with page exhaustion.
|
||||
|
||||
| Trial | Reply bytes | SDK seconds | Payload CPU seconds | Kernel peak bytes |
|
||||
| -------------------- | ----------: | ----------: | ------------------: | ----------------: |
|
||||
| HTTP complete-1 | 7,601 | 4.349 | 5.336 | 611,778,560 |
|
||||
| HTTP summary-1 | 17,540 | 4.972 | 5.973 | 601,661,440 |
|
||||
| HTTP summary-2 | 17,540 | 5.101 | 6.092 | 611,438,592 |
|
||||
| HTTP complete-2 | 7,601 | 4.463 | 5.503 | 617,148,416 |
|
||||
| Redirects complete-1 | 7,535 | 4.670 | 5.641 | 610,627,584 |
|
||||
| Redirects summary-1 | 17,240 | 5.099 | 6.104 | 615,522,304 |
|
||||
| Redirects summary-2 | 17,240 | 5.060 | 6.056 | 610,385,920 |
|
||||
| Redirects complete-2 | 7,535 | 4.466 | 5.458 | 608,141,312 |
|
||||
|
||||
Bytes are compact client-result JSON summed over one complete or two summary/page
|
||||
calls, not model tokens or raw wire bytes. Summary/page expands replies to more
|
||||
than twice the complete result on both recordings. All four SDK pairs are slower
|
||||
and use more payload CPU; memory differences are small/inconsistent. CPU includes
|
||||
driver/catalog work outside the SDK workflow timer.
|
||||
|
||||
Reproduce with `measure.mjs --artifact wireshark-http.cap` or
|
||||
`wireshark-http_redirects.pcapng`, pinned path/runtime/revision, explicit provider
|
||||
JSON, `--schemas selected`, and `--delivery complete|summary`. Use `compare.mjs`
|
||||
on completed output directories. Original replies, catalogs, comparison receipts
|
||||
and exact measured driver source stay private.
|
||||
|
||||
## Actual model tasks
|
||||
|
||||
Four actual Codex CLI 0.161.0 tasks use requested `gpt-5.6-terra` high, the same
|
||||
redirects question/runtime/artifact/provider settings, direct mode, two exact
|
||||
enabled tools and equal context/discovery capture flags. The requested alias is
|
||||
not verified server-resolved model identity. Equal cohort instructions select
|
||||
complete or summary plus one URI page. The experiment driver applies its existing
|
||||
private per-tool approval policy to `inspect_pcap_capture`; global configuration
|
||||
and REA authority are unchanged.
|
||||
|
||||
| Trial | Input | Cached | Uncached | Output | CLI wall seconds | MCP calls | Facts |
|
||||
| ---------- | -----: | -----: | -------: | -----: | ---------------: | --------: | ----- |
|
||||
| complete-1 | 38,947 | 9,984 | 28,963 | 2,119 | 69.469 | 1 | pass |
|
||||
| summary-1 | 58,779 | 38,144 | 20,635 | 3,241 | 141.304 | 2 | pass |
|
||||
| summary-2 | 58,584 | 38,144 | 20,440 | 4,261 | 50.916 | 2 | pass |
|
||||
| complete-2 | 38,917 | 23,040 | 15,877 | 2,813 | 106.896 | 1 | pass |
|
||||
|
||||
Summary/page total input **increases 50.92% / 50.54%** and output increases
|
||||
52.95% / 51.48%. Uncached input is mixed: 28.75% lower / 28.74% higher. Time is
|
||||
mixed: 103.41% slower / 52.37% faster. No cause is inferred for caching or timing
|
||||
variation. Cached input is included in total; each CLI reports one final aggregate
|
||||
usage. Output includes 1,149 / 1,162 / 2,196 / 1,837 reasoning tokens.
|
||||
|
||||
All calls succeed without extras/retries. Events contain only the selected MCP
|
||||
tools and agent messages. All source-digest and private-profile cleanup checks
|
||||
pass. Model payload CPU seconds/kernel peaks: 7.818/791,060,480;
|
||||
9.793/752,705,536; 7.685/757,891,072; 7.999/750,993,408. These do not establish
|
||||
a general CPU/memory gain.
|
||||
|
||||
The first summary initially fails the grader because it uses explicit equivalent
|
||||
labels and a URI object retaining page/source coverage. Its facts are correct.
|
||||
The grading boundary accepts these known forms while checking exact types,
|
||||
null presence, original limitations and extra scopes against actual receipts.
|
||||
Conflicting duplicates and unsupported claims remain failures. Each cited
|
||||
receipt's own ID must be reported; its parent lookup ID is checked separately.
|
||||
Original answers/reviews/grader source remain intact. No model is rerun. A grader
|
||||
review pause separates the pairs; it is not task time or a cold-cache claim.
|
||||
|
||||
## Verification and decision
|
||||
|
||||
An independent stdlib raw PCAP/PCAPNG/Ethernet/IP/TCP/complete-HTTP-header parser
|
||||
agrees on 43/271 packets, per-packet endpoint counts, first URI prefixes and 2/23
|
||||
distinct HTTP URIs. It invokes no Wireshark. DNS associations, protocol-stack
|
||||
dissection, nullable timestamps and live-network semantics are not independently
|
||||
validated.
|
||||
|
||||
Both real CLI complete results match full MCP normalized results, subjects,
|
||||
parameters, IDs and limitations. Portable CLI summary/URI views match MCP retained
|
||||
normalized facts and subjects; source forms/caller parameters remain distinct.
|
||||
Source bytes stay unchanged. Nine context/grading regressions pass. Four real-answer
|
||||
mutations—omitted null, false page exhaustion, contradictory source scope,
|
||||
unsupported absence claim—fail correctly. JS syntax, Python compilation and
|
||||
relevant standard lint pass.
|
||||
|
||||
**Decision: keep existing complete defaults; add no production optimization.**
|
||||
For these compact recordings and this question, summary/page costs more total
|
||||
input in both real-model pairs. No stable speed or CPU/memory benefit justifies
|
||||
another API, automatic delivery switch, cache or plugin layer. Existing pages
|
||||
remain available; their benefit needs evidence for the actual question/inventory.
|
||||
Large captures, other protocols, Windows/macOS/Bun Wireshark and other model
|
||||
workloads remain unverified here.
|
||||
|
||||
Heavy work stays serial with one CPU, 3 GiB early stop/4 GiB hard memory and no
|
||||
swap. Build kernel peak: 2,575,732,736 bytes. Payload cleanup and zero phase limit
|
||||
events are verified; missing terminal whole-scope counters remain unknown, so
|
||||
strict whole-resource grades stay false. No raw capture, trace, credential,
|
||||
runtime or result JSON is committed. No unrelated REA product bug is identified
|
||||
or fixed, and no release is performed.
|
||||
Reference in New Issue
Block a user