Check page receipt uniqueness at the SDK workflow boundary before recording observations. A duplicate-ID injection over real MCP succeeds before the fix and fails after it without writing an answer; an unmodified real reply remains valid. Keep original measurements and production defaults unchanged. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
178 lines
11 KiB
Markdown
178 lines
11 KiB
Markdown
# Small real PCAP recordings: keep complete delivery
|
|
|
|
Continues [#2016](https://github.com/morluto/rea/issues/2016) after the QGIS
|
|
SQLite round. Measured main is pinned to
|
|
`3ae826ceba6f25b08bce50bcb51001209bc16b57`, with 23 intervening commits.
|
|
These include bounded PCAP summaries/pages, authenticated artifact subjects,
|
|
unknown-snapshot reuse and shared transcript grading. This round measures existing
|
|
PCAP delivery choices; it adds no production, API, default, cache or skill change.
|
|
|
|
## Corpus and question
|
|
|
|
Use two unmodified recordings published on the official
|
|
[Wireshark sample-capture page](https://wiki.wireshark.org/SampleCaptures):
|
|
|
|
| Recording | Bytes | SHA-256 |
|
|
| --------------------------------------- | -----: | ------------------------------------------------------------------ |
|
|
| HTTP request/response, `http.cap` | 25,803 | `25a72bdf10339f2c29916920c8b9501d294923108de8f29b19aba7cc001ab60d` |
|
|
| HTTP redirects, `http_redirects.pcapng` | 47,804 | `ff432a6b71d06cd20720f7763e584014edd0a43c3b55981c682d1b7640207abc` |
|
|
|
|
`corpus.json` preserves original/resolved URLs. Local digests pin downloaded
|
|
bytes; no independently published upstream digest or immutable source commit is
|
|
claimed. These are upstream saved application-protocol recordings, not generated
|
|
REA fixtures. Recorded clients and current deployments are unknown. No new
|
|
traffic is captured and no recorded target/URL is executed or fetched.
|
|
|
|
The question asks for complete returned identity/client/scalar metadata, every
|
|
observed count and source coverage, six retained inventory counts, explicit nulls,
|
|
diagnostic UTF-16 lengths, and the first two retained HTTP URIs in original order.
|
|
Preserve each used receipt's limitations and identity. Summary/page scope and
|
|
unknowns remain distinct from source coverage; exhaustion does not establish
|
|
network completeness. Other inventory lists and diagnostic text are unrequested.
|
|
|
|
Compare default `{path}` with existing `{path, detail: "summary"}` plus a retained
|
|
`http-uris` page, offset 0, limit 2. Ordinary selected-field
|
|
`inspect_packet_capture` remains a separate boundary, not a substitute claim
|
|
about complete inventories. `questions/pcap-http.txt` contains the fixed question.
|
|
|
|
## SDK measurements
|
|
|
|
Frozen dist/bridge file-table SHA-256:
|
|
`db050545e94dc305a48042e06df96ac6c64b21ee6c5c4707c787d5e46df79e47`.
|
|
Linux x64, Node 24.18.0, SDK client/server 2.3.1, installed Wireshark 4.0.17.
|
|
Explicit commands are `/usr/bin/tshark`, `/usr/bin/capinfos`, `/usr/bin/prlimit`;
|
|
private receipts retain executable hashes. Each workflow starts a fresh server
|
|
and uses actual named advertised definitions/schema validation. Host page caches
|
|
are not flushed. Both tools read the owned snapshot.
|
|
|
|
Eight trials run complete/summary/summary/complete per recording. Both pairs
|
|
match selected capture facts, URI prefixes, settings and original source bytes.
|
|
Each delivery's exact limitations, identities and coverage remain separately
|
|
recorded, rather than equating source coverage with page exhaustion.
|
|
|
|
| Trial | Reply bytes | SDK seconds | Payload CPU seconds | Kernel peak bytes |
|
|
| -------------------- | ----------: | ----------: | ------------------: | ----------------: |
|
|
| HTTP complete-1 | 7,601 | 4.349 | 5.336 | 611,778,560 |
|
|
| HTTP summary-1 | 17,540 | 4.972 | 5.973 | 601,661,440 |
|
|
| HTTP summary-2 | 17,540 | 5.101 | 6.092 | 611,438,592 |
|
|
| HTTP complete-2 | 7,601 | 4.463 | 5.503 | 617,148,416 |
|
|
| Redirects complete-1 | 7,535 | 4.670 | 5.641 | 610,627,584 |
|
|
| Redirects summary-1 | 17,240 | 5.099 | 6.104 | 615,522,304 |
|
|
| Redirects summary-2 | 17,240 | 5.060 | 6.056 | 610,385,920 |
|
|
| Redirects complete-2 | 7,535 | 4.466 | 5.458 | 608,141,312 |
|
|
|
|
Bytes are compact client-result JSON summed over one complete or two summary/page
|
|
calls, not model tokens or raw wire bytes. Summary/page expands replies to more
|
|
than twice the complete result on both recordings. All four SDK pairs are slower
|
|
and use more payload CPU; memory differences are small/inconsistent. CPU includes
|
|
driver/catalog work outside the SDK workflow timer.
|
|
|
|
Reproduce with `measure.mjs --artifact wireshark-http.cap` or
|
|
`wireshark-http_redirects.pcapng`, pinned path/runtime/revision, explicit provider
|
|
JSON, `--schemas selected`, and `--delivery complete|summary`. Use `compare.mjs`
|
|
on completed output directories. Original replies, catalogs, comparison receipts
|
|
and exact measured driver source stay private. Repeated deliveries in the same
|
|
mode also compare the ordered receipt observations, including limitations, view
|
|
parameters, coverage and unknowns; only receipt/parent IDs are invocation-specific.
|
|
Comparator regressions run with `node --test experiments/analysis-efficiency/test_compare.mjs`
|
|
using synthetic measurement directories; they do not establish provider behavior.
|
|
|
|
## Actual model tasks
|
|
|
|
Four actual Codex CLI 0.161.0 tasks use requested `gpt-5.6-terra` high, the same
|
|
redirects question/runtime/artifact/provider settings, direct mode, two exact
|
|
enabled tools and equal context/discovery capture flags. The requested alias is
|
|
not verified server-resolved model identity. Equal cohort instructions select
|
|
complete or summary plus one URI page. The experiment driver applies its existing
|
|
private per-tool approval policy to `inspect_pcap_capture`; global configuration
|
|
and REA authority are unchanged.
|
|
|
|
| Trial | Input | Cached | Uncached | Output | CLI wall seconds | MCP calls | Facts |
|
|
| ---------- | -----: | -----: | -------: | -----: | ---------------: | --------: | ----- |
|
|
| complete-1 | 38,947 | 9,984 | 28,963 | 2,119 | 69.469 | 1 | pass |
|
|
| summary-1 | 58,779 | 38,144 | 20,635 | 3,241 | 141.304 | 2 | pass |
|
|
| summary-2 | 58,584 | 38,144 | 20,440 | 4,261 | 50.916 | 2 | pass |
|
|
| complete-2 | 38,917 | 23,040 | 15,877 | 2,813 | 106.896 | 1 | pass |
|
|
|
|
Summary/page total input **increases 50.92% / 50.54%** and output increases
|
|
52.95% / 51.48%. Uncached input is mixed: 28.75% lower / 28.74% higher. Time is
|
|
mixed: 103.41% slower / 52.37% faster. No cause is inferred for caching or timing
|
|
variation. Cached input is included in total; each CLI reports one final aggregate
|
|
usage. Output includes 1,149 / 1,162 / 2,196 / 1,837 reasoning tokens.
|
|
|
|
All calls succeed without extras/retries. Events contain only the selected MCP
|
|
tools and agent messages. All source-digest and private-profile cleanup checks
|
|
pass. Model payload CPU seconds/kernel peaks: 7.818/791,060,480;
|
|
9.793/752,705,536; 7.685/757,891,072; 7.999/750,993,408. These do not establish
|
|
a general CPU/memory gain.
|
|
|
|
The first summary initially fails the grader because it uses explicit equivalent
|
|
labels and a URI object retaining page/source coverage. Its facts are correct.
|
|
The grading boundary accepts these known forms while checking exact types,
|
|
null presence, original limitations and extra scopes against actual receipts.
|
|
Conflicting duplicates and unsupported claims remain failures. Each cited
|
|
receipt's own ID must be reported; its parent lookup ID is checked separately.
|
|
Original answers/reviews/grader source remain intact. No model is rerun. A grader
|
|
review pause separates the pairs; it is not task time or a cold-cache claim.
|
|
|
|
## Verification and decision
|
|
|
|
An independent stdlib raw PCAP/PCAPNG/Ethernet/IP/TCP/complete-HTTP-header parser
|
|
agrees on 43/271 packets, per-packet endpoint counts, first URI prefixes and 2/23
|
|
distinct HTTP URIs. It invokes no Wireshark. DNS associations, protocol-stack
|
|
dissection, nullable timestamps and live-network semantics are not independently
|
|
validated.
|
|
|
|
Both real CLI complete results match full MCP normalized results, subjects,
|
|
parameters, IDs and limitations. Portable CLI summary/URI views match MCP retained
|
|
normalized facts and subjects; source forms/caller parameters remain distinct.
|
|
Source bytes stay unchanged. Fifteen context/grading regressions pass, including
|
|
cited complete metadata/counts/UTF-16 diagnostics and exact ordered response ID
|
|
lists that reject parent lookups, duplicates and nested representations. URI
|
|
prefix coverage must come from its cited page, even if an uncited page has the
|
|
same items. The SDK driver also checks the returned view against the exact
|
|
requested kind, facet, offset and limit. Five real-answer
|
|
mutations—omitted null, false page exhaustion, contradictory source scope,
|
|
unsupported absence claim and non-string receipt identity—fail correctly.
|
|
Malformed identities produce failed facts rather than crashing the grader.
|
|
JS syntax, Python compilation and
|
|
relevant standard lint pass.
|
|
|
|
A fresh real SDK summary/URI smoke matches the original selected facts. Three
|
|
private consumer fault injections alter the returned view's offset, limit or
|
|
presence after a real SDK reply; the driver rejects each. These establish the
|
|
experiment consumer's guard, not provider malformed-reply behavior. The initial
|
|
injection setup loaded the separate CommonJS client and did not inject a fault;
|
|
that discarded setup attempt remains private alongside the corrected ESM runs.
|
|
Original measured driver sources, token totals and timing data remain unchanged.
|
|
|
|
The driver also requires a distinct Evidence receipt for the retained URI page.
|
|
A private SDK consumer injection reuses the summary's response ID after a real
|
|
MCP reply: the preceding driver accepts it, the corrected driver rejects it
|
|
without writing an answer, and an unmodified real reply still passes. This
|
|
checks receipt uniqueness before recording observations; it does not establish
|
|
that a provider actually returned duplicate IDs.
|
|
|
|
**Decision: keep existing complete defaults; add no production optimization.**
|
|
For these compact recordings and this question, summary/page costs more total
|
|
input in both real-model pairs. No stable speed or CPU/memory benefit justifies
|
|
another API, automatic delivery switch, cache or plugin layer. Existing pages
|
|
remain available; their benefit needs evidence for the actual question/inventory.
|
|
Large captures, other protocols, Windows/macOS/Bun Wireshark and other model
|
|
workloads remain unverified here.
|
|
|
|
Heavy work stays serial with one CPU, 3 GiB early stop/4 GiB hard memory and no
|
|
swap. Build kernel peak: 2,575,732,736 bytes. Payload cleanup and zero phase limit
|
|
events are verified; missing terminal whole-scope counters remain unknown, so
|
|
strict whole-resource grades stay false. No raw capture, trace, credential,
|
|
runtime or result JSON is committed. No unrelated REA product bug is identified
|
|
or fixed, and no release is performed.
|
|
|
|
Review follow-up binds the cited URI page to its exact requested facet/range,
|
|
reported values and the summary retained parent. Small synthetic trial receipts
|
|
exercise the entire grading entrypoint, including faithful but wrong-page/range/
|
|
parent citations, missing nulls, false scope and malformed identities. These are
|
|
grader regressions, not real-provider evidence. All four stored actual model
|
|
answers still pass; no model reruns are required. Transcript readers close their
|
|
owned file handles after grading.
|