fix(experiments): preserve PCAP receipt producer provenance
Keep tool and Evidence together in the receipt index, reject conflicting ID ownership, and validate requested tool and delegated operation roles. Preserve legitimate summary delegation to inspect_analysis_view and identical repeated receipts. Validate the same operation contract at the SDK observation boundary. Twenty grading/context tests and six comparator tests pass after reproducing the gap; real SDK fault/control runs and all four stored actual-model factual grades pass without model reruns. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
This commit is contained in:
committed by
morluto
co-authored by
factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Kaoru0822
parent
073f6c07e9
commit
3e88ebf735
@@ -538,6 +538,16 @@ const pcapObservation = (evidence) => {
|
||||
target?.path !== values.path
|
||||
)
|
||||
throw new Error("PCAP Evidence does not identify the selected capture");
|
||||
// Summary delivery delegates to a retained view, whose Evidence operation is
|
||||
// inspect_analysis_view even when inspect_pcap_capture was requested.
|
||||
const operation =
|
||||
values.delivery === "summary"
|
||||
? "inspect_analysis_view"
|
||||
: "inspect_pcap_capture";
|
||||
if (evidence.operation !== operation)
|
||||
throw new Error(
|
||||
"PCAP Evidence operation differs from the requested delivery",
|
||||
);
|
||||
if (
|
||||
values.delivery === "summary" &&
|
||||
(typeof value.parent_evidence_id !== "string" ||
|
||||
|
||||
@@ -126,7 +126,7 @@ validated.
|
||||
Both real CLI complete results match full MCP normalized results, subjects,
|
||||
parameters, IDs and limitations. Portable CLI summary/URI views match MCP retained
|
||||
normalized facts and subjects; source forms/caller parameters remain distinct.
|
||||
Source bytes stay unchanged. Eighteen context/grading regressions pass, including
|
||||
Source bytes stay unchanged. Twenty context/grading regressions pass, including
|
||||
cited complete metadata/counts/UTF-16 diagnostics and exact ordered response ID
|
||||
lists that reject parent lookups, duplicates and nested representations. URI
|
||||
prefix coverage must come from its cited page, even if an uncited page has the
|
||||
@@ -159,7 +159,7 @@ the preceding driver accepts inconsistent coverage, while the corrected driver
|
||||
rejects wrong status, next offset or exhaustion. Unmodified exhausted and
|
||||
continuable URI pages still pass. The grader also requires each cited receipt's
|
||||
subject digest and local path to identify the reference capture, and uses the
|
||||
reference page's verified coverage. Eighteen grading/context regressions pass,
|
||||
reference page's verified coverage. Twenty grading/context regressions pass,
|
||||
including missing or malformed subjects and faithful but inconsistent coverage;
|
||||
all four original model answers still pass factual regrading. Strict resource
|
||||
grades remain false where terminal whole-scope counters are unknown.
|
||||
@@ -171,6 +171,17 @@ links and differ from that response's own ID. The grader enforces the same link
|
||||
relationship. Private actual-reply injections and normal complete/summary
|
||||
controls exercise this boundary; these do not establish a provider defect.
|
||||
|
||||
The grader preserves each receipt's producing tool while indexing it; conflicting
|
||||
tool/envelope ownership for one ID remains ambiguous, while identical repeated
|
||||
receipts are allowed. Complete delivery requires the capture tool/operation.
|
||||
Summary delivery uses the capture tool but correctly delegates its Evidence
|
||||
operation to `inspect_analysis_view`; the URI page uses that view tool/operation.
|
||||
Both the grader and SDK observation producer validate this actual contract.
|
||||
Wrong-tool/operation regressions fail before the correction and pass afterward;
|
||||
private SDK operation faults are rejected while real complete/summary controls
|
||||
and all four stored model factual grades still pass. No operation is rewritten
|
||||
to make the tool name and delegated Evidence operation identical.
|
||||
|
||||
**Decision: keep existing complete defaults; add no production optimization.**
|
||||
For these compact recordings and this question, summary/page costs more total
|
||||
input in both real-model pairs. No stable speed or CPU/memory benefit justifies
|
||||
|
||||
@@ -181,8 +181,18 @@ def main():
|
||||
check('capture', ('capture' in answer, json.dumps(answer.get('capture'), sort_keys=True)),
|
||||
(True, json.dumps(reference['capture'], sort_keys=True)))
|
||||
prefix = answer.get('http_request_uris_first2')
|
||||
records = {value['evidence_id']: value for values in producer_results.values()
|
||||
for value in values if isinstance(value, dict) and 'evidence_id' in value}
|
||||
records = {}
|
||||
for tool, values in producer_results.items():
|
||||
for value in values:
|
||||
if not isinstance(value, dict) or not isinstance(value.get('evidence_id'), str):
|
||||
continue
|
||||
identity = value['evidence_id']
|
||||
entry = (tool, value)
|
||||
if identity not in records:
|
||||
records[identity] = entry
|
||||
elif records[identity] != entry:
|
||||
# An ID with conflicting producers or envelopes is ambiguous.
|
||||
records[identity] = None
|
||||
observations = answer.get('delivery_observations')
|
||||
canonical_observations = [pcap_observation_fields(value) for value in observations] if isinstance(observations, list) else observations
|
||||
if isinstance(prefix, dict):
|
||||
@@ -192,7 +202,8 @@ def main():
|
||||
if 'coverage' in prefix:
|
||||
last = canonical_observations[-1] if isinstance(canonical_observations, list) and canonical_observations else None
|
||||
identity = last.get('evidence_id') if isinstance(last, dict) else None
|
||||
record = records.get(identity) if isinstance(identity, str) else None
|
||||
entry = records.get(identity) if isinstance(identity, str) else None
|
||||
record = entry[1] if entry is not None else None
|
||||
page = record['normalized_result'] if record is not None else {}
|
||||
checks['pcap_prefix_coverage_observed'] = (
|
||||
page.get('kind') == 'pcap'
|
||||
@@ -208,7 +219,8 @@ def main():
|
||||
if isinstance(canonical_observations, list):
|
||||
for observation in canonical_observations:
|
||||
identity = observation.get('evidence_id') if isinstance(observation, dict) else None
|
||||
record = records.get(identity) if isinstance(identity, str) else None
|
||||
entry = records.get(identity) if isinstance(identity, str) else None
|
||||
record = entry[1] if entry is not None else None
|
||||
if record is None:
|
||||
expected_observations.append(None)
|
||||
continue
|
||||
@@ -237,7 +249,14 @@ def main():
|
||||
and all(isinstance(value, str) and re.fullmatch(r'ev_[a-f0-9]{64}', value) for value in identities)
|
||||
and len(set(identities)) == len(identities)
|
||||
)
|
||||
cited = [records.get(identity) if isinstance(identity, str) else None for identity in identities]
|
||||
entries = [records.get(identity) if isinstance(identity, str) else None for identity in identities]
|
||||
cited = [entry[1] if entry is not None else None for entry in entries]
|
||||
expected_tools = ['inspect_pcap_capture'] + (['inspect_analysis_view'] if len(reference['delivery_observations']) == 2 else [])
|
||||
check('pcap_cited_producer_tools', [entry[0] if entry is not None else None for entry in entries], expected_tools)
|
||||
# Summary delivery delegates to a retained view; its envelope operation
|
||||
# is inspect_analysis_view even though the requested tool is inspect_pcap_capture.
|
||||
expected_operations = ['inspect_pcap_capture'] if len(expected_tools) == 1 else ['inspect_analysis_view'] * 2
|
||||
check('pcap_cited_operations', [record.get('operation') if isinstance(record, dict) else None for record in cited], expected_operations)
|
||||
checks['pcap_cited_subjects_match_artifact'] = bool(cited) and all(
|
||||
isinstance(record, dict)
|
||||
and isinstance(record.get('subject'), dict)
|
||||
@@ -296,6 +315,8 @@ def main():
|
||||
checks['pcap_cited_capture'] = False
|
||||
checks['pcap_cited_subjects_match_artifact'] = False
|
||||
checks['pcap_cited_parents_linked'] = False
|
||||
checks['pcap_cited_producer_tools'] = False
|
||||
checks['pcap_cited_operations'] = False
|
||||
elif domain == 'flutter-aot':
|
||||
for key in ['target', 'abi', 'libapp', 'string_pool', 'coverage', 'limitations']:
|
||||
check(key, answer.get(key), reference[key])
|
||||
|
||||
@@ -82,13 +82,15 @@ class PcapGraderWorkflowTests(unittest.TestCase):
|
||||
prefix = ['http://example.test/one', 'http://example.test/two']
|
||||
coverage = {'status': 'page', 'examined': 2, 'total': 3, 'next_offset': 2, 'exhausted': False}
|
||||
subject = {'digest': {'sha256': capture['target']['sha256']}, 'local_path': capture['target']['path']}
|
||||
summary = {'evidence_id': summary_id, 'subject': copy.deepcopy(subject), 'evidence_links': [parent],
|
||||
summary = {'evidence_id': summary_id, 'operation': 'inspect_analysis_view',
|
||||
'subject': copy.deepcopy(subject), 'evidence_links': [parent],
|
||||
'limitations': ['source', 'view'],
|
||||
'normalized_result': {'kind': 'summary', 'view': {'kind': 'summary'},
|
||||
'parent_operation': 'inspect_pcap_capture', 'parent_evidence_id': parent,
|
||||
'summary': capture, 'coverage': {'status': 'complete-within-view', 'examined': 1, 'total': 1, 'next_offset': None, 'exhausted': True},
|
||||
'limitations': ['source', 'view'], 'unknowns': ['unrecorded traffic']}}
|
||||
page = {'evidence_id': page_id, 'subject': copy.deepcopy(subject), 'evidence_links': [parent],
|
||||
page = {'evidence_id': page_id, 'operation': 'inspect_analysis_view',
|
||||
'subject': copy.deepcopy(subject), 'evidence_links': [parent],
|
||||
'limitations': ['source', 'view'],
|
||||
'normalized_result': {'kind': 'pcap',
|
||||
'view': {'kind': 'pcap', 'facet': 'http-uris', 'offset': 0, 'limit': 2},
|
||||
@@ -111,7 +113,7 @@ class PcapGraderWorkflowTests(unittest.TestCase):
|
||||
'parent_evidence_id': value['parent_evidence_id'], 'coverage': value['coverage'],
|
||||
'unknowns': value['unknowns'], 'normalized_limitations': value['limitations']}
|
||||
|
||||
def grade(self, answer, records, reference):
|
||||
def grade(self, answer, records, reference, producer_tools=None):
|
||||
review = runpy.run_path(str(Path(__file__).with_name('review-model.py')))
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory);trial = root / 'trial';trial.mkdir()
|
||||
@@ -120,8 +122,9 @@ class PcapGraderWorkflowTests(unittest.TestCase):
|
||||
(trial / name).write_text(json.dumps(value))
|
||||
(trial / 'answer.txt').write_text(json.dumps(answer))
|
||||
events = [{'type': 'item.completed', 'item': {'type': 'mcp_tool_call',
|
||||
'tool': 'inspect_analysis_view' if value['normalized_result'].get('kind') == 'pcap' else 'inspect_pcap_capture',
|
||||
'status': 'completed', 'result': {'structuredContent': value}}} for value in records]
|
||||
'tool': producer_tools[index] if producer_tools is not None else
|
||||
('inspect_analysis_view' if value['normalized_result'].get('kind') == 'pcap' else 'inspect_pcap_capture'),
|
||||
'status': 'completed', 'result': {'structuredContent': value}}} for index, value in enumerate(records)]
|
||||
(trial / 'events.jsonl').write_text(''.join(json.dumps(value) + '\n' for value in events))
|
||||
source = root / 'reference';source.mkdir();(source / 'answer.json').write_text(json.dumps(reference))
|
||||
resource = root / 'resource.log'
|
||||
@@ -187,12 +190,16 @@ class PcapGraderWorkflowTests(unittest.TestCase):
|
||||
source.update({key: [] for key in capture['retained_counts']})
|
||||
source['http_request_uris'] = reference['http_request_uris_first2'] + ['http://example.test/three']
|
||||
source['diagnostics'] = {'tshark_stderr': '', 'capinfos_stderr': '\U0001f4a5'}
|
||||
receipt = {'evidence_id': 'ev_' + 'a' * 64, 'subject': copy.deepcopy(records[0]['subject']), 'evidence_links': [],
|
||||
receipt = {'evidence_id': 'ev_' + 'a' * 64, 'operation': 'inspect_pcap_capture',
|
||||
'subject': copy.deepcopy(records[0]['subject']), 'evidence_links': [],
|
||||
'limitations': ['source'], 'normalized_result': source}
|
||||
reference['delivery_observations'] = [{'evidence_id': receipt['evidence_id'],
|
||||
'limitations': receipt['limitations'], 'coverage': source['coverage']}]
|
||||
answer = copy.deepcopy({**reference, 'evidence_ids': [receipt['evidence_id']]})
|
||||
self.assertTrue(self.grade(answer, [receipt], reference)['all_factual_checks_pass'])
|
||||
self.assertFalse(self.grade(answer, [receipt], reference, ['inspect_analysis_view'])['all_factual_checks_pass'])
|
||||
wrong_operation = copy.deepcopy(receipt);wrong_operation['operation'] = 'inspect_analysis_view'
|
||||
self.assertFalse(self.grade(answer, [wrong_operation], reference)['all_factual_checks_pass'])
|
||||
for subject in [None, {'digest': {'sha256': 'e' * 64}, 'local_path': source['target']['path']}]:
|
||||
changed = copy.deepcopy(receipt);changed['subject'] = subject
|
||||
report = self.grade(answer, [changed], reference)
|
||||
@@ -264,6 +271,29 @@ class PcapGraderWorkflowTests(unittest.TestCase):
|
||||
self.assertFalse(report['factual_checks']['pcap_cited_parents_linked'])
|
||||
self.assertFalse(report['all_factual_checks_pass'])
|
||||
|
||||
def test_cited_receipts_preserve_tool_and_delegated_operation_provenance(self):
|
||||
for tools in [['inspect_analysis_view', 'inspect_analysis_view'],
|
||||
['inspect_pcap_capture', 'inspect_pcap_capture'],
|
||||
['unrelated_tool', 'inspect_analysis_view']]:
|
||||
with self.subTest(tools=tools):
|
||||
answer, records, reference = self.fixture()
|
||||
report = self.grade(answer, records, reference, tools)
|
||||
self.assertTrue(report['factual_checks']['pcap_original_delivery_observations'])
|
||||
self.assertFalse(report['all_factual_checks_pass'])
|
||||
for index in [0, 1]:
|
||||
answer, records, reference = self.fixture();records[index]['operation'] = 'inspect_pcap_capture'
|
||||
self.assertFalse(self.grade(answer, records, reference)['all_factual_checks_pass'])
|
||||
|
||||
def test_repeated_identical_receipts_pass_but_conflicting_id_ownership_fails(self):
|
||||
answer, records, reference = self.fixture();duplicate = copy.deepcopy(records[0])
|
||||
self.assertTrue(self.grade(answer, [duplicate, *records], reference)['all_factual_checks_pass'])
|
||||
for duplicate_first in [True, False]:
|
||||
with self.subTest(duplicate_first=duplicate_first):
|
||||
items = [duplicate, *records] if duplicate_first else [*records, duplicate]
|
||||
tools = ['unrelated_tool', 'inspect_pcap_capture', 'inspect_analysis_view'] if duplicate_first else \
|
||||
['inspect_pcap_capture', 'inspect_analysis_view', 'unrelated_tool']
|
||||
self.assertFalse(self.grade(answer, items, reference, tools)['all_factual_checks_pass'])
|
||||
|
||||
def test_reported_ids_are_only_the_ordered_unique_response_receipts(self):
|
||||
answer, records, reference = self.fixture()
|
||||
original = answer['evidence_ids']
|
||||
|
||||
Reference in New Issue
Block a user