fix(experiments): reject malformed PCAP receipt identities as failed facts
Parse model-supplied identity values before dictionary/set lookup so malformed JSON remains a factual failure. Validate the actual stored-answer mutation and related grading regressions without rerunning models. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
This commit is contained in:
committed by
morluto
co-authored by
factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Kaoru0822
parent
78b8e28b67
commit
56a90afdde
@@ -122,9 +122,11 @@ validated.
|
||||
Both real CLI complete results match full MCP normalized results, subjects,
|
||||
parameters, IDs and limitations. Portable CLI summary/URI views match MCP retained
|
||||
normalized facts and subjects; source forms/caller parameters remain distinct.
|
||||
Source bytes stay unchanged. Nine context/grading regressions pass. Four real-answer
|
||||
Source bytes stay unchanged. Nine context/grading regressions pass. Five real-answer
|
||||
mutations—omitted null, false page exhaustion, contradictory source scope,
|
||||
unsupported absence claim—fail correctly. JS syntax, Python compilation and
|
||||
unsupported absence claim and non-string receipt identity—fail correctly.
|
||||
Malformed identities produce failed facts rather than crashing the grader.
|
||||
JS syntax, Python compilation and
|
||||
relevant standard lint pass.
|
||||
|
||||
**Decision: keep existing complete defaults; add no production optimization.**
|
||||
|
||||
@@ -181,7 +181,8 @@ def main():
|
||||
expected_observations = []
|
||||
if isinstance(canonical_observations, list):
|
||||
for observation in canonical_observations:
|
||||
record = records.get(observation.get('evidence_id')) if isinstance(observation, dict) else None
|
||||
identity = observation.get('evidence_id') if isinstance(observation, dict) else None
|
||||
record = records.get(identity) if isinstance(identity, str) else None
|
||||
if record is None:
|
||||
expected_observations.append(None)
|
||||
continue
|
||||
@@ -201,7 +202,8 @@ def main():
|
||||
if isinstance(canonical_observations, list) and all(isinstance(x, dict) for x in canonical_observations):
|
||||
check('pcap_requested_delivery_chain', [x.get('kind', 'complete') for x in canonical_observations],
|
||||
[x.get('kind', 'complete') for x in reference['delivery_observations']])
|
||||
checks['pcap_observations_have_reported_ids'] = {x.get('evidence_id') for x in canonical_observations} <= reported_ids
|
||||
identities = [x.get('evidence_id') for x in canonical_observations]
|
||||
checks['pcap_observations_have_reported_ids'] = all(isinstance(value, str) for value in identities) and set(identities) <= reported_ids
|
||||
else:
|
||||
checks['pcap_requested_delivery_chain'] = False
|
||||
checks['pcap_observations_have_reported_ids'] = False
|
||||
|
||||
Reference in New Issue
Block a user