From 22326e8199975f31d8b7c99aaf5fd64140fbe00e Mon Sep 17 00:00:00 2001 From: N0zoM1z0 <1705004398@qq.com> Date: Sun, 11 Oct 2026 18:18:22 +0000 Subject: [PATCH] feat(experiments): add typed discovery and reproducible cross-domain questions --- experiments/codemode/README.md | 29 +++++++++++-- experiments/codemode/agent-trial.py | 15 +++++++ experiments/codemode/benchmark.mjs | 37 ++++++++++++++-- experiments/codemode/bridge.mjs | 35 +++++++++++---- experiments/codemode/bridge.test.mjs | 14 +++++- experiments/codemode/grade-trial.py | 27 +++++++++++- experiments/codemode/questions/android.txt | 3 ++ .../codemode/questions/electron-inventory.txt | 1 + experiments/codemode/questions/firmware.txt | 1 + experiments/codemode/questions/managed.txt | 1 + experiments/codemode/questions/native.txt | 1 + experiments/codemode/resource-run.py | 4 ++ experiments/codemode/results.md | 43 +++++++++++++++++++ 13 files changed, 194 insertions(+), 17 deletions(-) create mode 100644 experiments/codemode/questions/android.txt create mode 100644 experiments/codemode/questions/electron-inventory.txt create mode 100644 experiments/codemode/questions/firmware.txt create mode 100644 experiments/codemode/questions/managed.txt create mode 100644 experiments/codemode/questions/native.txt diff --git a/experiments/codemode/README.md b/experiments/codemode/README.md index c0041a18b..4baf1067e 100644 --- a/experiments/codemode/README.md +++ b/experiments/codemode/README.md @@ -28,7 +28,9 @@ Register that command as a separate, temporary MCP server for a trial. Existing REA registrations keep their original tools. `rea_search` accepts a query or exact names; use `schemas: false` for the complete lightweight name/description list. Query discovery returns names/descriptions by default; exact-name discovery -returns original input schemas. `schemas` explicitly overrides that choice. +returns original input schemas and derived input/output TypeScript declarations. +These declarations help write code; they do not replace JSON validation contracts. +`schemas` explicitly overrides that choice. Start with one domain word and request all selected exact names in one call. Request `output_schemas: true` when a result contract is needed; the original output schema is then returned unchanged. @@ -43,7 +45,7 @@ return { open: session.open, identity: session.server_identity }; The returned projection is not a new Evidence record. `calls` preserves operation, argument/result digests, upstream errors and actual producer Evidence IDs when -provided. The upstream ledger retains complete original Evidence for later views +provided, including original `evidence_links`. The upstream ledger retains complete original Evidence for later views and exports. Non-Evidence session responses have no invented Evidence ID. Calls and executions sharing one active target are serialized; `Promise.all` currently provides composition rather than parallel provider execution. @@ -89,6 +91,11 @@ Raw responses remain outside Git. Result summaries separate complete envelopes from single text content and count code explicitly. Measured findings and failures are recorded in [results.md](results.md). +`--artifact obsidian.asar --workflow obsidian-inventory` asks a distinct package +inventory question using existing tools. It must not be compared as an equivalent +completion of the JavaScript AST question. Workflow names and original arguments +are retained in replay summaries. + `agent-trial.py` runs a real local Codex CLI with a disposable account/config and one MCP registration. It copies authentication only, removes the profile on normal completion, timeout and handled termination, and retains JSONL events, the answer @@ -102,20 +109,34 @@ python3 resource-run.py -- python3 agent-trial.py \ --output /absolute/path/new-trial-directory ``` +Reproducible questions are in [questions/](questions/). Supply a template with +`--prompt questions/android.txt --artifact-path /absolute/path/NewPipe_v0.29.1.apk` +(or another matching corpus target). The runner preserves the supplied path in +the rendered prompt and records template and rendered-prompt digests. Native's +entry-address seed is derived from this corpus's jq header and is not reusable +for another build. Run comparisons serially with the same question and provider +settings; retain failures as well as successful repetitions. + Use `--mode bridge` for the experimental server; `--mode client-code` requests Codex's own Code Mode. Optional `--providers` selects a JSON object of explicit provider settings. No ambient environment snapshot is retained. A configured flag or a model's `tool_mode` does not prove the model actually composed tools or filtered an intermediate result. Inspect the transcript, and mark accuracy as unassessed until the requested facts and source support have been reviewed. -`grade-trial.py` checks Android (default) or `--task managed` against a successful -real-provider replay and verifies cited producer or bridge audit IDs. It accepts +`grade-trial.py` checks Android (default), managed, native, firmware or +electron-inventory (`--task NAME`) against a successful +real-provider replay and verifies cited IDs in producer values, their original +Evidence links or the bridge audit. This verifies references observed in the +transcript; the grader does not resolve every cited payload. It accepts JSON or one complete JSON fence, without repairing prose or partial values. This checks requested producer facts and reference existence, not runtime behavior or independent decompiler correctness. Unparseable result text is reported explicitly; IDs in a truncated JSON fragment cannot satisfy the automated reference check. The Android projection includes the provider's overload index, so a supported signature with that field omitted fails this stronger assertion. +Firmware's run-local region UUIDs remain in answers but are excluded from parity +against an independently acquired report. Classification evidence, explicit +unknowns and coverage remain part of the requested fact checks. The supplied feedback is a research lead. Its Ghidra timing claim also changed auto-analysis options, so it does not isolate Code Mode's effect. Relevant primary diff --git a/experiments/codemode/agent-trial.py b/experiments/codemode/agent-trial.py index 54972da72..b290b0152 100644 --- a/experiments/codemode/agent-trial.py +++ b/experiments/codemode/agent-trial.py @@ -16,6 +16,7 @@ def main(): parser = argparse.ArgumentParser() parser.add_argument('--mode', choices=['direct', 'client-code', 'bridge'], required=True) parser.add_argument('--prompt', type=Path, required=True) + parser.add_argument('--artifact-path', help='Replace {{artifact_path}} in a question template') parser.add_argument('--output', type=Path, required=True) parser.add_argument('--model', default='gpt-5.6-terra') parser.add_argument('--reasoning', default='high') @@ -125,12 +126,26 @@ def main(): (trial_codex_home / 'config.toml').write_text(config) (trial_codex_home / 'config.toml').chmod(0o600) prompt = args.prompt.read_text() + template_digest = hashlib.sha256(prompt.encode()).hexdigest() + if '{{artifact_path}}' in prompt: + if not args.artifact_path or not os.path.isabs(args.artifact_path): + raise ValueError('A question template requires an absolute --artifact-path') + prompt = prompt.replace('{{artifact_path}}', args.artifact_path) + elif args.artifact_path: + raise ValueError('--artifact-path requires the literal template marker') (args.output / 'prompt.txt').write_text(prompt) (args.output / 'trial-settings.json').write_text(json.dumps({ 'mode': args.mode, 'model': args.model, 'reasoning': args.reasoning, + 'experiment_commit': subprocess.check_output( + ['git', 'rev-parse', 'HEAD'], cwd=repository, text=True).strip(), + 'experiment_sources_sha256': { + name: hashlib.sha256((experiment / name).read_bytes()).hexdigest() + for name in ['agent-trial.py', 'bridge.mjs', 'package-lock.json'] + }, 'developer_instructions': instruction, 'developer_instructions_sha256': hashlib.sha256(instruction.encode()).hexdigest(), 'prompt_sha256': hashlib.sha256(prompt.encode()).hexdigest(), + 'question_template_sha256': template_digest, 'provider_settings_sha256': hashlib.sha256( json.dumps(provider_env, sort_keys=True).encode()).hexdigest(), 'features': {'shell_tool': False, 'unified_exec': False, diff --git a/experiments/codemode/benchmark.mjs b/experiments/codemode/benchmark.mjs index c24843c38..2ce332811 100644 --- a/experiments/codemode/benchmark.mjs +++ b/experiments/codemode/benchmark.mjs @@ -53,6 +53,30 @@ return {artifact_sha256: summary.subject?.digest.sha256, summary: summary.normalized_result, modules: page.normalized_result, evidence_ids: [summary.evidence_id, parent, page.evidence_id], limitations: [...summary.limitations, ...page.limitations]};`, + }, + "obsidian-inventory": { + artifact: "obsidian.asar", + tools: ["open_binary", "inspect_artifact", "close_binary"], + question: + "Report ASAR identity, complete inventory manifest, entry-kind counts, root package/main/preload occurrences, integrity contradictions and inventory coverage; do not claim AST or runtime analysis.", + code: ` +await call("open_binary", {path: input.path}); +try { + const inspected = await call("inspect_artifact", {integrity_policy: "fail"}); + const n = inspected.normalized_result; + const inventory = n.substeps.find(x => x.operation === "inventory_artifact").evidence; + const graph = inventory.normalized_result; + const counts = {}; + for (const row of graph.occurrences) + counts[row.entry_kind] = (counts[row.entry_kind] ?? 0) + 1; + return {artifact_sha256: inspected.subject.digest.sha256, + manifest: graph.manifest, entry_kind_counts: counts, + requested_paths: graph.occurrences.filter(x => ["package.json", "main.js", "preload.js"].includes(x.logical_path)), + integrity_contradictions: graph.integrity_contradictions, + coverage: n.coverage, unexplored_branches: n.unexplored_branches, + evidence_ids: [inspected.evidence_id, inventory.evidence_id], + limitations: [...inspected.limitations, ...n.limitations, ...inventory.limitations]}; +} finally { await call("close_binary", {}); }`, }, "newpipe.apk": { tools: [ @@ -117,6 +141,7 @@ try { const { values } = parseArgs({ options: { artifact: { type: "string" }, + workflow: { type: "string" }, mode: { type: "string", default: "direct" }, "artifact-root": { type: "string" }, providers: { type: "string" }, @@ -124,13 +149,13 @@ const { values } = parseArgs({ }, }); if ( - !workflows[values.artifact] || + !workflows[values.workflow ?? values.artifact] || !["direct", "discovery", "bridge"].includes(values.mode) || !values["artifact-root"] || !values.output ) throw new Error( - "Use --artifact ID --mode direct|discovery|bridge --artifact-root ABS --output NEW_DIR [--providers JSON]", + "Use --artifact ID [--workflow NAME] --mode direct|discovery|bridge --artifact-root ABS --output NEW_DIR [--providers JSON]", ); const experiment = dirname(fileURLToPath(import.meta.url)); const repository = resolve(experiment, "../.."); @@ -138,6 +163,12 @@ const corpus = JSON.parse( await readFile(resolve(experiment, "corpus.json"), "utf8"), ); const artifact = corpus.artifacts.find((x) => x.id === values.artifact); +const workflowName = values.workflow ?? values.artifact; +const workflow = workflows[workflowName]; +if (!artifact || artifact.id !== (workflow.artifact ?? workflowName)) + throw new Error( + "Selected workflow does not match the frozen corpus artifact", + ); const artifactPath = resolve(values["artifact-root"], artifact.file); const bytes = await readFile(artifactPath); const sha256 = createHash("sha256").update(bytes).digest("hex"); @@ -198,7 +229,6 @@ upstream.call = async (name, args, options) => { } }; const bridge = new CodeModeBridge(upstream); -const workflow = workflows[values.artifact]; const input = { path: artifactPath }; if (artifact.id === "jq") { if (bytes.subarray(0, 6).toString("hex") !== "7f454c460201") @@ -281,6 +311,7 @@ const summary = { scope: "deterministic_real_provider_replay_not_model_accuracy", baseline_sha: "fb94d9dde0ccda83ae520ac6fb22b1d283562cf3", artifact: artifact.id, + workflow: workflowName, artifact_sha256: sha256, mode: values.mode, question: workflow.question, diff --git a/experiments/codemode/bridge.mjs b/experiments/codemode/bridge.mjs index 8ebb371fb..1304302eb 100644 --- a/experiments/codemode/bridge.mjs +++ b/experiments/codemode/bridge.mjs @@ -59,9 +59,29 @@ export class CodeModeBridge { this.upstream = upstream; this.byName = new Map(upstream.catalog.map((tool) => [tool.name, tool])); this.catalogDigest = digest(upstream.catalog); + this.declarations = new Map(); this.queue = Promise.resolve(); } + /** The live catalog is immutable for this connection; derive each type once. */ + declarationsFor(tool) { + if (!this.declarations.has(tool.name)) { + const options = { bannerComment: "", format: false, unknownAny: true }; + const pending = Promise.all([ + compile(tool.inputSchema, `${tool.name}_Input`, options), + tool.outputSchema + ? compile( + { ...tool.outputSchema, title: `${tool.name}_Output` }, + `${tool.name}_Output`, + options, + ) + : undefined, + ]); + this.declarations.set(tool.name, pending); + } + return this.declarations.get(tool.name); + } + /** Search exposes original schemas and generated input types on demand. */ async search({ query = "", names = [], schemas, output_schemas = false }) { const includeSchemas = schemas ?? (names.length > 0 || output_schemas); @@ -94,19 +114,16 @@ export class CodeModeBridge { .map(({ tool }) => ({ name: tool.name, description: tool.description })); const tools = []; for (const tool of selected) { - const inputType = includeSchemas - ? await compile(tool.inputSchema, `${tool.name}_Input`, { - bannerComment: "", - format: false, - unknownAny: true, - }) - : undefined; + const [inputType, outputType] = includeSchemas + ? await this.declarationsFor(tool) + : []; tools.push( includeSchemas ? { ...tool, ...(!output_schemas ? { outputSchema: undefined } : {}), typescript: inputType, + output_typescript: outputType, } : { name: tool.name, @@ -206,6 +223,8 @@ export class CodeModeBridge { try { const record = json(reply); if (record.evidence_id) audit.evidence_id = record.evidence_id; + if (Array.isArray(record.evidence_links)) + audit.evidence_links = record.evidence_links; const partial = record.error?.details?.partial_observation; if (partial?.kind === "retained-evidence") audit.partial_evidence_id = partial.evidence_id; @@ -318,7 +337,7 @@ export function createBridgeServer(bridge) { "rea_search", { description: - "Discover original REA tools. Start with one domain word, e.g. query=android; query words must all occur. Query/empty discovery defaults to names and descriptions. Then supply all selected exact names together to obtain original input schemas and TypeScript. schemas=true/false overrides this choice. Request output_schemas=true only when needed. catalog_sha256 hashes the SDK catalog, not REA server_identity.catalog.digests. Code calls tools.(original_arguments); json(reply) parses the original MCP value.", + "Discover original upstream REA tools. Start with one domain word, e.g. query=android; query words must all occur. Query/empty discovery defaults to names and descriptions. Then supply all selected exact names together to obtain exact input schemas plus derived input/output TypeScript for code authoring. schemas=true/false overrides this choice. Request output_schemas=true only for original JSON validation contracts. rea_search and rea_execute are already declared bridge tools, not upstream catalog entries. catalog_sha256 hashes the SDK catalog, not REA server_identity.catalog.digests. Code calls tools.(original_arguments); json(reply) parses the original MCP value.", inputSchema: z.strictObject({ query: z.string().default(""), names: z.array(z.string()).default([]), diff --git a/experiments/codemode/bridge.test.mjs b/experiments/codemode/bridge.test.mjs index 35130fb81..0716a4bff 100644 --- a/experiments/codemode/bridge.test.mjs +++ b/experiments/codemode/bridge.test.mjs @@ -39,12 +39,14 @@ test("discovery preserves exact current REA schemas and names", async () => { const original = upstream.catalog.find( (tool) => tool.name === "binary_session", ); - const { typescript, ...discovered } = result.tools[0]; + const { typescript, output_typescript, ...discovered } = result.tools[0]; assert.deepEqual(discovered, original); assert.match(typescript, /export interface BinarySession_Input/); + assert.match(output_typescript, /export interface BinarySession_Output/); const inputOnly = await bridge.search({ names: ["binary_session"] }); assert.deepEqual(inputOnly.tools[0].inputSchema, original.inputSchema); assert.equal(inputOnly.tools[0].outputSchema, undefined); + assert.equal(inputOnly.tools[0].output_typescript, output_typescript); const all = await bridge.search({ schemas: false }); assert.deepEqual( all.tools.map((tool) => tool.name), @@ -135,6 +137,7 @@ test("error after successful call preserves original Evidence reference", async }); assert.match(result.bridge_error.message, /after-call/); assert.ok(result.calls[0].evidence_id); + assert.ok(result.calls[0].evidence_links.length > 0); const readback = await upstream.call("get_evidence_bundle", { detail: "summary", }); @@ -142,6 +145,15 @@ test("error after successful call preserves original Evidence reference", async assert.ok( JSON.stringify(json(readback)).includes(result.calls[0].evidence_id), ); + for (const evidence_id of result.calls[0].evidence_links) { + const view = json( + await upstream.call("inspect_analysis_view", { + source: { kind: "retained-evidence", evidence_id }, + view: { kind: "summary" }, + }), + ); + assert.equal(view.normalized_result.parent_evidence_id, evidence_id); + } }); test("provider timeout retains its actual partial Evidence reference", async () => { diff --git a/experiments/codemode/grade-trial.py b/experiments/codemode/grade-trial.py index 28b70ccfb..6439b0637 100644 --- a/experiments/codemode/grade-trial.py +++ b/experiments/codemode/grade-trial.py @@ -31,6 +31,28 @@ def read_answer(path): def requested_facts(value, task): + if task == 'native': + return {field: value.get(field) for field in [ + 'artifact_sha256', 'entry_address', 'containing_procedure', + 'procedure', 'pseudocode_prefix', 'caller_count', 'callee_count']} + if task == 'electron-inventory': + return {field: value.get(field) for field in [ + 'artifact_sha256', 'manifest', 'entry_kind_counts', 'requested_paths', + 'integrity_contradictions', 'coverage', 'unexplored_branches']} + if task == 'firmware': + result = value.get('result') + if not isinstance(result, dict): + result = {} + regions = result.get('regions') + if isinstance(regions, list) and all(isinstance(row, dict) for row in regions): + # Binwalk's UUID is run-local; preserve it in the answer but do not + # compare it with the independent replay's different occurrence ID. + regions = [{key: row.get(key) for key in [ + 'offset', 'reported_size', 'signature', 'description', + 'reported_confidence', 'size_basis']} for row in regions] + return {'artifact_sha256': value.get('artifact_sha256'), + 'engine': result.get('engine'), 'input_size': result.get('input_size'), + 'regions': regions, 'coverage': result.get('coverage')} if task == 'managed': def selected(field, keys): row = value.get(field) @@ -57,8 +79,10 @@ def producer_ids(reply): if not isinstance(reply, dict): return set(), 0 ids = evidence_ids(reply.get('evidence_id')) + ids |= evidence_ids(reply.get('evidence_links')) for call in reply.get('calls', []): ids |= evidence_ids(call.get('evidence_id')) + ids |= evidence_ids(call.get('evidence_links')) ids |= evidence_ids(call.get('partial_evidence_id')) unreadable = 0 # SDK envelopes and Codex JSONL use distinct declared field spellings. @@ -84,7 +108,8 @@ def main(): parser = argparse.ArgumentParser() parser.add_argument('--trial', type=Path, required=True) parser.add_argument('--oracle', type=Path, required=True) - parser.add_argument('--task', choices=['android', 'managed'], default='android') + parser.add_argument('--task', choices=[ + 'android', 'managed', 'native', 'firmware', 'electron-inventory'], default='android') args = parser.parse_args() oracle = json.loads(args.oracle.read_text()) if oracle.get('failure') or not oracle.get('projection'): diff --git a/experiments/codemode/questions/android.txt b/experiments/codemode/questions/android.txt new file mode 100644 index 000000000..e303575f7 --- /dev/null +++ b/experiments/codemode/questions/android.txt @@ -0,0 +1,3 @@ +Statically investigate the actual shipped APK at {{artifact_path}} using only the registered REA tools. Report the exact artifact SHA-256, package name, version name/code, every class-name match for the case-sensitive substring MainActivity, and all onCreate method descriptors reported for org.schabi.newpipe.MainActivity. Preserve reported signatures and explicit unknowns. For each onCreate entry include reported_signature, overload_index and dex_descriptor exactly as reported, preserving null. Final answer must be JSON with artifact_sha256, package_name, version_name, version_code, matches, onCreate, evidence_ids, limitations. Include real Evidence IDs supporting each result. Do not execute the APK or use shell, host file tools, network or other agents. Stop once these facts are supported. + +When a client JavaScript executor is available, compose the required REA calls in one execution and return only selected facts from normalized_result, real Evidence IDs, target identity from subject.digest.sha256 and limitations. Parse original MCP text content if necessary. Do not return raw_result, whole Evidence or unrelated inventories. All matching classes and onCreate overloads must still be included. diff --git a/experiments/codemode/questions/electron-inventory.txt b/experiments/codemode/questions/electron-inventory.txt new file mode 100644 index 000000000..bb3e70cf1 --- /dev/null +++ b/experiments/codemode/questions/electron-inventory.txt @@ -0,0 +1 @@ +Statically inventory {{artifact_path}} using only the registered REA tools. Open this ASAR as an archive, inspect its artifact inventory, and close the session. Report exact target SHA-256, the complete inventory manifest, counts of every entry_kind across all occurrences, the original occurrence records for root paths package.json, main.js and preload.js if present, all integrity contradictions, inspection coverage, unexplored branches, real Evidence IDs and limitations. This question asks about package bytes and inventory, not AST interpretation or runtime behavior. Do not execute target code, analyze all JavaScript, extract files, launch a GUI, use shell/file tools/network or other agents. When a client JavaScript executor is available, compose original calls in dependency order and return only requested fields from normalized_result; the full inventory is the Evidence nested in the inspect_artifact inventory_artifact substep. Use Evidence.subject.digest.sha256 for the target identity and retain both inspection and inventory Evidence IDs. Final JSON fields: artifact_sha256, manifest, entry_kind_counts, requested_paths, integrity_contradictions, coverage, unexplored_branches, evidence_ids, limitations. Stop when these facts and cleanup are supported. diff --git a/experiments/codemode/questions/firmware.txt b/experiments/codemode/questions/firmware.txt new file mode 100644 index 000000000..de03fe9ff --- /dev/null +++ b/experiments/codemode/questions/firmware.txt @@ -0,0 +1 @@ +Statically inspect {{artifact_path}} using only the registered REA tools. Report exact target SHA-256 and every reported firmware region with offsets, signatures, reported extents, provider provenance and examined coverage. Preserve unknowns and limitations; a signature scan does not prove partition boundaries or boot/runtime validity. Do not execute target code, extract files, launch a GUI, use shell/file tools/network or other agents. When a client JavaScript executor is available, select the requested normalized_result fields, Evidence.subject.digest.sha256 and real Evidence IDs before returning output. Final JSON fields: artifact_sha256, result, evidence_ids, limitations. Stop when the requested facts are supported. diff --git a/experiments/codemode/questions/managed.txt b/experiments/codemode/questions/managed.txt new file mode 100644 index 000000000..57125c432 --- /dev/null +++ b/experiments/codemode/questions/managed.txt @@ -0,0 +1 @@ +Statically inspect {{artifact_path}} using only the registered REA tools. Report exact target SHA-256, assembly name/version, module name/MVID, managed classification and architecture with its evidence, every reported target framework, and all declared native ModuleRef/PInvoke boundaries with coverage and limitations. Do not load or execute the assembly, launch a GUI, use shell/file tools/network or other agents. When a client JavaScript executor is available, compose the required original MCP calls and return only requested normalized_result fields, target identity from subject.digest.sha256 and real Evidence IDs. Do not return whole Evidence, raw_result or unrelated resources/custom-attribute inventories. Preserve dependency order for operations sharing the selected managed target. Final JSON fields: artifact_sha256, assembly, module, classification, target_frameworks, native_boundaries, coverage, evidence_ids, limitations. Stop when these facts are supported. diff --git a/experiments/codemode/questions/native.txt b/experiments/codemode/questions/native.txt new file mode 100644 index 000000000..2a790b1e6 --- /dev/null +++ b/experiments/codemode/questions/native.txt @@ -0,0 +1 @@ +Statically inspect {{artifact_path}} using only the registered REA tools and Ghidra. The pinned ELF header was independently observed to report entry address 0x403a50; treat that address as the analyst's seed, not as a function-name assumption. Keep default auto-analysis enabled and allow at least 600 seconds for native analysis (including the script deadline if using a bridge). Open the target with provider_id ghidra, resolve the containing procedure for that address, analyze it, and close the ephemeral session in a finally block or a subsequent cleanup call. Report exact target SHA-256, the entry_address, containing_procedure, procedure metadata, the first 512 UTF-16 units of pseudocode, caller_count, callee_count, real Evidence IDs and limitations. Do not execute target code, launch a GUI, use shell/file tools/network or other agents. When a client JavaScript executor is available, compose the original calls in dependency order and select requested fields before returning output; use Evidence.subject.digest.sha256 for the target identity. Final JSON fields: artifact_sha256, entry_address, containing_procedure, procedure, pseudocode_prefix, caller_count, callee_count, evidence_ids, limitations. The address seed applies only to the pinned jq artifact in corpus.json. Stop when these facts and cleanup are supported. diff --git a/experiments/codemode/resource-run.py b/experiments/codemode/resource-run.py index 08db216b1..175ecdeb2 100644 --- a/experiments/codemode/resource-run.py +++ b/experiments/codemode/resource-run.py @@ -63,6 +63,7 @@ def main(): peak = 0 peak_tasks = 0 events = {} + cpu_stats = {} abort = None started = time.monotonic() try: @@ -82,6 +83,8 @@ def main(): try: peak = max(peak, int(current.read_text())) peak_tasks = max(peak_tasks, int((group / 'pids.current').read_text())) + cpu_stats = dict((key, int(value)) for key, value in + (line.split() for line in (group / 'cpu.stat').read_text().splitlines())) for event_file in ['memory.events', 'pids.events']: events[event_file] = dict( (key, int(value)) for key, value in @@ -115,6 +118,7 @@ def main(): 'cpu_affinity': [cpu], 'task_ceiling': 192, 'peak_tasks_sampled': peak_tasks, 'resource_events_last_observed': events, + 'cpu_stat_last_observed': cpu_stats, }), file=sys.stderr) raise SystemExit(125 if abort else code) diff --git a/experiments/codemode/results.md b/experiments/codemode/results.md index 60da7278a..b2c414363 100644 --- a/experiments/codemode/results.md +++ b/experiments/codemode/results.md @@ -209,3 +209,46 @@ exited 143 and left no trial profile while preserving the dummy source auth file This verifies credential cleanup only. The eleven actual stdio tests pass after the adaptive discovery change, with a sampled peak of 1365106688 bytes, 42 tasks and zero observed memory/task limit events. + +## Serial comparison and package inventory + +A serial Android v3 pair, without concurrent provider/build work, passes every +fact and cited-reference check. Normal MCP uses 56.962 seconds, 105861 input +(75776 cached, 30085 uncached), 1106 output tokens and three original calls. The +adaptive-discovery bridge uses 106.134 seconds, 200537 input (175872 cached, +24665 uncached), 2177 output tokens, four searches and one execution. Its smaller +uncached input does not imply lower overall cost: output, cache rates and latency +also matter. It still requests original output schemas and mistakenly searches +for the already exposed bridge executor. Sampled memory peaks are 1333710848 and +1656098816 bytes respectively; neither run touches memory/task limits. One pair +is insufficient for a statistical latency or reliability claim. + +A separate Obsidian package-inventory replay succeeds with existing REA tools. +Direct and bridge return byte-for-byte equal selected facts: 397 graph nodes and +occurrences, 369 files, 28 directories, root main.js/package.json occurrences, +no root preload.js, no observed integrity contradiction, complete-within-substeps +coverage and an explicit unexplored JavaScript analysis branch. Both original +Evidence IDs are preserved. Direct/bridge operations take 6.267/6.618 seconds; +original single-text results total 178958 estimated tokens versus a 1404-token +bridge result, 1305 for discovery and 322 for code. These are replay estimates, +not model latency or success. Inventory does not complete the timed-out AST task. + +An output-type discovery probe compiles seven real SDK schemas successfully. +Derived TypeScript estimates are 784/719/751 tokens for Android package/search/ +class, 1558/1420 for managed artifact/native boundaries, 1264 for artifact inventory +and 2289 for function analysis. Their original JSON schemas use 1666/1480/1576, +3524/2932, 2812 and 5091 respectively. These declarations describe code values; +they do not preserve every JSON Schema validation rule. Exact-name discovery now +includes them without replacing explicitly requested original JSON schemas. Real +model benefit from this addition remains to be tested. The audit also carries +original Evidence links; a real retained-view readback verifies linked IDs rather +than trusting references invented by a projection. + +The output-declaration boundary suite first failed its type-name assertion because +the compiler preferred the schema's URN identity as a display name. The derived +compilation now supplies a descriptive title while returning the original schema +unchanged. All eleven tests then pass, including original-link readback, at a +sampled peak of 1399996416 bytes and zero observed limit events. Declarations are +cached per exact tool name for this immutable live-catalog connection; repeated +discovery does not regenerate them. No provider implementation or published +REA API has changed.