Compare commits

..
Author SHA1 Message Date
Kaoru0822andClaude Opus 5.5 110151e523 fix(capture): keep URLs that WHATWG rejects instead of redacting userinfo
File URLs cannot carry userinfo (an "@" in a file host fails the parse),
and a special authority whose host or port WHATWG rejects fails the whole
URL. Neither has credentials, so the bridge now keeps their source text
and byte sidecar. Checked against Node's URL over 30,000 spellings: no
under-redaction, over-redaction 20,540 -> 5,865 (remaining: empty
userinfo, IPv4 range, IDNA, non-special and baseless references).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:26:29 +00:00
Kaoru0822 b93e585d68 Merge remote-tracking branch 'origin/main' into fix/issue-2671 2026-10-14 00:22:25 +00:00
Xiangfang ChenandGitHub e049ef70f7 feat(setup): add WorkBuddy client support (#3041)
* feat(setup): add WorkBuddy client support

Register rea in the global ~/.workbuddy/mcp.json through a new workbuddy
entry in SUPPORTED_CLIENT_DEFINITIONS (json dialect, ~/.workbuddy marker,
~/.workbuddy/skills skill directory; WorkBuddy AI shares the file).
Document the client, extend the unreleased-client note, and cover
registration and uninstall with boundary tests.

* docs(setup): record verified WorkBuddy tool calls

Real-client verification completed: WorkBuddy loads the registration and
real REA tool calls execute through the registered server.
2026-10-14 08:12:33 +08:00
bbf0638e88 fix(javascript): report EventSource and sendBeacon URLs as network endpoints (#3044)
Co-authored-by: xpeng5278-web <330743750+xpeng5278-web@users.noreply.github.com>
2026-10-14 08:10:04 +08:00
242e2b0001 fix(ghidra): preserve function facet and call coverage (#3002)
* fix(ghidra): preserve observed function facet and call coverage

Declare successful query availability in the Java producer, including explicit unavailable pseudocode reasons. Carry observed static call-set coverage through the parsed provider result and Evidence without inventing facts for legacy omissions. Use Ghidra indexed comment addresses over the function body and entry, retaining cancellation checks and sparse collection.

Closes #2951

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

* fix(ghidra): retain unresolved call coverage in observations

Share decoded unresolved-call facts between dossiers, callee coverage and reference observations. Combine terminal and unresolved observations in one instruction traversal and retain site records only when requested. Keep missing procedure identities incomplete and validate unresolved locations at the provider boundary.

Bind snapshots to the corrected call coverage profile so older complete claims cannot replay under the new semantics. Add real public MCP callback trace/comparison regressions.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

* fix(ghidra): separate reference targets from callee coverage

Reuse containing-function ownership for resolved reference identities and independently compare those identities with the actual returned callee set. Keep known interior routes out of unresolved site records while retaining partial inventory coverage.

Advance the coverage profile to v3 and reject interim v2 cache bindings. Add a real Ghidra producer regression with restored reference metadata; the previous bridge fails the known-route assertion.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

* fix(ghidra): measure incoming call coverage across function bodies

Compare incoming reference source identities across the owned function body and entry with the actual returned caller set. Stream caller-only checks and reuse dossier reference observations. Combine caller/callee coverage from observed counts without reinterpreting serialized metadata, and share the body scope owner with comments and references.

Bind snapshots to function-body coverage semantics v4 and preserve the real off-entry caller regression.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

* fix(ghidra): preserve whole-scope directional call coverage

Check body-and-entry call references independently of decoded instruction flow. Reuse collected dossier references and stream inventory-only coverage checks. Keep caller-only gaps distinct from incomplete outgoing coverage in offline path reconstruction, and invalidate prior snapshot profiles.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

* fix(mcp): retain compact Evidence field types

Share optional diagnostics across the three tagged relationship coverage states instead of advertising duplicate branches. Preserve coverage validation while keeping compact Evidence properties within the existing reference expansion budget. Verify invalid direction cardinalities against the actual SDK-advertised schema.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

---------

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-14 07:52:14 +08:00
morluto 929aa67336 chore(main): merge pinned CI checkpoint 2895a18 2026-10-14 03:12:47 +08:00
morluto 4e8475e965 test(ci): isolate alternate launcher process fixture 2026-10-14 03:02:01 +08:00
morluto 557148ed3f fix(ci): declare process inventory callback 2026-10-14 02:25:19 +08:00
morluto f839046776 chore(merge): refresh PR #2767 with latest main 2026-10-14 02:23:57 +08:00
Morluto 281c4b8ec5 test(ci): isolate Hopper process inventory fixtures 2026-10-14 02:15:33 +08:00
Morluto a0da18d7fa test(capture): cover empty URL userinfo markers 2026-10-14 02:14:51 +08:00
Morluto 2c6e074cac fix(capture): remove empty URL userinfo markers 2026-10-14 02:14:51 +08:00
morluto 29c871e51b chore(merge): refresh PR #2767 with main 2026-10-14 02:10:19 +08:00
REA MaintenanceandKaoru0822 c0018ca403 fix(ci): isolate Linux Hopper process fixture inventory 2026-10-14 01:23:30 +08:00
morlutoandKaoru0822 ad16d250e3 fix(android): make DEX descriptor pattern portable 2026-10-14 01:23:25 +08:00
Lucienandmorluto af2f55cda9 fix(comparison): recheck historical calls-only changes (#2223)
* fix(comparison): preserve incomplete function call evidence

Keep missing, unknown, partial and unresolved call relationships unknown in function comparisons and reconstruction claims. Aggregate IDA caller and callee coverage without overwriting uncertainty.

Validate with 144 focused tests, typecheck/lint, docs checks and test discovery. Real providers were not verified; the full headless IDA session suite requires unavailable Windows native controls.

* fix(comparison): recheck historical calls-only changes

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-14 01:23:19 +08:00
morluto 4d5d1f5daa fix(capture): preserve file-relative path evidence after separators 2026-10-13 23:02:00 +08:00
morluto 0aff997b31 fix(capture): distinguish native paths and URL base authorities 2026-10-13 21:35:04 +08:00
morluto e8361b7715 fix(capture): parse URL controls and mixed network authorities 2026-10-13 20:31:54 +08:00
morluto 72e931f3a3 chore(ci): retain contributor updates alongside verification fixes 2026-10-13 20:21:10 +08:00
morluto 0266e6552b fix(capture): preserve same-scheme relative redirect paths 2026-10-13 20:19:00 +08:00
Kaoru0822 7d972a7e85 Merge remote-tracking branch 'origin/main' into fix/issue-2671 2026-10-13 12:04:14 +00:00
Kaoru0822andClaude Opus 5.5 e1fb40460d fix(capture): skip special-scheme leading slashes before userinfo
Ending the authority at a backslash exposed a credential: for
`https://\user:pass@host/path`, WHATWG parsing skips every leading "/"
or "\" after a special scheme (and in scheme-relative references), so
`user:pass` is userinfo, but the new boundary stopped at the first
backslash and found no "@". The same applied to `https:\\user:pass@host`
and `https:user:pass@host`.

Start the authority after that whole slash run for non-file special
schemes and scheme-relative references. File URLs keep their "//"
boundary because they carry no userinfo. Load the adapter in tests with
fresh mitmproxy stand-ins scoped to its import instead of leaving them
in sys.modules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 11:25:05 +00:00
Kaoru0822andClaude Opus 5.5 553cd6e358 fix(capture): end special-scheme URL authorities at a backslash
The native mitmproxy adapter removed transport userinfo with
`^(...//)[^/?#]*@`, so the authority ran to the last "@" before "/", "?"
or "#". For `https://example.test\part@tail` it treated
`example.test\part` as userinfo, reported `https://tail` as the
Location, and withdrew the original bytes as a transport credential.
WHATWG URL parsing, used by the shared TypeScript owner
(recordedTransportUrl.ts) and by HAR captures, ends a special-scheme or
scheme-relative authority at a backslash, so that "@" is ordinary path
text.

Locate the authority with the same boundary rule before removing
userinfo. Bytes are handled as Latin-1 text so unrelated bytes stay
identical. Userinfo before the backslash is still removed and redacted.
Add source unit tests for the adapter projection to test:python.

Fixes #2671

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 11:06:25 +00:00
46 changed files with 1836 additions and 1051 deletions
+282 -97
View File
@@ -81,6 +81,7 @@ import ghidra.program.model.listing.Function;
import ghidra.program.model.listing.FunctionIterator;
import ghidra.program.model.listing.Instruction;
import ghidra.program.model.listing.InstructionIterator;
import ghidra.program.model.listing.Listing;
import ghidra.program.model.listing.Variable;
import ghidra.program.model.mem.MemoryBlock;
import ghidra.program.model.mem.MemoryBlockSourceInfo;
@@ -405,7 +406,9 @@ public final class ReaGhidraBridge extends HeadlessScript {
case "xrefs" -> xrefs(request.params);
case "analyze_function" -> {
FunctionAnalysisObservation observation = analyzeFunctionObservation(request.params);
yield functionObservation(observation.dossier(), observation.limitations());
JsonObject result = functionObservation(observation.dossier(), observation.limitations());
result.add("relationship_coverage", observation.relationshipCoverage());
yield result;
}
case "set_address_name", "set_addresses_names" -> {
if (!descriptor.transport.equals("unix-socket"))
@@ -1016,12 +1019,19 @@ public final class ReaGhidraBridge extends HeadlessScript {
for (Function item : ordered) {
result.add(canonicalAddress(item.getEntryPoint()));
}
return functionObservation(
result,
noReturnCallLimitations(
if (callers) {
JsonObject observation = functionObservation(result, noReturnCallLimitations(
currentProgram.getListing().getInstructions(function.getBody(), true)
)
));
observation.add("relationship_coverage", callRelationshipCoverage(null,
unrepresentedIncomingCalls(function, observed)));
monitor.checkCancelled();
return observation;
}
CallBoundaryObservation calls = callBoundaryObservation(
function, currentProgram.getListing().getInstructions(function.getBody(), true), observed, false, null
);
return callRelationshipObservation(result, calls);
}
private JsonObject procedureInfo(JsonObject params) throws Exception {
@@ -1071,18 +1081,10 @@ public final class ReaGhidraBridge extends HeadlessScript {
result.addProperty("direction", direction);
result.add("references", edges);
result.addProperty("reference_kinds_available", true);
JsonArray unresolvedCalls = new JsonArray();
if (direction.equals("outgoing")) for (Instruction instruction : scanInstructions(function, Integer.MAX_VALUE).instructions) {
if (!instruction.getFlowType().isCall()) continue;
boolean resolved = false;
for (Reference reference : instruction.getReferencesFrom()) if (reference.getReferenceType().isCall()) resolved = true;
if (!resolved) {
JsonObject site = new JsonObject();
site.addProperty("address", canonicalAddress(instruction.getAddress()));
site.addProperty("reason", "Instruction decoder identifies a call but ReferenceManager supplies no call destination");
unresolvedCalls.add(site);
}
}
JsonArray unresolvedCalls = direction.equals("outgoing")
? callBoundaryObservation(
function, currentProgram.getListing().getInstructions(function.getBody(), true), null, true, references
).unresolvedCalls() : new JsonArray();
result.add("unresolved_calls", unresolvedCalls);
return result;
}
@@ -1490,16 +1492,28 @@ public final class ReaGhidraBridge extends HeadlessScript {
InstructionScan scan = scanInstructions(function, Integer.MAX_VALUE);
DecompileResults decompilation = decompile(function);
String pseudocode = decompiledText(decompilation);
List<Reference> incomingReferences = collectReferences(function, "incoming")
.stream()
JsonObject availability = new JsonObject();
availability.add("pseudocode", facetAvailability(
pseudocode != null,
function.isExternal() ? "Ghidra does not decompile external functions."
: function.getBody().isEmpty() ? "Ghidra has no function body to decompile."
: "Ghidra completed decompilation without returning pseudocode."
));
if (pseudocode == null) {
pseudocode = "";
}
List<Reference> bodyIncoming = collectReferences(function, "incoming");
List<Reference> incomingReferences = bodyIncoming.stream()
.filter(reference -> !function.getBody().contains(reference.getFromAddress()))
.toList();
List<Reference> outgoingReferences = collectReferences(function, "outgoing");
JsonArray incoming = referenceEdges(incomingReferences);
JsonArray outgoing = referenceEdges(outgoingReferences);
JsonArray comments = comments(scan.instructions);
JsonArray callers = procedureIdentities(function.getCallingFunctions(monitor));
JsonArray callees = procedureIdentities(function.getCalledFunctions(monitor));
JsonArray comments = comments(function);
Set<Function> callerFunctions = function.getCallingFunctions(monitor);
JsonArray callers = procedureIdentities(callerFunctions);
Set<Function> calleeFunctions = function.getCalledFunctions(monitor);
JsonArray callees = procedureIdentities(calleeFunctions);
JsonArray referencedStrings = referencedStrings(outgoingReferences);
JsonArray referencedNames = referencedNames(outgoingReferences);
JsonArray blocks = basicBlockValues(function);
@@ -1511,7 +1525,7 @@ public final class ReaGhidraBridge extends HeadlessScript {
procedure.add("locals", functionLocals(function));
JsonObject result = new JsonObject();
result.add("procedure", procedure);
result.addProperty("pseudocode", pseudocode == null ? "" : pseudocode);
result.addProperty("pseudocode", pseudocode);
result.add("assembly", assembly);
result.add("comments", comments);
result.add("callers", callers);
@@ -1520,20 +1534,6 @@ public final class ReaGhidraBridge extends HeadlessScript {
result.add("outgoing_references", outgoing);
result.add("referenced_strings", referencedStrings);
result.add("referenced_names", referencedNames);
JsonObject facets = new JsonObject();
JsonObject pseudocodeAvailability = new JsonObject();
pseudocodeAvailability.addProperty("available", pseudocode != null);
if (pseudocode == null) {
pseudocodeAvailability.addProperty("reason", "Ghidra did not supply pseudocode.");
}
facets.add("pseudocode", pseudocodeAvailability);
for (String facet : List.of("assembly", "comments", "callers", "callees",
"references", "strings_names", "cfg")) {
JsonObject available = new JsonObject();
available.addProperty("available", true);
facets.add(facet, available);
}
result.add("facet_availability", facets);
boolean swift = swiftProgram();
boolean aarch64 = currentProgram.getLanguageID().getIdAsString().startsWith("AARCH64:");
SwiftStringLiteralDecode swiftLiterals = swift && aarch64 ? swiftStringLiterals(function) : null;
@@ -1547,6 +1547,15 @@ public final class ReaGhidraBridge extends HeadlessScript {
nativeApiBoundary(function, decompilation)
);
result.add("native_value_flow", nativeValueFlow(decompilation));
// Successful full queries distinguish collected empty sets from missing facts.
// Availability does not claim that Ghidra resolved every runtime relationship.
for (String facet : List.of("comments", "callers", "callees", "references", "strings_names", "cfg")) {
availability.add(facet, facetAvailability(true, null));
}
availability.add("assembly", facetAvailability(
!scan.truncated(), "The function instruction scan was truncated."
));
result.add("facet_availability", availability);
JsonArray limitations = new JsonArray();
limitations.add(
"Unresolved computed or indirect flows without target addresses are not represented as reference edges."
@@ -1574,8 +1583,10 @@ public final class ReaGhidraBridge extends HeadlessScript {
"Swift String literal decoding is implemented for AArch64 register pairs only; no Swift literals were decoded for this architecture."
);
}
List<String> boundaryLimitations = noReturnCallLimitations(scan.instructions);
for (String limitation : boundaryLimitations) {
CallBoundaryObservation calls = callBoundaryObservation(
function, scan.instructions, calleeFunctions, true, outgoingReferences);
result.add("unresolved_calls", calls.unresolvedCalls());
for (String limitation : calls.limitations()) {
limitations.add(limitation);
}
if (decompilation != null && decompilation.getErrorMessage() != null &&
@@ -1583,7 +1594,14 @@ public final class ReaGhidraBridge extends HeadlessScript {
limitations.add("Ghidra decompiler diagnostic: " + decompilation.getErrorMessage());
}
result.add("limitations", limitations);
return new FunctionAnalysisObservation(result, boundaryLimitations);
monitor.checkCancelled();
int missingCallers = 0;
for (Reference reference : bodyIncoming) {
monitor.checkCancelled();
if (unrepresentedCaller(reference, callerFunctions)) missingCallers++;
}
return new FunctionAnalysisObservation(result, calls.limitations(),
callRelationshipCoverage(calls, missingCallers));
}
private JsonObject nativeValueFlow(DecompileResults decompilation) throws Exception {
@@ -3287,54 +3305,202 @@ public final class ReaGhidraBridge extends HeadlessScript {
return result;
}
private static JsonObject facetAvailability(boolean available, String reason) {
JsonObject result = new JsonObject();
result.addProperty("available", available);
if (!available) result.addProperty("reason", reason);
return result;
}
private JsonObject callRelationshipObservation(JsonElement value, CallBoundaryObservation calls)
throws Exception {
monitor.checkCancelled();
JsonObject result = functionObservation(value, calls.limitations());
result.add("relationship_coverage", callRelationshipCoverage(calls, null));
return result;
}
private static JsonObject callRelationshipCoverage(
CallBoundaryObservation calls, Integer missingCallers) {
JsonObject coverage = completeResolvedCallCoverage();
Integer omitted = calls == null ? null : calls.unrepresentedCalleeCalls();
JsonArray directions = new JsonArray();
if (missingCallers != null && missingCallers > 0) directions.add("callers");
String callerLimitation = missingCallers != null && missingCallers > 0
? "Ghidra's caller inventory omits sources of " + missingCallers +
" call reference(s) into this function's body." : null;
if ((calls != null && omitted == null) || (calls == null && missingCallers == null)) {
coverage.addProperty("status", "unknown");
String reason = calls == null ? "No call inventory was examined."
: "The callee inventory was not examined alongside decoded call targets.";
coverage.addProperty("reason", reason + (callerLimitation == null ? "" : " " + callerLimitation));
if (calls == null && missingCallers == null) directions.add("callers");
directions.add("callees");
coverage.add("limited_directions", directions);
return coverage;
}
JsonArray limitations = new JsonArray();
if (callerLimitation != null) limitations.add(callerLimitation);
if (omitted != null && omitted > 0) {
limitations.add("Ghidra's callee inventory omits targets of " + omitted +
" call reference(s) or decoded call(s) without destinations.");
directions.add("callees");
}
if (!limitations.isEmpty()) {
coverage.addProperty("status", "partial");
coverage.add("limitations", limitations);
coverage.add("limited_directions", directions);
}
return coverage;
}
private static JsonObject completeResolvedCallCoverage() {
JsonObject coverage = new JsonObject();
coverage.addProperty("status", "complete");
JsonArray diagnostics = new JsonArray();
diagnostics.add(
"Only resolved Ghidra call references are returned; unresolved computed or indirect calls remain unknown, while function classifications distinguish thunks and externals."
);
coverage.add("diagnostics", diagnostics);
return coverage;
}
private boolean unrepresentedCaller(Reference reference, Set<Function> callers) {
if (!reference.getReferenceType().isCall()) return false;
Function caller = containingFunction(reference.getFromAddress());
return caller == null || !callers.contains(caller);
}
private int unrepresentedIncomingCalls(Function function, Set<Function> callers)
throws Exception {
int[] omitted = { 0 };
visitReferences(function, "incoming", reference -> {
if (unrepresentedCaller(reference, callers)) omitted[0]++;
});
return omitted[0];
}
private List<String> noReturnCallLimitations(Iterable<Instruction> instructions)
throws Exception {
List<String> limitations = new ArrayList<>();
for (Instruction instruction : instructions) {
monitor.checkCancelled();
if (!instruction.getFlowType().isCall() || !instruction.getFlowType().isTerminal()) {
continue;
}
Address[] destinations = instruction.getFlows();
StringBuilder limitation = new StringBuilder();
limitation.append("Ghidra Listing reports a terminal call at ")
.append(canonicalAddress(instruction.getAddress()));
if (destinations.length == 0) {
limitation.append(" with no resolved callee; its no-return basis is unknown.");
}
else {
limitation.append(" to ");
for (int index = 0; index < destinations.length; index++) {
if (index > 0) limitation.append(", ");
Address destination = destinations[index];
limitation.append(canonicalAddress(destination));
Function target = currentProgram.getFunctionManager().getFunctionAt(destination);
if (target == null) {
limitation.append(" (callee identity unknown)");
}
else {
limitation.append(" ").append(procedureName(target));
limitation.append(" (external=").append(target.isExternal());
limitation.append(", thunk=").append(target.isThunk());
limitation.append(", hasNoReturn=").append(target.hasNoReturn()).append(")");
}
}
limitation.append("; fallthrough is excluded by Ghidra's flow model.");
Function target = destinations.length == 1
? currentProgram.getFunctionManager().getFunctionAt(destinations[0])
: null;
if (target != null && target.hasNoReturn()) {
limitation.append(" This records Ghidra's FunctionManager flag, not an independent verification that the callee cannot return.");
}
else {
limitation.append(" The reason for the terminal flow is not established by this observation.");
}
}
limitations.add(limitation.toString());
String limitation = terminalCallLimitation(instruction);
if (limitation != null) limitations.add(limitation);
}
return limitations;
}
private CallBoundaryObservation callBoundaryObservation(
Function function, Iterable<Instruction> instructions, Set<Function> returnedCallees,
boolean includeUnresolvedSites, List<Reference> observedReferences)
throws Exception {
List<String> limitations = new ArrayList<>();
CallBoundaryReferences calls = new CallBoundaryReferences(returnedCallees, includeUnresolvedSites);
if (observedReferences == null) visitReferences(function, "outgoing", calls::observe);
else for (Reference reference : observedReferences) {
monitor.checkCancelled();
calls.observe(reference);
}
for (Instruction instruction : instructions) {
monitor.checkCancelled();
String limitation = terminalCallLimitation(instruction);
if (limitation != null) limitations.add(limitation);
if (!instruction.getFlowType().isCall()) continue;
boolean destination = false;
for (Reference reference : instruction.getReferencesFrom()) {
if (!reference.getReferenceType().isCall()) continue;
destination = true;
break;
}
if (!destination) {
calls.unrepresented++;
calls.unresolved(instruction.getAddress(),
"Instruction decoder identifies a call but ReferenceManager supplies no call destination");
}
}
return new CallBoundaryObservation(List.copyOf(limitations),
returnedCallees == null ? null : calls.unrepresented, calls.unresolvedCalls());
}
private final class CallBoundaryReferences {
private final Set<Function> returnedCallees;
private final TreeMap<Address, String> unresolved;
private int unrepresented;
CallBoundaryReferences(Set<Function> returnedCallees, boolean includeUnresolvedSites) {
this.returnedCallees = returnedCallees;
unresolved = includeUnresolvedSites ? new TreeMap<>() : null;
}
void observe(Reference reference) {
if (!reference.getReferenceType().isCall()) return;
// Reference identity follows the same containing-function owner as referenceEdge.
Function target = containingFunction(reference.getToAddress());
if (returnedCallees != null && (target == null || !returnedCallees.contains(target)))
unrepresented++;
if (target == null) unresolved(reference.getFromAddress(),
"A call destination has no known containing Ghidra procedure");
}
void unresolved(Address address, String reason) {
if (unresolved != null) unresolved.put(address, reason);
}
JsonArray unresolvedCalls() {
if (unresolved == null) return null;
JsonArray result = new JsonArray();
for (var item : unresolved.entrySet()) {
JsonObject site = new JsonObject();
site.addProperty("address", canonicalAddress(item.getKey()));
site.addProperty("reason", item.getValue());
result.add(site);
}
return result;
}
}
private String terminalCallLimitation(Instruction instruction) {
if (!instruction.getFlowType().isCall() || !instruction.getFlowType().isTerminal())
return null;
Address[] destinations = instruction.getFlows();
StringBuilder limitation = new StringBuilder();
limitation.append("Ghidra Listing reports a terminal call at ")
.append(canonicalAddress(instruction.getAddress()));
if (destinations.length == 0) {
limitation.append(" with no resolved callee; its no-return basis is unknown.");
}
else {
limitation.append(" to ");
for (int index = 0; index < destinations.length; index++) {
if (index > 0) limitation.append(", ");
Address destination = destinations[index];
limitation.append(canonicalAddress(destination));
Function target = currentProgram.getFunctionManager().getFunctionAt(destination);
if (target == null) {
limitation.append(" (callee identity unknown)");
}
else {
limitation.append(" ").append(procedureName(target));
limitation.append(" (external=").append(target.isExternal());
limitation.append(", thunk=").append(target.isThunk());
limitation.append(", hasNoReturn=").append(target.hasNoReturn()).append(")");
}
}
limitation.append("; fallthrough is excluded by Ghidra's flow model.");
Function target = destinations.length == 1
? currentProgram.getFunctionManager().getFunctionAt(destinations[0])
: null;
if (target != null && target.hasNoReturn()) {
limitation.append(" This records Ghidra's FunctionManager flag, not an independent verification that the callee cannot return.");
}
else {
limitation.append(" The reason for the terminal flow is not established by this observation.");
}
}
return limitation.toString();
}
private static String renderAssembly(List<Instruction> instructions) {
return String.join("\n", renderAssemblyLines(instructions));
}
@@ -3358,15 +3524,27 @@ public final class ReaGhidraBridge extends HeadlessScript {
private List<Reference> collectReferences(
Function function,
String direction) throws Exception {
TreeMap<String, Reference> observed = new TreeMap<>();
visitReferences(function, direction, reference -> addReference(observed, reference));
List<Reference> result = new ArrayList<>(observed.values());
result.sort(REFERENCE_ORDER);
return result;
}
@FunctionalInterface
private interface ReferenceVisitor {
void accept(Reference reference) throws Exception;
}
private void visitReferences(Function function, String direction, ReferenceVisitor visitor)
throws Exception {
// References can target instruction interiors or originate in embedded
// data. Use actual body ownership, including an external's exact entry.
AddressSet scope = new AddressSet(function.getBody());
scope.add(function.getEntryPoint());
AddressSet scope = functionObservationScope(function);
boolean outgoing = direction.equals("outgoing");
AddressIterator addresses = outgoing
? currentProgram.getReferenceManager().getReferenceSourceIterator(scope, true)
: currentProgram.getReferenceManager().getReferenceDestinationIterator(scope, true);
TreeMap<String, Reference> observed = new TreeMap<>();
while (addresses.hasNext()) {
monitor.checkCancelled();
Address address = addresses.next();
@@ -3374,7 +3552,7 @@ public final class ReaGhidraBridge extends HeadlessScript {
for (Reference reference : currentProgram.getReferenceManager()
.getReferencesFrom(address)) {
monitor.checkCancelled();
addReference(observed, reference);
if (!reference.isEntryPointReference()) visitor.accept(reference);
}
}
else {
@@ -3382,19 +3560,14 @@ public final class ReaGhidraBridge extends HeadlessScript {
.getReferencesTo(address);
while (iterator.hasNext()) {
monitor.checkCancelled();
addReference(observed, iterator.next());
Reference reference = iterator.next();
if (!reference.isEntryPointReference()) visitor.accept(reference);
}
}
}
List<Reference> result = new ArrayList<>(observed.values());
result.sort(REFERENCE_ORDER);
return result;
}
private static void addReference(TreeMap<String, Reference> observed, Reference reference) {
if (reference.isEntryPointReference()) {
return;
}
String key = canonicalAddress(reference.getFromAddress()) + "\u0000" +
canonicalAddress(reference.getToAddress()) + "\u0000" +
reference.getReferenceType().getName() + "\u0000" +
@@ -3403,6 +3576,12 @@ public final class ReaGhidraBridge extends HeadlessScript {
observed.putIfAbsent(key, reference);
}
private static AddressSet functionObservationScope(Function function) {
AddressSet scope = new AddressSet(function.getBody());
scope.add(function.getEntryPoint());
return scope;
}
private JsonArray referenceEdges(List<Reference> references) {
JsonArray result = new JsonArray();
for (Reference reference : references) {
@@ -3556,23 +3735,28 @@ public final class ReaGhidraBridge extends HeadlessScript {
return result;
}
private static JsonArray comments(List<Instruction> instructions) {
private JsonArray comments(Function function) throws Exception {
Listing listing = currentProgram.getListing();
AddressSet scope = functionObservationScope(function);
AddressIterator addresses = listing.getCommentAddressIterator(scope, true);
TreeMap<String, JsonObject> observed = new TreeMap<>();
for (Instruction instruction : instructions) {
while (addresses.hasNext()) {
monitor.checkCancelled();
Address address = addresses.next();
for (CommentType type : CommentType.values()) {
String text = instruction.getComment(type);
String text = listing.getComment(type, address);
if (text == null || text.isEmpty()) {
continue;
}
String kind = type == CommentType.EOL ? "inline" : "comment";
JsonObject item = new JsonObject();
item.addProperty("address", canonicalAddress(instruction.getAddress()));
item.addProperty("address", canonicalAddress(address));
item.addProperty("kind", kind);
item.addProperty("text", text);
String key = canonicalAddress(instruction.getAddress()) + "\u0000" +
String key = canonicalAddress(address) + "\u0000" +
kind + "\u0000" + type.name() + "\u0000" + text;
observed.put(key, item);
}
}
}
JsonArray result = new JsonArray();
observed.values().forEach(result::add);
@@ -4508,7 +4692,8 @@ public final class ReaGhidraBridge extends HeadlessScript {
private record InventoryItem(Address address, String value, JsonObject facts) {}
private record AddressNameRequest(String inputAddress, Address address, String name) {}
private record FunctionEntry(Function function, InventoryItem item) {}
private record FunctionAnalysisObservation(JsonObject dossier, List<String> limitations) {}
private record FunctionAnalysisObservation(JsonObject dossier, List<String> limitations, JsonObject relationshipCoverage) {}
private record CallBoundaryObservation(List<String> limitations, Integer unrepresentedCalleeCalls, JsonArray unresolvedCalls) {}
private record SwiftStringLiteralDecode(JsonArray literals, JsonArray omissions) {}
private record SwiftLiteralLayout(
String representation,
+133 -12
View File
@@ -11,6 +11,7 @@ import math
import re
import resource
from pathlib import Path
from urllib.parse import unquote_to_bytes
from mitmproxy import ctx, version
from mitmproxy.io import tnetstring
@@ -41,6 +42,113 @@ class BoundedReader:
return self.handle.read(count)
SPECIAL_SCHEMES = {"ftp", "file", "http", "https", "ws", "wss"}
SCHEME_PREFIX = r"^[\x00-\x20]*([a-z][a-z0-9+.-]*):"
NETWORK_PREFIX = r"^[\x00-\x20]*[/\\]{2}"
URL_IGNORABLE_CONTROLS = "\t\n\r"
URL_CONTROL_TRANSLATION = {ord(char): None for char in URL_IGNORABLE_CONTROLS}
FORBIDDEN_DOMAIN_CODE_POINTS = {chr(code) for code in range(0x21)} | set("#%/:<>?@[\\]^|\x7f")
def authority_start(text, base_scheme=None):
"""Return where an authority begins and whether it is special, or None.
Follows WHATWG URL parsing as src/domain/recordedTransportUrl.ts does:
special schemes (and scheme-relative references) skip every leading "/"
or "\\" before the authority, so `https://\\user:pass@host` still has
userinfo. File URLs never carry userinfo: an "@" in a file host makes
WHATWG reject the whole URL, so there are no credentials to remove.
"""
scheme = re.match(SCHEME_PREFIX, text, flags=re.I)
name = scheme.group(1).lower() if scheme is not None else None
if name == "file" or (name is None and base_scheme == "file"):
return None
if name is not None and name in SPECIAL_SCHEMES:
# A same-scheme Location without two slash-equivalent separators is
# relative to the recorded request, so its @ belongs to the path.
following = text[scheme.end():scheme.end() + 2]
if name == base_scheme and (len(following) < 2 or any(char not in "/\\" for char in following)):
return None
start = scheme.end()
else:
if scheme is not None:
if text[scheme.end():scheme.end() + 2] != "//":
return None
return scheme.end() + 2, False
# A non-special base treats backslashes as authority data and only
# recognizes two forward slashes as a network reference.
if base_scheme is not None and base_scheme not in SPECIAL_SCHEMES:
prefix = re.match(r"^[\x00-\x20]*//", text)
return None if prefix is None else (prefix.end(), False)
prefix = re.match(NETWORK_PREFIX, text)
if prefix is None:
return None
start = prefix.end()
while start < len(text) and text[start] in "/\\":
start += 1
return start, True
def special_host_parses(host_port):
"""Whether WHATWG could accept a special authority's host and port.
A rejected host fails the whole URL, so it has no credentials to remove.
This mirrors the forbidden-code-point, empty-host and port rules; IDNA,
IPv4 range and IPv6 syntax failures are not modeled and still redact.
"""
if host_port.startswith("["):
host_end = host_port.find("]") + 1
if host_end == 0:
return False
else:
colon = host_port.find(":")
host_end = len(host_port) if colon < 0 else colon
host = unquote_to_bytes(host_port[:host_end]).decode("utf-8", errors="replace")
if host == "" or any(char in FORBIDDEN_DOMAIN_CODE_POINTS for char in host):
return False
port = host_port[host_end:]
if port == "":
return True
digits = port[1:]
return port[0] == ":" and (digits == "" or (digits.isascii() and digits.isdigit() and int(digits) <= 65535))
def without_userinfo(value, base_scheme=None):
"""Remove an authority's userinfo using the WHATWG authority boundary.
Special schemes and scheme-relative references also end the authority
at a backslash, so later path text containing "@" is not userinfo.
"""
binary = isinstance(value, bytes)
text = value.decode("latin-1") if binary else value
# WHATWG removes these controls before interpreting scheme/separators.
# Keep that parse view separate from the exact source spelling and bytes.
parsed_text = text.translate(URL_CONTROL_TRANSLATION)
located = authority_start(parsed_text, base_scheme)
if located is None:
return value
start, special = located
terminators = "/?#\\" if special else "/?#"
end = start
while end < len(parsed_text) and parsed_text[end] not in terminators:
end += 1
at = parsed_text.rfind("@", start, end)
if at < start or (special and not special_host_parses(parsed_text[at + 1 : end])):
return value
parsed_offset = 0
raw_start = 0
for raw_offset, char in enumerate(text):
if char in URL_IGNORABLE_CONTROLS:
continue
if parsed_offset == start:
raw_start = raw_offset
if parsed_offset == at:
safe = text[:raw_start] + text[raw_offset + 1 :]
return safe.encode("latin-1") if binary else safe
parsed_offset += 1
raise CaptureFailure("decoder", "Normalized URL authority could not be bound to its source.")
def project_record(state, secrets):
binaries, numbers, redactions = [], [], []
byte_secrets = []
@@ -58,16 +166,23 @@ def project_record(state, secrets):
return None
return text
def transport_value(value, path, transport_url):
def transport_value(value, path, transport_url, base_scheme):
if not (transport_url or re.fullmatch(r"/(?:backup/)*request/(?:path|authority)", path)):
return value, False
binary = isinstance(value, bytes)
pattern = rb"^([\x00-\x20]*(?:[a-z][a-z0-9+.-]*:)?//)[^/?#]*@" if binary else r"^([\x00-\x20]*(?:[a-z][a-z0-9+.-]*:)?//)[^/?#]*@"
replacement = rb"\1" if binary else r"\1"
separator = b"@" if binary else "@"
safe_url = re.sub(pattern, replacement, value, flags=re.I)
if path.endswith("/authority") and separator in safe_url:
safe_url = safe_url.rsplit(separator, 1)[1]
if transport_url:
safe_url = without_userinfo(value, base_scheme)
elif path.endswith("/authority"):
safe_url = value.rsplit(separator, 1)[-1]
else:
# Native request.path is origin-form unless it explicitly carries
# a scheme (absolute-form). Leading separators are path evidence,
# not a header URL reference's network authority.
text = value.decode("latin-1") if binary else value
safe_url = without_userinfo(value) if re.match(
SCHEME_PREFIX, text.translate(URL_CONTROL_TRANSLATION), flags=re.I
) else value
changed = safe_url != value
if changed:
redactions.append({"pointer": path, "reason": "transport-credential"})
@@ -80,7 +195,7 @@ def project_record(state, secrets):
del numbers[before[1]:]
del redactions[before[2]:]
def visit(value, path, depth=0, credential=False, transport_url=False):
def visit(value, path, depth=0, credential=False, transport_url=False, base_scheme=None):
if depth > 64:
raise CaptureFailure("input-limit", "Native capture exceeds the 64-level evidence nesting budget.", path)
if credential:
@@ -90,7 +205,7 @@ def project_record(state, secrets):
binaries.append({"pointer": path, "representation": "producer-bytes", "state": "redacted", "content_base64": None, "bytes": None, "sha256": None})
return None
if isinstance(value, bytes):
safe_bytes, credential_url = transport_value(value, path, transport_url)
safe_bytes, credential_url = transport_value(value, path, transport_url, base_scheme)
content_base64 = base64.b64encode(value).decode("ascii")
digest = hashlib.sha256(value).hexdigest()
hidden = any(secret in value or secret in safe_bytes for secret in byte_secrets) or any(secret in content_base64 or secret in digest or secret in str(len(value)) for secret in secrets)
@@ -107,7 +222,7 @@ def project_record(state, secrets):
return None
return safe_text
if isinstance(value, str):
safe_text, _ = transport_value(value, path, transport_url)
safe_text, _ = transport_value(value, path, transport_url, base_scheme)
return redact(safe_text, path, value)
if isinstance(value, bool) or value is None:
return value
@@ -125,10 +240,16 @@ def project_record(state, secrets):
for index, field in enumerate(value):
if not isinstance(field, (list, tuple)) or len(field) != 2 or not isinstance(field[0], bytes):
raise CaptureFailure("format", "Native header/trailer is not an ordered byte pair.", pointer(path, index))
result.append([visit(field[0], pointer(pointer(path, index), 0), depth + 2), visit(field[1], pointer(pointer(path, index), 1), depth + 2, field[0].lower() in CREDENTIALS, field[0].lower() in {b"location", b"referer", b"origin"})])
result.append([visit(field[0], pointer(pointer(path, index), 0), depth + 2), visit(field[1], pointer(pointer(path, index), 1), depth + 2, field[0].lower() in CREDENTIALS, field[0].lower() in {b"location", b"referer", b"origin"}, base_scheme if field[0].lower() == b"location" else None)])
return result
return [visit(item, pointer(path, index), depth + 1) for index, item in enumerate(value)]
return [visit(item, pointer(path, index), depth + 1, base_scheme=base_scheme) for index, item in enumerate(value)]
if isinstance(value, dict):
if path == "" or re.fullmatch(r"/backup(?:/backup)*", path):
request = value.get("request")
scheme = request.get("scheme") if isinstance(request, dict) else None
if isinstance(scheme, bytes):
scheme = scheme.decode("latin-1")
base_scheme = scheme.lower() if isinstance(scheme, str) and re.fullmatch(r"[a-z][a-z0-9+.-]*", scheme, flags=re.I) else None
result = {}
for key, item in value.items():
if not isinstance(key, str):
@@ -139,7 +260,7 @@ def project_record(state, secrets):
validate_omitted(item, pointer(path, key), depth + 1)
redactions.append({"pointer": path, "reason": "explicit-sensitive-value", "scope": "property-name"})
continue
result[key] = visit(item, pointer(path, key), depth + 1)
result[key] = visit(item, pointer(path, key), depth + 1, base_scheme=base_scheme)
return result
raise CaptureFailure("decoder", "Native parser returned an unexpected value type: " + type(value).__name__ + ".", path)
+214
View File
@@ -0,0 +1,214 @@
"""Source-owned tests for the mitmproxy adapter's record projection."""
import importlib.util
import os
import sys
import types
import unittest
from unittest import mock
def load_capture():
"""Load the adapter with fresh mitmproxy stand-ins scoped to its import."""
fake_mitmproxy = types.ModuleType("mitmproxy")
fake_mitmproxy.ctx = types.SimpleNamespace()
fake_mitmproxy.version = types.SimpleNamespace(VERSION="12.2.3")
fake_io = types.ModuleType("mitmproxy.io")
fake_io.tnetstring = types.ModuleType("mitmproxy.io.tnetstring")
stand_ins = {
"mitmproxy": fake_mitmproxy,
"mitmproxy.io": fake_io,
"mitmproxy.io.tnetstring": fake_io.tnetstring,
}
spec = importlib.util.spec_from_file_location(
"rea_mitmproxy_capture", os.path.join(os.path.dirname(__file__), "capture.py")
)
module = importlib.util.module_from_spec(spec)
with mock.patch.dict(sys.modules, stand_ins):
spec.loader.exec_module(module)
return module
capture = load_capture()
def location(value, scheme=None):
state = {"response": {"headers": [[b"Location", value]]}}
if scheme is not None:
state["request"] = {"scheme": scheme}
record = capture.project_record(state, [])
sidecar = next(
field for field in record["binary_fields"] if field["pointer"] == "/response/headers/0/1"
)
return record["reported"]["response"]["headers"][0][1], sidecar, record["redactions"]
class TransportUserinfoTests(unittest.TestCase):
def test_ignorable_controls_cannot_hide_authority_credentials(self):
for control in [b"\t", b"\n", b"\r"]:
for value in [b"https://" + control + b"/user:pass@host/path", b"ht" + control + b"tps://user:" + control + b"pass@host/path"]:
with self.subTest(value=value):
text, sidecar, redactions = location(value, b"https")
self.assertNotIn("user", text)
self.assertNotIn("pass", text)
self.assertEqual(sidecar["state"], "redacted")
self.assertEqual(len(redactions), 1)
def test_mixed_network_separators_remove_credentials(self):
for prefix in [b"\\\\", b"/\\", b"\\/"]:
with self.subTest(prefix=prefix):
text, sidecar, redactions = location(prefix + b"user:pass@host/path", b"https")
self.assertEqual(text, prefix.decode("ascii") + "host/path")
self.assertEqual(sidecar["state"], "redacted")
self.assertEqual(len(redactions), 1)
def test_non_special_base_keeps_backslashes_inside_network_authorities(self):
text, sidecar, redactions = location(b"//user\\name:pass@host/x", b"foo")
self.assertEqual(text, "//host/x")
self.assertEqual(sidecar["state"], "redacted")
self.assertEqual(len(redactions), 1)
text, sidecar, redactions = location(b"/\\invoice@customer", b"foo")
self.assertEqual(text, "/\\invoice@customer")
self.assertEqual(sidecar["state"], "retained")
self.assertEqual(redactions, [])
def test_native_request_paths_preserve_origin_form_and_redact_absolute_form(self):
for value in [b"/\\invoice@customer", b"//invoice@customer", b"\\\\invoice@customer"]:
with self.subTest(value=value):
record = capture.project_record({"request": {"path": value}}, [])
self.assertEqual(record["reported"]["request"]["path"], value.decode("ascii"))
self.assertEqual(record["binary_fields"][0]["state"], "retained")
self.assertEqual(record["redactions"], [])
record = capture.project_record({"request": {"path": b"https://user:pass@host/x"}}, [])
self.assertEqual(record["reported"]["request"]["path"], "https://host/x")
self.assertEqual(record["binary_fields"][0]["state"], "redacted")
def test_same_scheme_locations_preserve_relative_path_text_and_bytes(self):
for value in [b"https:invoice@customer", b"https:/invoice@customer", b"https:\\invoice@customer"]:
with self.subTest(value=value):
text, sidecar, redactions = location(value, b"https")
self.assertEqual(text, value.decode("ascii"))
self.assertEqual(sidecar["state"], "retained")
self.assertEqual(redactions, [])
def test_location_authorities_still_remove_credentials_with_a_base_scheme(self):
for value, scheme, expected in [
(b"https://user:pass@host/path", b"https", "https://host/path"),
(b"https:\\\\user:pass@host/path", b"https", "https:\\\\host/path"),
(b"https:user:pass@host/path", b"http", "https:host/path"),
]:
with self.subTest(value=value, scheme=scheme):
text, sidecar, redactions = location(value, scheme)
self.assertEqual(text, expected)
self.assertEqual(sidecar["state"], "redacted")
self.assertEqual(len(redactions), 1)
def test_backup_location_uses_its_own_request_scheme(self):
record = capture.project_record({
"request": {"scheme": b"http"},
"backup": {
"request": {"scheme": b"https"},
"response": {"headers": [[b"Location", b"https:invoice@customer"]]},
},
}, [])
self.assertEqual(record["reported"]["backup"]["response"]["headers"][0][1], "https:invoice@customer")
self.assertEqual(record["redactions"], [])
def test_backslash_ends_a_special_scheme_authority(self):
text, sidecar, redactions = location(b"https://example.test\\part@tail")
self.assertEqual(text, "https://example.test\\part@tail")
self.assertEqual(sidecar["state"], "retained")
self.assertIsNotNone(sidecar["content_base64"])
self.assertEqual(redactions, [])
def test_userinfo_before_a_backslash_is_still_removed(self):
text, sidecar, redactions = location(b"https://user:pw@example.test\\path")
self.assertEqual(text, "https://example.test\\path")
self.assertEqual(sidecar["state"], "redacted")
self.assertEqual(
redactions,
[{"pointer": "/response/headers/0/1", "reason": "transport-credential"}],
)
def test_ordinary_userinfo_is_removed(self):
text, _sidecar, _redactions = location(b"https://user:pw@example.test/a@b")
self.assertEqual(text, "https://example.test/a@b")
def test_empty_userinfo_marker_is_removed(self):
self.assertEqual(
capture.without_userinfo("https://@host"), "https://host"
)
def test_special_schemes_skip_leading_slashes_before_userinfo(self):
for value, expected in [
(b"https://\\user:pass@host/path", b"https://\\host/path"),
(b"https:\\\\user:pass@host/path", b"https:\\\\host/path"),
(b"https:user:pass@host", b"https:host"),
(b"//\\user:pass@host", b"//\\host"),
]:
with self.subTest(value=value):
self.assertEqual(capture.without_userinfo(value), expected)
text, sidecar, redactions = location(b"https://\\user:pass@host/path")
self.assertEqual(text, "https://\\host/path")
self.assertEqual(sidecar["state"], "redacted")
self.assertEqual(len(redactions), 1)
def test_file_base_keeps_paths_after_the_fixed_network_boundary(self):
for value in [b"///u@h", b"////u@h"]:
with self.subTest(value=value):
text, sidecar, redactions = location(value, b"file")
self.assertEqual(text, value.decode("ascii"))
self.assertEqual(sidecar["state"], "retained")
self.assertIsNotNone(sidecar["content_base64"])
self.assertEqual(redactions, [])
def test_file_urls_keep_path_text(self):
self.assertEqual(capture.without_userinfo("file:///a@b"), "file:///a@b")
def test_file_urls_reject_userinfo_and_keep_their_source(self):
# WHATWG rejects an "@" in a file host, so nothing parses as credentials.
self.assertEqual(
capture.without_userinfo("file://user:pass@host/x"), "file://user:pass@host/x"
)
for value in [b"\\\\user:pass@host/x", b"/\\user:pass@host/x", b"//user:pass@host/x"]:
with self.subTest(value=value):
text, sidecar, redactions = location(value, b"file")
self.assertEqual(text, value.decode("ascii"))
self.assertEqual(sidecar["state"], "retained")
self.assertEqual(redactions, [])
def test_rejected_special_hosts_keep_their_source(self):
# A rejected host or port fails the whole URL, so it has no userinfo.
for value in [
"https:user@%",
"https://user@",
"https://user:pass@h^st/x",
"https://user@ho st/x",
"https://user@host:99999/x",
"https://user@host:12a",
"https://user@[::1",
]:
with self.subTest(value=value):
self.assertEqual(capture.without_userinfo(value), value)
text, sidecar, redactions = location(b"https://user:pass@h^st/x", b"https")
self.assertEqual(text, "https://user:pass@h^st/x")
self.assertEqual(sidecar["state"], "retained")
self.assertEqual(redactions, [])
def test_accepted_special_hosts_still_remove_userinfo(self):
for value, expected in [
("https:user@%41", "https:%41"),
("https://user@[::1]:8443/x", "https://[::1]:8443/x"),
("https://user@host:/x", "https://host:/x"),
("wss://user:pass@host:65535", "wss://host:65535"),
]:
with self.subTest(value=value):
self.assertEqual(capture.without_userinfo(value), expected)
def test_non_special_schemes_keep_backslashes_in_the_authority(self):
self.assertEqual(capture.without_userinfo("foo://a\\b@host/x"), "foo://host/x")
self.assertEqual(capture.without_userinfo(b"//a\\b@host"), b"//a\\b@host")
if __name__ == "__main__":
unittest.main()
+15
View File
@@ -265,6 +265,21 @@ The user-level MCP paths follow the editor's VS Code layout. Trae's official
documentation describes the raw JSON editor without specifying those paths;
loading this registration in Trae remains unverified.
## WorkBuddy
For [WorkBuddy](https://www.workbuddy.cn/docs/workbuddy/From-Beginner-to-Expert-Guide/Function-Description/MCP-Guide), setup writes `mcpServers.rea` into the
global `~/.workbuddy/mcp.json` (a standard `mcpServers` JSON file at the user
home root on every platform). Entries carry `command`, `args`, and `env`;
stdio is the default transport. WorkBuddy discovers personal skills from
`~/.workbuddy/skills` (`%userprofile%\.workbuddy\skills` on Windows), so
setup installs its skill copy there.
The global MCP path is a plain JSON file; setup preserves every other server
and top-level key when it upserts `rea`. WorkBuddy also reads a project-level
`<project>/.workbuddy/mcp.json`, which setup does not touch: it registers the
user-level file only. Loading this registration and real REA tool calls
through WorkBuddy are verified.
## ZCode
For [ZCode](https://zcode.z.ai/en/docs/mcp-services), setup registers `rea` in the nested `mcp.servers` table of the
+3 -2
View File
@@ -288,11 +288,12 @@ hosted connector; see [its configuration](client-configuration.md#other-client-s
| Qoder | `qoder` |
| Factory Droid | `droid` |
| Trae | `trae` |
| WorkBuddy | `workbuddy` |
| ZCode | `zcode` |
| DeepSeek Harness | `dsh` |
`dsh` and `kimi` require a release containing them. The registry check above does
not cover them.
`dsh`, `kimi`, and `workbuddy` require a release containing them. The registry
check above does not cover them.
See [client configuration and troubleshooting](client-configuration.md) for
configuration paths, profiles, trust policies, and schema-limit workarounds.
+1 -1
View File
@@ -138,7 +138,7 @@
"test": "npm run test:with-build",
"test:with-build": "npm run test:prepare && vitest run",
"test:run": "vitest run",
"test:python": "python3 -m unittest bridge/native/test_rea_lldb_tracer.py -v && python3 -m unittest discover -s experiments/analysis-efficiency -p 'test_*.py' -v",
"test:python": "python3 -m unittest bridge/native/test_rea_lldb_tracer.py bridge/mitmproxy/test_capture.py -v && python3 -m unittest discover -s experiments/analysis-efficiency -p 'test_*.py' -v",
"test:local": "node scripts/run-development-tests.mjs local",
"test:fast": "node scripts/ci/test-group.mjs source",
"test:built": "node scripts/ci/test-group.mjs built",
@@ -1,4 +1,5 @@
export function assertCleanLinuxHopperRuntime(
launcherPaths?: readonly string[],
procRoot?: string,
listProcesses?: () => Promise<{ stdout: string }>,
): Promise<void>;
+2 -1
View File
@@ -17,8 +17,9 @@ export const assertCleanLinuxHopperRuntime = async (
: [process.env.HOPPER_LAUNCHER_PATH]),
],
procRoot = "/proc",
listProcesses = () => execFileAsync("ps", ["-ax", "-o", "command="]),
) => {
const { stdout } = await execFileAsync("ps", ["-ax", "-o", "command="]);
const { stdout } = await listProcesses();
if (/\/opt\/hopper\/bin\/Hopper|\bXvfb\b|hopper-demo-x11\.py/u.test(stdout))
throw new Error("Clean Linux runner retained a Hopper demo process");
const launchers = [];
+82
View File
@@ -215,6 +215,27 @@ async function verifyNativeValueAdapters(
const functionEvidence = parseEvidence(analyzed.structuredContent);
assert.equal(functionEvidence.operation, "analyze_function");
assert.equal(functionEvidence.provider.id, "ghidra");
assert.deepEqual(
functionEvidence.normalized_result.facet_availability,
Object.fromEntries(
[
"pseudocode",
"assembly",
"comments",
"callers",
"callees",
"references",
"strings_names",
"cfg",
].map((facet) => [facet, { available: true }]),
),
);
assert.equal(functionEvidence.relationship_coverage?.status, "complete");
assert.ok(
functionEvidence.relationship_coverage.diagnostics.some((diagnostic) =>
diagnostic.includes("indirect calls remain unknown"),
),
);
const functionProfile = analysisProfileSchema.parse(
functionEvidence.analysis_profile,
);
@@ -247,6 +268,7 @@ async function verifyNativeValueAdapters(
assert.ok(
comparisonEvidence.evidence_links.includes(functionEvidence.evidence_id),
);
await verifyUnresolvedCallEvidence(client);
const bundle = await client.callTool({
name: "get_evidence_bundle",
arguments: {},
@@ -307,3 +329,63 @@ async function verifyNativeValueAdapters(
unknowns: cliGraph.unknowns.length,
};
}
async function verifyUnresolvedCallEvidence(client) {
const inventory = await client.callTool({
name: "list_procedures",
arguments: {},
});
assert.notEqual(inventory.isError, true, JSON.stringify(inventory));
const procedures = parseEvidence(
inventory.structuredContent,
).normalized_result;
const indirect = procedures.find((item) =>
item.value.endsWith("rea_ghidra_inventory_indirect"),
);
const leaf = procedures.find((item) =>
item.value.endsWith("rea_ghidra_inventory_leaf"),
);
assert.ok(
indirect && leaf,
"Source fixture lacks the unresolved callback and its possible target",
);
const analyzed = await client.callTool(
{ name: "analyze_function", arguments: { procedure: indirect.address } },
{ timeout: 180000 },
);
assert.notEqual(analyzed.isError, true, JSON.stringify(analyzed));
const evidence = parseEvidence(analyzed.structuredContent);
assert.equal(evidence.relationship_coverage?.status, "partial");
assert.ok(evidence.normalized_result.unresolved_calls.length > 0);
const traced = await client.callTool(
{
name: "trace_call_path",
arguments: {
start: indirect.address,
goal: leaf.address,
direction: "forward",
},
},
{ timeout: 180000 },
);
assert.notEqual(traced.isError, true, JSON.stringify(traced));
const trace = parseEvidence(traced.structuredContent).normalized_result;
assert.equal(trace.goal_status, "unknown");
assert.equal(trace.truncated, true);
assert.ok(trace.residual_unknowns.length > 0);
const compared = await client.callTool(
{
name: "compare_functions",
arguments: {
left: analyzed.structuredContent,
right: analyzed.structuredContent,
},
},
{ timeout: 180000 },
);
assert.notEqual(compared.isError, true, JSON.stringify(compared));
assert.equal(
parseEvidence(compared.structuredContent).normalized_result.status,
"unknown",
);
}
+1 -1
View File
@@ -136,7 +136,7 @@ export async function verifyGhidraBoundaries(
assert.deepEqual(externalDossier.callees, []);
assert.deepEqual(externalDossier.facet_availability.pseudocode, {
available: false,
reason: "Ghidra did not supply pseudocode.",
reason: "Ghidra does not decompile external functions.",
});
for (const facet of [
"assembly",
+5 -5
View File
@@ -42,7 +42,6 @@ import {
workflowAnalysisProfile,
} from "./InvestigationProviders.js";
import {
executeComposedWorkflow,
createWorkflowEvidence,
workflowSnapshotRecord,
recordWorkflowUnknowns,
@@ -402,17 +401,18 @@ const executeAnalysisTool = async (input: {
tool === "trace_native_ui_action"
) {
const { EnhancedTools } = await import("./EnhancedTools.js");
const result = await executeComposedWorkflow(session, (observed) =>
new EnhancedTools(observed).execute(tool, input.arguments, signal),
const result = await new EnhancedTools(session).execute(
tool,
input.arguments,
signal,
);
if (!result.ok) return { output: cliError(result.error) };
const evidence = createWorkflowEvidence({
target: input.openedTarget,
operation: tool,
parameters: input.arguments,
result: result.value.result,
result: result.value,
upstreamProfile: evidenceProfile,
upstreamLimitations: result.value.limitations,
});
return { output: evidence, evidence };
}
+3 -3
View File
@@ -19,12 +19,12 @@ export const workflowAnalysisProfile = (
createAnalysisProfile(REA_WORKFLOW_PROVIDER, {
upstream_analysis_profile: upstream,
...(operation === "binary_overview"
? { workflow_revision: 4 }
? { workflow_revision: 3 }
: operation === "batch_decompile" ||
operation === "get_call_graph" ||
operation === "trace_feature"
? { workflow_revision: 3 }
: { workflow_revision: 1 }),
? { workflow_revision: 2 }
: {}),
});
/** Provider identity for deterministic artifact inventories. */
+8
View File
@@ -606,6 +606,14 @@ export const SUPPORTED_CLIENT_DEFINITIONS = [
markerPath: traeUserDirectory,
format: "json",
},
{
name: "workbuddy",
displayName: "WorkBuddy",
skillPath: [".workbuddy", "skills"],
configPath: [".workbuddy", "mcp.json"],
markerPath: [".workbuddy"],
format: "json",
},
{
name: "grok_build",
displayName: "Grok Build",
+1 -135
View File
@@ -3,145 +3,15 @@ import { describe, expect, it } from "vitest";
import {
recordWorkflowUnknowns,
createWorkflowEvidence,
executeComposedWorkflow,
} from "./WorkflowEvidence.js";
import { InvestigationRecords } from "./investigation/InvestigationRecords.js";
import { UnknownRegistryError } from "../domain/unknownRegistryError.js";
import { err, ok } from "../domain/result.js";
import {
createAnalysisExecution,
type AnalysisOperationPort,
} from "./AnalysisProvider.js";
import { AnalysisCancelledError } from "../domain/analysisErrorCore.js";
import { err } from "../domain/result.js";
import { nativeApiBoundarySchema } from "../domain/native/nativeApiBoundary.js";
import { projectNativeApiInspection } from "./native/NativeApiInspection.js";
import { functionDossierSchema } from "../domain/hopperValues.js";
import { ghidraFunctionDossier } from "../domain/ghidraValues.fixture.js";
const provider = { id: "fixture", name: "Fixture", version: "1" };
it("keeps Evidence identity stable when dependent branches finish in opposite orders", async () => {
const run = async (order: readonly number[]) => {
const gate = () => {
let resolve: () => void = () => {
throw new Error("gate not initialized");
};
const promise = new Promise<void>((release) => {
resolve = release;
});
return { promise, resolve };
};
const parents = [gate(), gate()];
const children = [gate(), gate()];
const analysis: AnalysisOperationPort = {
execute: async (operation, parameters) => {
const index = Number(parameters.index);
if (operation === "procedure_address") {
await parents[index]!.promise;
return ok(createAnalysisExecution(String(index), provider));
}
children[index]!.resolve();
return ok(
createAnalysisExecution(String(index), provider, {
limitations: [
index === 0 ? "first child caveat" : "second child caveat",
],
}),
);
},
};
const composed = await executeComposedWorkflow(
analysis,
async (observed) => {
const branches = Promise.all(
[0, 1].map(async (index) => {
await observed.execute("procedure_address", { index });
const child = await observed.execute("address_name", { index });
if (!child.ok) throw child.error;
return child.value.result;
}),
);
for (const index of order) {
parents[index]!.resolve();
await children[index]!.promise;
}
return ok(await branches);
},
);
if (!composed.ok) throw composed.error;
return createWorkflowEvidence({
target: undefined,
operation: "batch_decompile",
parameters: {},
result: composed.value.result,
upstreamProfile: undefined,
upstreamLimitations: composed.value.limitations,
});
};
const left = await run([0, 1]);
const right = await run([1, 0]);
expect(left.normalized_result).toEqual(right.normalized_result);
expect(left.limitations).toEqual(right.limitations);
expect(left.evidence_id).toBe(right.evidence_id);
});
it("retains unique upstream limits in canonical order across concurrent observations", async () => {
let release: () => void = () => {
throw new Error("gate not initialized");
};
const gate = new Promise<void>((resolve) => {
release = resolve;
});
let calls = 0;
const options = { signal: new AbortController().signal };
const analysis: AnalysisOperationPort = {
execute: async (_, parameters, forwarded) => {
expect(parameters).toEqual({ marker: true });
expect(forwarded).toBe(options);
const call = calls++;
if (call === 0) await gate;
else release();
return ok(
createAnalysisExecution([], provider, {
limitations: call === 0 ? ["first", "shared"] : ["second", "shared"],
}),
);
},
};
const result = await executeComposedWorkflow(analysis, async (observed) => {
await Promise.all([
observed.execute("list_strings", { marker: true }, options),
observed.execute("list_procedures", { marker: true }, options),
]);
return ok({ count: 0 });
});
expect(calls).toBe(2);
expect(result).toEqual(
ok({ result: { count: 0 }, limitations: ["first", "second", "shared"] }),
);
});
it("preserves a failed composed operation without an extra provider read", async () => {
const failure = err(new AnalysisCancelledError("list_strings"));
let calls = 0;
const result = await executeComposedWorkflow(
{
execute: async () => {
calls++;
return failure;
},
},
async (observed) => {
const observedResult = await observed.execute("list_strings", {});
return observedResult.ok
? ok(observedResult.value.result)
: observedResult;
},
);
expect(result).toBe(failure);
expect(calls).toBe(1);
});
describe("workflow residual question retention", () => {
it("keeps questions linked to workflow Evidence and records repeats idempotently", () => {
const records = new InvestigationRecords();
@@ -154,7 +24,6 @@ describe("workflow residual question retention", () => {
parameters: { procedure: "0x1000" },
result,
upstreamProfile: undefined,
upstreamLimitations: [],
});
expect(records.recordEvidence(evidence).ok).toBe(true);
const input = {
@@ -192,7 +61,6 @@ describe("workflow residual question retention", () => {
parameters: {},
result,
upstreamProfile: undefined,
upstreamLimitations: [],
});
expect(records.recordEvidence(evidence).ok).toBe(true);
@@ -220,7 +88,6 @@ describe("workflow residual question retention", () => {
parameters: {},
result,
upstreamProfile: undefined,
upstreamLimitations: [],
});
expect(records.recordEvidence(evidence).ok).toBe(true);
@@ -272,7 +139,6 @@ it.each([
parameters: { procedure: "0x1000" },
result,
upstreamProfile: undefined,
upstreamLimitations: [],
});
const records = new InvestigationRecords();
expect(records.recordEvidence(evidence).ok).toBe(true);
+2 -42
View File
@@ -2,12 +2,7 @@ import {
REA_WORKFLOW_PROVIDER,
workflowAnalysisProfile,
} from "./InvestigationProviders.js";
import type {
AnalysisOperation,
AnalysisOperationPort,
} from "./AnalysisProvider.js";
import type { AnalysisError } from "../domain/analysisErrorBase.js";
import { ok, type Result } from "../domain/result.js";
import type { AnalysisOperation } from "./AnalysisProvider.js";
import type { AnalysisProfileCommitment } from "../domain/analysisProfile.js";
import type { WorkflowSnapshotRecordInput } from "./binary/BinarySessionRecords.js";
import {
@@ -23,37 +18,6 @@ import {
import type { JsonValue } from "../domain/jsonValue.js";
import type { UnknownRegistryPort } from "./investigation/InvestigationRecordPort.js";
import { UnknownRegistryError } from "../domain/unknownRegistryError.js";
import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js";
/** Retain the limits of exactly the observations used by one composed call. */
export const executeComposedWorkflow = async (
analysis: AnalysisOperationPort,
execute: (
observed: AnalysisOperationPort,
) => Promise<Result<JsonValue, AnalysisError>>,
): Promise<
Result<
{ readonly result: JsonValue; readonly limitations: readonly string[] },
AnalysisError
>
> => {
const limitations = new Set<string>();
const result = await execute({
execute: async (operation, parameters, options) => {
const execution = await analysis.execute(operation, parameters, options);
if (execution.ok)
execution.value.limitations.forEach((limitation) =>
limitations.add(limitation),
);
return execution;
},
});
if (!result.ok) return result;
return ok({
result: result.value,
limitations: [...limitations].sort(compareUnicodeCodePoints),
});
};
/** Build the shared Evidence record for a successful composed binary workflow. */
export const createWorkflowEvidence = (input: {
@@ -62,7 +26,6 @@ export const createWorkflowEvidence = (input: {
readonly parameters: Readonly<Record<string, JsonValue>>;
readonly result: JsonValue;
readonly upstreamProfile: AnalysisProfileCommitment | undefined;
readonly upstreamLimitations: readonly string[];
}): Evidence =>
createEvidence(input.target, REA_WORKFLOW_PROVIDER, {
operation: input.operation,
@@ -77,10 +40,7 @@ export const createWorkflowEvidence = (input: {
),
}),
confidence: "derived",
limitations: [
"Derived by an REA composed workflow.",
...input.upstreamLimitations,
],
limitations: ["Derived by an REA composed workflow."],
...(input.operation === "batch_decompile"
? { locations: batchProcedureLocations(input.result) }
: {}),
+36
View File
@@ -337,6 +337,42 @@ describe("call path topology and capacity", () => {
});
});
describe("directional call relationship coverage", () => {
it.each([
{
directions: ["callers"] as ["callers"],
status: "not_found",
exhaustive: true,
},
{
directions: ["callees"] as ["callees"],
status: "unknown",
exhaustive: false,
},
{
directions: ["callers", "callees"] as ["callers", "callees"],
status: "unknown",
exhaustive: false,
},
])(
"uses $directions coverage for the forward search",
({ directions, status, exhaustive }) => {
const limitation = "The inventory omitted a reference.";
const relationshipCoverage: RelationshipCoverage = {
status: "partial",
limitations: [limitation],
limited_directions: directions,
};
const result = run([
observe("0x1000", ["0x2000"], "rea-workflow", { relationshipCoverage }),
observe("0x2000", [], "rea-workflow", { relationshipCoverage }),
]);
expect(result).toMatchObject({ status, search_scope: { exhaustive } });
expect(result.limitations).toContain(limitation);
},
);
});
describe("explicit call relationship coverage", () => {
it("accumulates 150,000 partial reasons without argument-stack exhaustion", () => {
const result = run([
+2 -2
View File
@@ -10,7 +10,7 @@ import {
parseFunctionEvidence,
type FunctionSnapshot,
} from "./functionDossierEvidence.js";
import { isCallerOnlyUnknownCoverage } from "./functionCallCoverage.js";
import { isCallerOnlyIncompleteCoverage } from "./functionCallCoverage.js";
import { compareUnicodeCodePoints } from "./unicodeCodePointOrder.js";
/** Minimal directed caller-to-callee adjacency for call-path search. */
@@ -268,7 +268,7 @@ const inspectSearch = ({
coverage === null
? snapshot.limitations.length > 0
: coverage.status !== "complete" &&
!isCallerOnlyUnknownCoverage(coverage)
!isCallerOnlyIncompleteCoverage(coverage)
)
incompleteExpandedCoverage = true;
} else if (node === goal) {
+33 -7
View File
@@ -2,7 +2,7 @@ import { describe, expect, it } from "vitest";
import {
combineFunctionRelationshipCoverage,
isCallerOnlyUnknownCoverage,
isCallerOnlyIncompleteCoverage,
} from "./functionCallCoverage.js";
import type { RelationshipCoverage } from "./relationshipCoverage.js";
@@ -22,14 +22,14 @@ const cappedCallees: RelationshipCoverage = {
};
const complete: RelationshipCoverage = { status: "complete" };
describe("isCallerOnlyUnknownCoverage", () => {
describe("isCallerOnlyIncompleteCoverage", () => {
it("detects unknown callers beside complete callees", () => {
const combined = combineFunctionRelationshipCoverage(
unknownCallers,
complete,
);
expect(combined.status).toBe("unknown");
expect(isCallerOnlyUnknownCoverage(combined)).toBe(true);
expect(isCallerOnlyIncompleteCoverage(combined)).toBe(true);
});
it("rejects unknown callees beside complete callers", () => {
@@ -37,7 +37,7 @@ describe("isCallerOnlyUnknownCoverage", () => {
complete,
unknownCallees,
);
expect(isCallerOnlyUnknownCoverage(combined)).toBe(false);
expect(isCallerOnlyIncompleteCoverage(combined)).toBe(false);
});
it("rejects unknown coverage on both sides", () => {
@@ -45,7 +45,7 @@ describe("isCallerOnlyUnknownCoverage", () => {
unknownCallers,
unknownCallees,
);
expect(isCallerOnlyUnknownCoverage(combined)).toBe(false);
expect(isCallerOnlyIncompleteCoverage(combined)).toBe(false);
});
it("rejects unknown callers beside IDA-capped callees", () => {
@@ -54,11 +54,37 @@ describe("isCallerOnlyUnknownCoverage", () => {
cappedCallees,
);
expect(combined.status).toBe("unknown");
expect(isCallerOnlyUnknownCoverage(combined)).toBe(false);
expect(isCallerOnlyIncompleteCoverage(combined)).toBe(false);
});
it("rejects complete coverage", () => {
const combined = combineFunctionRelationshipCoverage(complete, complete);
expect(isCallerOnlyUnknownCoverage(combined)).toBe(false);
expect(isCallerOnlyIncompleteCoverage(combined)).toBe(false);
});
it("retains caller-only partial coverage without relying on limitation wording", () => {
const combined = combineFunctionRelationshipCoverage(
{ status: "partial", limitations: ["Missing incoming references."] },
complete,
);
expect(combined).toMatchObject({ limited_directions: ["callers"] });
expect(isCallerOnlyIncompleteCoverage(combined)).toBe(true);
});
it("uses explicit directions even when explanation text contains direction labels", () => {
expect(
isCallerOnlyIncompleteCoverage({
status: "unknown",
reason: "Callers: unavailable. Callees: diagnostic text.",
limited_directions: ["callers"],
}),
).toBe(true);
expect(
isCallerOnlyIncompleteCoverage({
status: "unknown",
reason: "Callers: unavailable.",
limited_directions: ["callees"],
}),
).toBe(false);
});
});
+27 -10
View File
@@ -22,6 +22,16 @@ export const combineFunctionRelationshipCoverage = (
),
);
const details = diagnostics.length > 0 ? { diagnostics } : {};
const callersLimited = callers?.status !== "complete";
const calleesLimited = callees?.status !== "complete";
const limited_directions: Extract<
RelationshipCoverage,
{ status: "partial" }
>["limited_directions"] = callersLimited
? calleesLimited
? ["callers", "callees"]
: ["callers"]
: ["callees"];
const reasons = reads.flatMap(({ direction, coverage }) => {
const label =
direction === "callers" ? callerCoveragePrefix : calleeCoveragePrefix;
@@ -43,30 +53,37 @@ export const combineFunctionRelationshipCoverage = (
return {
status: "unknown",
reason: uniqueSorted(reasons).join(" "),
limited_directions,
...details,
};
if (reasons.length > 0)
return {
status: "partial",
limitations: uniqueSorted(reasons),
limited_directions,
...details,
};
return { status: "complete", ...details };
};
/**
* Whether combined relationship coverage is unknown only on the caller side.
* The combine helper above prefixes each direction's reasons, so a reason
* that carries the caller prefix and no callee prefix records observed
* callees with unobservable callers. Forward (callee-edge) searches stay
* exhaustive under that caveat; the caller reason remains a limitation.
* Forward searches depend on callee coverage. Preserve the historical
* unknown-reason encoding when reading observations without explicit directions.
*/
export const isCallerOnlyUnknownCoverage = (
export const isCallerOnlyIncompleteCoverage = (
coverage: RelationshipCoverage,
): boolean =>
coverage.status === "unknown" &&
coverage.reason.startsWith(callerCoveragePrefix) &&
!coverage.reason.includes(calleeCoveragePrefix);
): boolean => {
if (coverage.status === "complete") return false;
if (coverage.limited_directions !== undefined)
return !coverage.limited_directions.some(
(direction) => direction === "callees",
);
return (
coverage.status === "unknown" &&
coverage.reason.startsWith(callerCoveragePrefix) &&
!coverage.reason.includes(calleeCoveragePrefix)
);
};
/** Explain why two resolved call sets cannot establish call-set equality. */
export const functionCallCoverageLimitations = (
@@ -0,0 +1,40 @@
import { describe, expect, it } from "vitest";
import { analyzeJavaScriptStaticSource } from "./javascriptStaticAnalysis.js";
const endpoints = (source: string) =>
analyzeJavaScriptStaticSource(source).endpoints.map(
({ kind, value, mechanism }) => ({ kind, value, mechanism }),
);
describe("server-sent event endpoints", () => {
it("reports EventSource URLs like WebSocket URLs", () => {
expect(
endpoints(`
new WebSocket("wss://chat.example/ws");
new EventSource("/api/chat/stream");
new window.EventSource("https://chat.example/events", { withCredentials: true });
`),
).toEqual([
{
kind: "network",
value: "wss://chat.example/ws",
mechanism: "call:WebSocket",
},
{
kind: "network",
value: "/api/chat/stream",
mechanism: "call:EventSource",
},
{
kind: "network",
value: "https://chat.example/events",
mechanism: "call:window.EventSource",
},
]);
});
it("keeps a dynamic EventSource URL unreported", () => {
expect(endpoints("new EventSource(url);")).toEqual([]);
});
});
@@ -0,0 +1,34 @@
import { describe, expect, it } from "vitest";
import { analyzeJavaScriptStaticSource } from "./javascriptStaticAnalysis.js";
const endpoints = (source: string) =>
analyzeJavaScriptStaticSource(source).endpoints.map(
({ kind, value, mechanism }) => ({ kind, value, mechanism }),
);
describe("beacon endpoints", () => {
it("reports navigator.sendBeacon URLs", () => {
expect(
endpoints(`
navigator.sendBeacon("/api/telemetry", payload);
window.navigator.sendBeacon("https://metrics.example/collect");
`),
).toEqual([
{
kind: "network",
value: "/api/telemetry",
mechanism: "call:navigator.sendBeacon",
},
{
kind: "network",
value: "https://metrics.example/collect",
mechanism: "call:window.navigator.sendBeacon",
},
]);
});
it("keeps a dynamic beacon URL unreported", () => {
expect(endpoints("navigator.sendBeacon(url, payload);")).toEqual([]);
});
});
@@ -554,7 +554,13 @@ export const endpointArgument = (
receiverFact?: JavaScriptOpenReceiverFact,
): string | undefined => {
if (receiverFact === "window") return undefined;
if (name === "fetch" || name.endsWith(".fetch") || name === "WebSocket")
if (
name === "fetch" ||
name.endsWith(".fetch") ||
name === "WebSocket" ||
name === "EventSource" ||
name.endsWith(".EventSource")
)
return stringValue(args[0]);
if (name.endsWith(".open"))
return xhrOpenUrl(name, args[0], args[1], receiverFact);
@@ -574,6 +580,9 @@ export const endpointArgument = (
? undefined
: value;
}
// navigator.sendBeacon(url, data) queues a POST to its first argument.
if (name === "sendBeacon" || name.endsWith(".sendBeacon"))
return stringValue(args[0]);
if (name.endsWith("loadURL")) return stringValue(args[0]);
return undefined;
};
+11
View File
@@ -0,0 +1,11 @@
import { expect, it } from "vitest";
import { parseRecordedTransportUrl } from "./recordedTransportUrl.js";
it("recognizes an empty userinfo marker at the authority start", () => {
expect(parseRecordedTransportUrl("https://@host")).toMatchObject({
origin: "https://host",
path: "/",
credentials: { start: 8, end: 9 },
});
});
+17 -6
View File
@@ -1,24 +1,35 @@
import { z } from "zod";
const diagnostics = z.array(z.string().min(1)).min(1);
const completeCoverage = z.strictObject({ status: z.literal("complete") });
const diagnostics = z.array(z.string().min(1)).min(1).exactOptional();
const completeCoverage = z.strictObject({
status: z.literal("complete"),
diagnostics,
});
const limitedDirections = z.union([
z.tuple([z.literal("callers")]).check(z.minLength(1), z.maxLength(1)),
z.tuple([z.literal("callees")]).check(z.minLength(1), z.maxLength(1)),
z
.tuple([z.literal("callers"), z.literal("callees")])
.check(z.minLength(2), z.maxLength(2)),
]);
const partialCoverage = z.strictObject({
status: z.literal("partial"),
limitations: z.array(z.string().min(1)).min(1),
limited_directions: limitedDirections.exactOptional(),
diagnostics,
});
const unknownCoverage = z.strictObject({
status: z.literal("unknown"),
reason: z.string().min(1),
limited_directions: limitedDirections.exactOptional(),
diagnostics,
});
/** Explicit coverage state for a provider's observed caller/callee set. */
export const relationshipCoverageSchema = z.union([
export const relationshipCoverageSchema = z.discriminatedUnion("status", [
completeCoverage,
completeCoverage.extend({ diagnostics }),
partialCoverage,
partialCoverage.extend({ diagnostics }),
unknownCoverage,
unknownCoverage.extend({ diagnostics }),
]);
export type RelationshipCoverage = z.infer<typeof relationshipCoverageSchema>;
+5 -3
View File
@@ -40,6 +40,8 @@ import {
NATIVE_AOT_PROFILE_CONTRACT_REVISION,
} from "../domain/native/nativeAotPe.js";
const callRelationshipCoverage = "function-body-call-coverage-v5";
/**
* The processor language REA pins for this target, or undefined when Ghidra may
* select from the header. The committed profile and the session readback both
@@ -125,7 +127,7 @@ export const resolveGhidraAnalysisProfile = async (
load_image_observations: "source-mappings-entry-context-v2",
function_body_evidence: "complete-inclusive-ranges-v1",
function_references: "complete-body-and-entry-reference-manager-v2",
call_relationship_coverage: "resolved-call-reference-diagnostics-v1",
call_relationship_coverage: callRelationshipCoverage,
function_dossier_coverage:
"observed-facets-and-call-relationships-v1",
string_inventory_evidence: "defined-data-coverage-v1",
@@ -216,13 +218,13 @@ export const resolveGhidraAnalysisProfile = async (
].sort(),
import_mode: "ephemeral-source-immutable",
...(target.format === "pe"
? { pdb_observations: "program-info-v4" }
? { pdb_observations: "program-info-v1" }
: {}),
annotation_policy: "atomic-function-entry-metadata-v1",
load_image_observations: "source-mappings-entry-context-v2",
function_body_evidence: "complete-inclusive-ranges-v1",
function_references: "complete-body-and-entry-reference-manager-v2",
call_relationship_coverage: "resolved-call-reference-diagnostics-v1",
call_relationship_coverage: callRelationshipCoverage,
function_dossier_coverage: "observed-facets-and-call-relationships-v1",
string_inventory_evidence: "defined-data-coverage-v1",
location_resolution: "explicit-address-exact-entry-symbol-first-v3",
+113
View File
@@ -43,7 +43,120 @@ const parseFunctionValueFixture = (
return parsed.ok ? { ok: true as const, value: parsed.value.value } : parsed;
};
describe("Ghidra unresolved call locations", () => {
it("rejects a noncanonical unresolved call site rather than retaining unverifiable location evidence", () => {
expect(
parseGhidraFunctionResult("analyze_function", {
value: {
...functionDossierSchema.parse(ghidraFunctionDossier()),
unresolved_calls: [
{ address: "not-an-address", reason: "No call destination." },
],
},
limitations: [],
relationship_coverage: {
status: "partial",
limitations: ["An unresolved call remains."],
},
}),
).toMatchObject({ ok: false, error: { _tag: "AnalysisOutputError" } });
});
});
describe("Ghidra function-analysis result values", () => {
it("retains unavailable pseudocode instead of treating its empty representation as observed source", () => {
const availability = {
pseudocode: {
available: false,
reason: "Ghidra does not decompile external functions.",
},
comments: { available: true },
};
const parsed = parseGhidraFunctionResult("analyze_function", {
value: {
...functionDossierSchema.parse(ghidraFunctionDossier()),
pseudocode: "",
facet_availability: availability,
},
limitations: [],
});
expect(parsed).toMatchObject({
ok: true,
value: { value: { pseudocode: "", facet_availability: availability } },
});
});
it.each([
{ status: "complete", diagnostics: ["Resolved static references only."] },
{
status: "partial",
limitations: ["Only the first instruction range was examined."],
},
{ status: "unknown", reason: "Relationship inventory was unavailable." },
{
status: "partial",
limitations: ["Missing callers."],
limited_directions: ["callers"],
},
{
status: "unknown",
reason: "Missing callees.",
limited_directions: ["callees"],
},
])("preserves observed $status relationship coverage", (coverage) => {
const parsed = parseGhidraFunctionResult("analyze_function", {
value: ghidraFunctionDossier(),
limitations: [],
relationship_coverage: coverage,
});
expect(parsed).toMatchObject({
ok: true,
value: { relationshipCoverage: coverage },
});
});
it("preserves omitted relationship coverage for legacy observations", () => {
const parsed = parseGhidraFunctionResult("analyze_function", {
value: ghidraFunctionDossier(),
limitations: [],
});
if (!parsed.ok) throw parsed.error;
expect(parsed.value).not.toHaveProperty("relationshipCoverage");
});
});
describe("Ghidra relationship metadata validation", () => {
it.each([
{ status: "partial", limitations: [] },
{ status: "unknown", reason: "" },
{ status: "complete", limitations: ["Truncated."] },
{ status: "partial", limitations: ["Truncated."], limited_directions: [] },
{
status: "partial",
limitations: ["Truncated."],
limited_directions: ["callers", "callers"],
},
{ status: "unknown", reason: "Missing.", limited_directions: ["incoming"] },
{ status: "complete", limited_directions: ["callers"] },
])("rejects malformed relationship coverage %j", (coverage) => {
expect(
parseGhidraFunctionResult("analyze_function", {
value: ghidraFunctionDossier(),
limitations: [],
relationship_coverage: coverage,
}),
).toMatchObject({ ok: false, error: { _tag: "AnalysisOutputError" } });
});
it("rejects relationship metadata on unrelated function operations", () => {
expect(
parseGhidraFunctionResult("procedure_assembly", {
value: "ret",
limitations: [],
relationship_coverage: { status: "complete" },
}),
).toMatchObject({ ok: false, error: { _tag: "AnalysisOutputError" } });
});
it("preserves file-offset resolution in Evidence locations and rejects inexact offsets", () => {
const selection = {
file_offset: 0x210,
+16 -1
View File
@@ -27,6 +27,10 @@ import {
import { nativeApiBoundarySchema } from "../domain/native/nativeApiBoundary.js";
import { nativeValueFlowSchema } from "../domain/native/nativeValueFlow.js";
import type { EvidenceLocation } from "../domain/evidence.js";
import {
relationshipCoverageSchema,
type RelationshipCoverage,
} from "../domain/relationshipCoverage.js";
import { err, ok, type Result } from "../domain/result.js";
import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js";
import {
@@ -307,6 +311,7 @@ const ghidraFunctionDossier = functionDossierSchema
...value.comments.map(({ address }) => address),
...value.callers.map(({ address }) => address),
...value.callees.map(({ address }) => address),
...(value.unresolved_calls ?? []).map(({ address }) => address),
...value.incoming_references.flatMap(referenceAddresses),
...value.outgoing_references.flatMap(referenceAddresses),
...value.referenced_strings.flatMap(({ address, source_address }) => [
@@ -419,6 +424,7 @@ const functionBoundaryObservationOperations: ReadonlySet<GhidraFunctionOperation
const functionObservationEnvelope = z.strictObject({
value: z.unknown(),
limitations: z.array(z.string()),
relationship_coverage: relationshipCoverageSchema.exactOptional(),
selection: z
.strictObject({
file_offset: z.number().int().min(0).max(Number.MAX_SAFE_INTEGER),
@@ -432,6 +438,7 @@ export interface ParsedGhidraFunctionResult {
readonly value: JsonValue;
readonly limitations: readonly string[];
readonly locations?: readonly EvidenceLocation[];
readonly relationshipCoverage?: RelationshipCoverage;
}
/** Require exact, bounded Java-bridge function output before creating Evidence. */
@@ -442,6 +449,7 @@ export const parseGhidraFunctionResult = (
let observation: {
readonly value: unknown;
readonly limitations: readonly string[];
readonly relationship_coverage?: RelationshipCoverage;
readonly selection?: z.infer<
typeof functionObservationEnvelope
>["selection"];
@@ -450,13 +458,17 @@ export const parseGhidraFunctionResult = (
const envelope = functionObservationEnvelope.safeParse(value);
if (
!envelope.success ||
(operation !== "analyze_function" &&
operation !== "procedure_callers" &&
operation !== "procedure_callees" &&
envelope.data.relationship_coverage !== undefined) ||
(operation !== "procedure_pseudo_code" &&
envelope.data.selection !== undefined)
)
return err(
new AnalysisOutputError(
operation,
"Ghidra bridge omitted the function observation envelope",
"Ghidra bridge returned an invalid function observation envelope for this operation",
envelope.success ? undefined : { cause: envelope.error },
),
);
@@ -469,6 +481,9 @@ export const parseGhidraFunctionResult = (
? ok({
value: jsonValueSchema.parse(parsed.data),
limitations: observation.limitations,
...(observation.relationship_coverage === undefined
? {}
: { relationshipCoverage: observation.relationship_coverage }),
...(observation.selection === undefined
? {}
: {
+84 -225
View File
@@ -1,236 +1,95 @@
import { expect, it } from "vitest";
import { describe, expect, it } from "vitest";
import type { BinaryTarget } from "../domain/binaryTargetTypes.js";
import { ghidraPePdbLimitations } from "./GhidraPeProfile.js";
import {
ghidraPdbLimitations,
type GhidraSessionInfo,
} from "./GhidraSessionValues.js";
import { ghidraPeCodeViewLimitations } from "./GhidraPeProfile.js";
const target: BinaryTarget = {
path: "/fixture/public.exe",
sha256: "a".repeat(64),
kind: "executable",
format: "pe",
architecture: "x86_64",
availableArchitectures: ["x86_64"],
executableRole: "application",
managed: false,
codeView: {
pdbPath: "public.pdb",
pdbFileName: "public.pdb",
guid: "01234567-89ab-cdef-0123-456789abcdef",
age: 10,
},
};
const absentAttributes = {
file: null,
guid: null,
age: null,
signature: null,
version: null,
loaded: null,
};
it.each(["file", "guid", "age"] as const)(
"retains declared identity when only reported %s is omitted",
(attribute) => {
const pdb: GhidraSessionInfo["pdb"] = {
file: "program.pdb",
guid: "reported-guid",
age: "a",
signature: null,
version: "RSDS",
loaded: null,
resource_limit: { omitted_attributes: [attribute], utf16_budget: 4096 },
[attribute]: null,
describe("ghidraPeCodeViewLimitations", () => {
it("returns empty limitations for non-PE targets", () => {
const elfTarget: BinaryTarget = {
path: "/test/file",
sha256: "abc",
kind: "executable",
format: "elf",
architecture: "x86_64",
availableArchitectures: ["x86_64"],
};
const caveats = ghidraPePdbLimitations(target, pdb).join(" ");
expect(caveats).toContain("omitted");
expect(caveats).toContain('PDB "public.pdb"');
expect(caveats).toContain("01234567-89ab-cdef-0123-456789abcdef");
expect(caveats).toContain("load state unknown");
},
);
it.each([undefined, absentAttributes])(
"retains declared identity and unknown load state when reported attributes are %j",
(pdb) => {
const caveats = ghidraPePdbLimitations(target, pdb);
expect(caveats).toHaveLength(1);
expect(caveats[0]).toContain(
'PDB "public.pdb" (GUID 01234567-89ab-cdef-0123-456789abcdef, age (decimal) 10)',
);
expect(caveats[0]).toContain("load state unknown");
expect(caveats[0]).not.toContain("were not applied");
expect(caveats[0]).not.toContain("names are analysis-generated");
},
);
it.each([true, false, null])(
"uses reported PDB observations once with loaded = %s",
(loaded) => {
const pdb = {
file: "public.pdb",
guid: "{01234567-89AB-CDEF-0123-456789ABCDEF}",
age: "0X000A",
signature: null,
version: "RSDS",
loaded,
};
const caveats = ghidraPePdbLimitations(target, pdb);
expect(caveats).toEqual(ghidraPdbLimitations(pdb));
expect(caveats).toHaveLength(1);
expect(caveats.join(" ")).not.toContain('PDB "public.pdb"');
},
);
it.each([
{ file: "different.pdb", guid: "0123456789abcdef0123456789abcdef", age: "a" },
{ file: "public.pdb", guid: "1123456789abcdef0123456789abcdef", age: "a" },
{ file: "public.pdb", guid: "0123456789abcdef0123456789abcdef", age: "10" },
{ file: "public.pdb", guid: "unrecognized-guid", age: "a" },
{ file: "public.pdb", guid: "0123456789abcdef0123456789abcdef", age: "ten" },
])("retains non-equivalent declared PDB identity for %j", (identity) => {
for (const loaded of [false, null, true]) {
const pdb = { ...absentAttributes, ...identity, version: "RSDS", loaded };
const caveats = ghidraPePdbLimitations(target, pdb);
expect(caveats[0]).toBe(ghidraPdbLimitations(pdb)[0]);
expect(caveats[0]).toContain(identity.file);
expect(caveats[0]).toContain(identity.guid);
expect(caveats[0]).toContain(`age (hex) ${identity.age}`);
expect(caveats).toHaveLength(2);
expect(caveats[1]).toContain('PDB "public.pdb"');
expect(caveats[1]).toContain("01234567-89ab-cdef-0123-456789abcdef");
expect(caveats[1]).toContain("age (decimal) 10");
expect(caveats[1]).toContain("equivalence is not established");
expect(pdb).toMatchObject({ ...identity, loaded });
}
});
it.each([
"0123456789ABCDEF0123456789ABCDEF",
"01234567-89ab-cdef-0123-456789abcdef",
])("avoids duplicate identity for normalized matching GUID %s", (guid) => {
const pdb = { ...absentAttributes, file: "public.pdb", guid, age: "a" };
expect(ghidraPePdbLimitations(target, pdb)).toEqual(
ghidraPdbLimitations(pdb),
);
});
it("retains explicit failure to load without asserting all names are generated", () => {
const caveats = ghidraPePdbLimitations(target, {
...absentAttributes,
loaded: false,
expect(ghidraPeCodeViewLimitations(elfTarget)).toEqual([]);
});
expect(caveats[0]).toContain("reports not loaded");
expect(caveats[0]).not.toContain("names are analysis-generated");
});
it("retains a loaded-state attestation caveat even when reference attributes are absent", () => {
const caveats = ghidraPePdbLimitations(target, {
...absentAttributes,
loaded: true,
});
expect(caveats[0]).toEqual(
ghidraPdbLimitations({ ...absentAttributes, loaded: true })[0],
);
expect(caveats).toHaveLength(2);
expect(caveats[1]).toContain('PDB "public.pdb"');
expect(caveats[0]).toContain("not an independent PDB identity");
expect(caveats[0]).not.toContain("reports not loaded");
});
it("preserves omitted-attribute resource coverage with declared fallback identity", () => {
const pdb: GhidraSessionInfo["pdb"] = {
...absentAttributes,
resource_limit: { omitted_attributes: ["file"], utf16_budget: 4096 },
};
const caveats = ghidraPePdbLimitations(target, pdb);
expect(caveats).toHaveLength(2);
expect(caveats[0]).toContain("omitted");
expect(caveats[1]).toContain("public.pdb");
});
it("reports NB10 declared age without inventing a GUID", () => {
const caveats = ghidraPePdbLimitations(
{
...target,
it("does not claim CodeView symbols were unapplied when the session loaded a PDB", () => {
const peTarget: BinaryTarget = {
path: "/test/file.exe",
sha256: "abc",
kind: "executable",
architecture: "x86_64",
availableArchitectures: ["x86_64"],
executableRole: "application",
managed: false,
format: "pe",
codeView: {
pdbPath: "legacy.pdb",
pdbPath: "C:\\symbols.pdb",
pdbFileName: "symbols.pdb",
guid: "12345678-1234-1234-1234-123456789abc",
age: 1,
},
};
expect(ghidraPeCodeViewLimitations(peTarget, true)).toEqual([]);
});
it("returns empty limitations for PE target without codeView", () => {
const peTarget: BinaryTarget = {
path: "/test/file.exe",
sha256: "def",
kind: "executable",
format: "pe",
architecture: "x86_64",
availableArchitectures: ["x86_64"],
executableRole: "application",
managed: false,
};
expect(ghidraPeCodeViewLimitations(peTarget)).toEqual([]);
});
it("returns limitation with GUID and age for RSDS CodeView records", () => {
const peTarget: BinaryTarget = {
path: "/test/file.exe",
sha256: "def",
kind: "executable",
format: "pe",
architecture: "x86_64",
availableArchitectures: ["x86_64"],
executableRole: "application",
managed: false,
codeView: {
pdbPath: "C:\\build\\app.pdb",
pdbFileName: "app.pdb",
guid: "01234567-89ab-cdef-0123-456789abcdef",
age: 1,
},
};
expect(ghidraPeCodeViewLimitations(peTarget)).toEqual([
'The target references debug symbols "app.pdb" (GUID 01234567-89ab-cdef-0123-456789abcdef, age 1) that were not applied; procedure names are analysis-generated.',
]);
});
it("returns limitation without GUID for NB10 CodeView records", () => {
const peTarget: BinaryTarget = {
path: "/test/file.exe",
sha256: "def",
kind: "executable",
format: "pe",
architecture: "x86",
availableArchitectures: ["x86"],
executableRole: "application",
managed: false,
codeView: {
pdbPath: "D:\\test\\legacy.pdb",
pdbFileName: "legacy.pdb",
guid: null,
age: 2,
},
},
undefined,
);
expect(caveats[0]).toContain('PDB "legacy.pdb" (age (decimal) 2)');
expect(caveats[0]).not.toContain("GUID");
});
it("retains missing-report uncertainty without a declared CodeView record", () => {
const { codeView: _, ...withoutReference } = target;
expect(ghidraPePdbLimitations(withoutReference, undefined)).toEqual(
ghidraPdbLimitations(undefined),
);
});
it("adds no PDB warning for a non-PE target", () => {
const elf: BinaryTarget = {
path: "/fixture/program",
sha256: "a".repeat(64),
kind: "executable",
format: "elf",
architecture: "x86_64",
availableArchitectures: ["x86_64"],
};
expect(ghidraPePdbLimitations(elf, undefined)).toEqual([]);
});
it("bounds an escaped inline locator while preserving its declared source value", () => {
const name = '"'.repeat(5000) + ".pdb";
const selected = {
...target,
codeView: { ...target.codeView!, pdbFileName: name },
};
const caveats = ghidraPePdbLimitations(selected, undefined).join(" ");
expect(Buffer.byteLength(caveats)).toBeLessThan(1024);
expect(caveats).toContain("locator omitted");
expect(caveats).toContain("01234567-89ab-cdef-0123-456789abcdef");
expect(selected.codeView.pdbFileName).toBe(name);
});
it.each([
["long ASCII", "a".repeat(171)],
["ASCII at the byte budget", "a".repeat(1022)],
["ASCII beyond the byte budget", "a".repeat(1023)],
["quotes at the byte budget", '"'.repeat(511)],
["quotes beyond the byte budget", '"'.repeat(512)],
["short control escapes", "\n".repeat(511)],
["short control escapes beyond the byte budget", "\n".repeat(512)],
["six-byte control escapes", "\u0001".repeat(170)],
["six-byte control escapes beyond the byte budget", "\u0001".repeat(171)],
["UTF-8 BMP text", "中".repeat(340)],
["UTF-8 BMP text beyond the byte budget", "中".repeat(341)],
["surrogate pairs", "😀".repeat(255)],
["surrogate pairs beyond the byte budget", "😀".repeat(256)],
["unpaired high surrogates", "\ud800".repeat(170)],
["unpaired high surrogates beyond the byte budget", "\ud800".repeat(171)],
["unpaired low surrogates", "\udfff".repeat(170)],
["unpaired low surrogates beyond the byte budget", "\udfff".repeat(171)],
])("uses actual JSON UTF-8 size for %s", (_, name) => {
const selected = {
...target,
codeView: { ...target.codeView!, pdbFileName: name },
};
const caveat = ghidraPePdbLimitations(selected, undefined).join(" ");
const encoded = JSON.stringify(name);
if (Buffer.byteLength(encoded) <= 1024) {
expect(caveat).toContain(`PDB ${encoded} (`);
expect(caveat).not.toContain("locator omitted");
} else {
expect(caveat).toContain("locator omitted");
expect(caveat).not.toContain(encoded);
}
expect(selected.codeView.pdbFileName).toBe(name);
};
expect(ghidraPeCodeViewLimitations(peTarget)).toEqual([
'The target references debug symbols "legacy.pdb" (age 2) that were not applied; procedure names are analysis-generated.',
]);
});
});
+19 -104
View File
@@ -1,109 +1,24 @@
import type {
BinaryTarget,
PeCodeViewRecord,
} from "../domain/binaryTargetTypes.js";
import {
ghidraPdbLimitations,
type GhidraSessionInfo,
} from "./GhidraSessionValues.js";
import type { BinaryTarget } from "../domain/binaryTargetTypes.js";
const PDB_LOCATOR_INLINE_BYTE_BUDGET = 1024;
/** Establish equivalence only for the declared fields and recognized formats. */
const reportedPdbMatchesDeclaration = (
declared: PeCodeViewRecord,
reported: NonNullable<GhidraSessionInfo["pdb"]>,
): boolean => {
if (
(reported.file !== declared.pdbPath &&
reported.file !== declared.pdbFileName) ||
reported.age === null ||
!/^(?:0x)?[0-9a-f]+$/iu.test(reported.age) ||
Number.parseInt(reported.age, 16) !== declared.age
)
return false;
if (declared.guid === null) return reported.guid === null;
if (reported.guid === null) return false;
let guid = reported.guid;
if (guid.startsWith("{") && guid.endsWith("}")) guid = guid.slice(1, -1);
if (
!/^(?:[0-9a-f]{32}|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})$/iu.test(
guid,
)
)
return false;
return (
guid.replaceAll("-", "").toLowerCase() ===
declared.guid.replaceAll("-", "").toLowerCase()
);
};
/** Measure JSON quotes, escapes and UTF-8 without allocating the encoded string. */
const pdbLocatorFitsInlineBudget = (value: string): boolean => {
if (value.length * 6 + 2 <= PDB_LOCATOR_INLINE_BYTE_BUDGET) return true;
if (value.length + 2 > PDB_LOCATOR_INLINE_BYTE_BUDGET) return false;
let bytes = 2;
for (let index = 0; index < value.length; index += 1) {
const point = value.codePointAt(index) ?? 0;
if (point > 0xffff) {
bytes += 4;
index += 1;
} else if (point >= 0xd800 && point <= 0xdfff) {
// JSON.stringify escapes unpaired UTF-16 surrogates as six ASCII bytes.
bytes += 6;
} else if (point < 0x20) {
bytes +=
point === 8 ||
point === 9 ||
point === 10 ||
point === 12 ||
point === 13
? 2
: 6;
} else if (point === 0x22 || point === 0x5c) {
bytes += 2;
} else {
bytes += point < 0x80 ? 1 : point < 0x800 ? 2 : 3;
}
if (bytes > PDB_LOCATOR_INLINE_BYTE_BUDGET) return false;
}
return true;
};
/** Use reported Program facts first, retaining declared PE facts as a fallback. */
export const ghidraPePdbLimitations = (
/**
* Report unapplied PE companion PDB symbols when CodeView debug information is present.
*
* Ghidra does not automatically locate or apply sidecar PDBs in REA's isolated
* runtime directories without companion admission. When the PE target header
* references debug symbols, surface this limitation so callers understand that
* procedure names remain analysis-generated.
*/
export const ghidraPeCodeViewLimitations = (
target: BinaryTarget,
pdb: GhidraSessionInfo["pdb"],
pdbLoaded = false,
): readonly string[] => {
if (target.format !== "pe") return [];
const reported = ghidraPdbLimitations(pdb);
if (
target.codeView === undefined ||
(pdb !== undefined && reportedPdbMatchesDeclaration(target.codeView, pdb))
)
return reported;
if (target.format !== "pe" || target.codeView === undefined || pdbLoaded) {
return [];
}
const { pdbFileName, guid, age } = target.codeView;
// Include JSON escaping in the inline budget before allocating that representation.
const locator = pdbLocatorFitsInlineBudget(pdbFileName)
? JSON.stringify(pdbFileName)
: `(declared locator omitted from this inline caveat: ${PDB_LOCATOR_INLINE_BYTE_BUDGET}-byte resource budget)`;
const identity =
guid === null
? `age (decimal) ${age}`
: `GUID ${guid}, age (decimal) ${age}`;
const state =
pdb === undefined
? "Ghidra did not report PDB attributes; load state unknown"
: pdb.loaded === false
? "Ghidra reports not loaded"
: pdb.loaded === true
? "Ghidra reports a loaded PDB; this declared identity is not an independent matching-PDB attestation"
: "Ghidra reports load state unknown";
const comparison =
pdb !== undefined &&
[pdb.file, pdb.guid, pdb.age].some((value) => value !== null)
? " Reported and declared PDB observations are retained separately; equivalence is not established."
: "";
const declared = `The PE CodeView record references PDB ${locator} (${identity}); ${state}.${comparison} Original PDB names and types may be missing. Use list_procedures and address-based analysis to inspect recovered functions. REA does not admit a caller-selected PDB companion.`;
return pdb === undefined ? [declared] : [...reported, declared];
return [
guid !== null
? `The target references debug symbols "${pdbFileName}" (GUID ${guid}, age ${age}) that were not applied; procedure names are analysis-generated.`
: `The target references debug symbols "${pdbFileName}" (age ${age}) that were not applied; procedure names are analysis-generated.`,
];
};
+41 -115
View File
@@ -35,11 +35,6 @@ import { WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON } from "../process/WindowsA
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
import { ProviderStartupDeadline } from "../process/ProviderDeadline.js";
import { createAnalysisProfile } from "../domain/analysisProfile.js";
import { createAnalysisExecution } from "../application/AnalysisProvider.js";
import { AnalysisSnapshotCache } from "../application/binary/AnalysisSnapshotCache.js";
import { createEvidence } from "../domain/evidence.js";
import { createEvidenceBundle } from "../domain/evidenceBundle.js";
const INSTALL = "/opt/ghidra_12.1.4_PUBLIC";
const installationHost = (
@@ -85,60 +80,6 @@ const createElfClient = async (ghidra: GhidraProvider) => {
return ghidra.createClient(target, resolved.value.profile);
};
it.each(["program-info-v1", "program-info-v2", "program-info-v3"])(
"retains historical PE PDB Evidence for %s without replaying its old projection",
async (pdbObservations) => {
const target = peTarget("x86_64");
const resolved = await provider().resolveAnalysisProfile(target);
if (!resolved.ok || resolved.value.profile === null)
throw new Error("Expected admitted PE profile");
const current = resolved.value.profile;
const prior = createAnalysisProfile(current.provider, {
...current.parameters,
pdb_observations: pdbObservations,
});
const operation = "search_procedures";
const parameters = { pattern: "wmain" };
const limitations = ["Legacy projection collapsed unknown PDB load state."];
const cache = new AnalysisSnapshotCache();
cache.record({
target,
profile: prior,
operation,
parameters,
execution: createAnalysisExecution([], prior.provider, {
subject: target,
analysisProfile: prior,
limitations,
}),
});
const evidence = createEvidence(target, prior.provider, {
operation,
parameters,
result: [],
analysisProfile: prior,
limitations,
});
const exported = cache.export(
target,
prior,
createEvidenceBundle([evidence]),
);
if (!exported.ok) throw exported.error;
const imported = new AnalysisSnapshotCache();
expect(imported.import(exported.value, undefined, () => ok(0)).ok).toBe(
true,
);
expect(
imported.lookup(target, current, operation, parameters),
).toBeUndefined();
expect(
imported.lookup(target, prior, operation, parameters)?.limitations,
).toEqual(expect.arrayContaining(limitations));
expect(exported.value.evidence_bundle.records).toContainEqual(evidence);
},
);
describe("Ghidra provider", () => {
it("commits DOS loader and real-mode language while retaining the CPU family", async () => {
const ghidra = provider();
@@ -692,7 +633,7 @@ describe("Ghidra client projection", () => {
expect(resolved.value.profile.parameters).toMatchObject({
language_id: "auto-from-header",
compiler_spec_id: "auto-default",
call_relationship_coverage: "resolved-call-reference-diagnostics-v1",
call_relationship_coverage: "function-body-call-coverage-v5",
function_dossier_coverage: "observed-facets-and-call-relationships-v1",
});
@@ -889,60 +830,36 @@ describe("Ghidra result projection", () => {
});
describe("Ghidra PE CodeView debug symbol limitations", () => {
it.each([
["absent report", undefined, "load state unknown"],
["unknown load", null, "load state unknown"],
["not loaded", false, "reports not loaded"],
["loaded", true, "reports a loaded PDB"],
] as const)(
"preserves %s with one PDB caveat for a declared CodeView record",
async (_, loaded, expected) => {
const ghidra = provider(installationHost(), () => ({
start: () =>
Promise.resolve(
ok({
...sessionInfo(),
...(loaded === undefined
? {}
: {
pdb: {
file: "test.pdb",
guid: "01234567-89ab-cdef-0123-456789abcdef",
age: "1",
signature: null,
version: "RSDS",
loaded,
},
}),
}),
),
callTool: () => Promise.resolve(ok(null)),
close: () => Promise.resolve(ok(null)),
}));
const target = {
...peTarget("x86_64"),
codeView: {
pdbPath: "C:\\projects\\app\\build\\test.pdb",
pdbFileName: "test.pdb",
guid: "01234567-89ab-cdef-0123-456789abcdef",
age: 1,
},
};
const profile = await ghidra.resolveAnalysisProfile(target);
if (!profile.ok || profile.value.profile === null)
throw new Error("expected PE profile");
const client = ghidra.createClient(target, profile.value.profile);
const health = await client.execute("health", {});
if (!health.ok) throw health.error;
const pdbCaveats = health.value.limitations.filter((limit) =>
limit.includes("PDB"),
);
expect(pdbCaveats).toHaveLength(1);
expect(pdbCaveats[0]).toContain(expected);
expect(pdbCaveats[0]).not.toContain("names are analysis-generated");
await client.close();
},
);
it("projects unapplied companion PDB limitation for PE target with CodeView record", async () => {
const ghidra = provider(installationHost(), () => ({
start: () => Promise.resolve(ok(sessionInfo())),
callTool: () => Promise.resolve(ok(null)),
close: () => Promise.resolve(ok(null)),
}));
const target = {
...peTarget("x86_64"),
codeView: {
pdbPath: "C:\\projects\\app\\build\\test.pdb",
pdbFileName: "test.pdb",
guid: "01234567-89ab-cdef-0123-456789abcdef",
age: 1,
},
};
const profile = await ghidra.resolveAnalysisProfile(target);
if (!profile.ok || profile.value.profile === null)
throw new Error("expected PE profile");
const client = ghidra.createClient(target, profile.value.profile);
const health = await client.execute("health", {});
expect(health).toMatchObject({
ok: true,
value: {
limitations: expect.arrayContaining([
'The target references debug symbols "test.pdb" (GUID 01234567-89ab-cdef-0123-456789abcdef, age 1) that were not applied; procedure names are analysis-generated.',
]),
},
});
await client.close();
});
});
describe("Ghidra relationship coverage", () => {
@@ -953,7 +870,16 @@ describe("Ghidra relationship coverage", () => {
Promise.resolve(
ok(
operation === "procedure_callees"
? { value: ["0x2000"], limitations: [] }
? {
value: ["0x2000"],
limitations: [],
relationship_coverage: {
status: "complete",
diagnostics: [
"Only resolved Ghidra call references are returned; unresolved computed or indirect calls remain unknown, while function classifications distinguish thunks and externals.",
],
},
}
: { value: [], limitations: [] },
),
),
-17
View File
@@ -2,7 +2,6 @@ import type {
CapabilityDescriptor,
ProviderIdentity,
} from "../application/AnalysisProvider.js";
import type { RelationshipCoverage } from "../domain/relationshipCoverage.js";
import {
hasWindowsNativeAuthority,
windowsNativeAuthorityUnavailableReason,
@@ -44,22 +43,6 @@ const stringInventoryLimitation =
const searchLimitation =
"Regex mode uses Java Pattern semantics. Matcher or compiler stack exhaustion returns a resource constraint without closing the session; literal mode avoids regex recursion and returns complete matching values.";
const resolvedCallReferenceLimitation =
"Only resolved Ghidra call references are returned; unresolved computed or indirect calls remain unknown, while function classifications distinguish thunks and externals.";
/** Relationship-only caveats belong to each observed caller/callee result. */
export const relationshipCoverageFor = (
operation: string,
): RelationshipCoverage | undefined =>
operation === "procedure_callers" ||
operation === "procedure_callees" ||
operation === "analyze_function"
? {
status: "complete",
diagnostics: [resolvedCallReferenceLimitation],
}
: undefined;
/** Limitation text for one admitted operation, including the common base. */
export const limitationsFor = (operation: string): readonly string[] => {
const common = [
+14 -5
View File
@@ -34,6 +34,7 @@ import {
} from "../domain/analysisErrorCore.js";
import type { AnalysisError } from "../domain/analysisErrorBase.js";
import type { EvidenceLocation } from "../domain/evidence.js";
import type { RelationshipCoverage } from "../domain/relationshipCoverage.js";
import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js";
import { ProviderAdapterError } from "../domain/providerAdapterError.js";
import { err, ok, type Result } from "../domain/result.js";
@@ -68,7 +69,7 @@ import {
isGhidraMipsSoftFloatTarget,
} from "./GhidraMipsProfile.js";
import { ghidraRiscvLimitations } from "./GhidraRiscvProfile.js";
import { ghidraPePdbLimitations } from "./GhidraPeProfile.js";
import { ghidraPeCodeViewLimitations } from "./GhidraPeProfile.js";
import { unverifiedGhidraBuildLimitation } from "./GhidraInstallationPolicy.js";
import { GhidraHeadlessLauncher } from "./GhidraLauncher.js";
import { attestGhidraNativeLoadImage } from "./GhidraLoadImageAttest.js";
@@ -86,10 +87,12 @@ import {
healthLimitations,
windowsP0Limitations,
limitationsFor,
relationshipCoverageFor,
} from "./GhidraProviderCapabilities.js";
import type { GhidraSessionError } from "./GhidraSessionError.js";
import type { GhidraSessionInfo } from "./GhidraSessionValues.js";
import {
ghidraPdbLimitations,
type GhidraSessionInfo,
} from "./GhidraSessionValues.js";
import { ghidraExtensionFailure } from "./extensions/GhidraExtensionFailures.js";
import {
ghidraExtensionSchema,
@@ -528,6 +531,7 @@ export const createGhidraProviderClient = (input: {
// Replaced by the reported outcome once the session handshake is checked.
const seedLimitations: string[] = [...ghidraSeedLimitations(seeds)];
const pdbLimitations: string[] = [];
let pdbLoaded = false;
// Replaced by the reported storage outcome once the handshake is checked.
const softFloatLimitations: string[] = [
...ghidraMipsSoftFloatLimitations(softFloat),
@@ -653,10 +657,11 @@ export const createGhidraProviderClient = (input: {
info: GhidraSessionInfo,
): Promise<AnalysisError | undefined> => {
if (target.format === "pe") {
pdbLoaded = info.pdb?.loaded === true;
pdbLimitations.splice(
0,
pdbLimitations.length,
...ghidraPePdbLimitations(target, info.pdb),
...ghidraPdbLimitations(info.pdb),
);
}
const softFloatReason = ghidraMipsSoftFloatFailure(
@@ -711,6 +716,7 @@ export const createGhidraProviderClient = (input: {
...ghidraExtensionLimitations(extensions),
...seedLimitations,
...pdbLimitations,
...ghidraPeCodeViewLimitations(target, pdbLoaded),
...(releaseLimitation === undefined ? [] : [releaseLimitation]),
];
const nativeAotOwner =
@@ -727,8 +733,8 @@ export const createGhidraProviderClient = (input: {
rawResult: JsonValue,
observationLimitations: readonly string[] = [],
locations: readonly EvidenceLocation[] = [],
relationshipCoverage?: RelationshipCoverage,
) => {
const relationshipCoverage = relationshipCoverageFor(operation);
return createAnalysisExecution(normalized, committedProfile.provider, {
rawResult,
locations,
@@ -832,12 +838,14 @@ export const createGhidraProviderClient = (input: {
let normalized: JsonValue;
let observationLimitations: readonly string[] = [];
let observationLocations: readonly EvidenceLocation[] = [];
let relationshipCoverage: RelationshipCoverage | undefined;
if (isGhidraFunctionOperation(operation)) {
const result = parseGhidraFunctionResult(operation, called.value);
if (!result.ok) return result;
normalized = result.value.value;
observationLimitations = result.value.limitations;
observationLocations = result.value.locations ?? [];
relationshipCoverage = result.value.relationshipCoverage;
} else {
const result = parseGhidraInventoryResult(operation, called.value);
if (!result.ok) return result;
@@ -933,6 +941,7 @@ export const createGhidraProviderClient = (input: {
called.value,
observationLimitations,
observationLocations,
relationshipCoverage,
),
);
},
+8 -13
View File
@@ -121,26 +121,21 @@ export const ghidraPdbLimitations = (
limitations.push(
`PDB attributes ${pdb.resource_limit.omitted_attributes.join(", ")} were omitted because the handshake's ${pdb.resource_limit.utf16_budget}-UTF-16-unit PDB attribute budget was exhausted. Omitted values remain unknown; they were not truncated or reported as absent.`,
);
const hasReference = [
pdb.file,
pdb.guid,
pdb.age,
pdb.signature,
pdb.version,
].some((value) => value !== null);
const reference = hasReference
? `The Ghidra Program references PDB ${pdb.file ?? "(file unknown)"} (GUID ${pdb.guid ?? "unknown"}, age (hex) ${pdb.age ?? "unknown"}, signature ${pdb.signature ?? "unknown"}, version ${pdb.version ?? "unknown"}); `
: "";
if (pdb.loaded === true)
return [
...limitations,
`${reference}Ghidra reports a loaded PDB; this is not an independent PDB identity or complete symbol/type-coverage attestation. Original names and types may still be missing; an empty name search does not prove a procedure is absent. Use list_procedures and address-based analysis to inspect recovered functions.`,
"Ghidra reports a loaded PDB; this is not an independent PDB identity or complete symbol/type-coverage attestation. Original names and types may still be missing; an empty name search does not prove a procedure is absent. Use list_procedures and address-based analysis to inspect recovered functions.",
];
if (!hasReference) return limitations;
if (
[pdb.file, pdb.guid, pdb.age, pdb.signature, pdb.version].every(
(value) => value === null,
)
)
return limitations;
const state = pdb.loaded === false ? "not loaded" : "load state unknown";
return [
...limitations,
`${reference}Ghidra reports ${state}. Original names and types may be missing; an empty name search does not prove a procedure is absent. Use list_procedures and address-based analysis to inspect recovered functions. REA does not admit a caller-selected PDB companion.`,
`The target references PDB ${pdb.file ?? "(file unknown)"} (GUID ${pdb.guid ?? "unknown"}, age (hex) ${pdb.age ?? "unknown"}, signature ${pdb.signature ?? "unknown"}, version ${pdb.version ?? "unknown"}); Ghidra reports ${state}. Original names and types may be missing; an empty name search does not prove a procedure is absent. Use list_procedures and address-based analysis to inspect recovered functions. REA does not admit a caller-selected PDB companion.`,
];
};
+121 -37
View File
@@ -11,6 +11,7 @@ import { silentLogger } from "../logger.js";
import { GhidraProvider } from "./GhidraProvider.js";
import { GhidraSessionError } from "./GhidraSessionError.js";
import { GHIDRA_SESSION_CAPABILITIES } from "./GhidraSessionValues.js";
import type { RelationshipCoverage } from "../domain/relationshipCoverage.js";
const TARGET: BinaryTarget = {
path: "/tmp/ghidra-cache-fixture",
@@ -23,7 +24,7 @@ const TARGET: BinaryTarget = {
// Exercise the real provider descriptors, profile, input/output projection and
// session cache; only the installed Ghidra process is replaced at its seam.
const fixture = (version = "12.1.4") => {
const fixture = (version = "12.1.4", coverage?: RelationshipCoverage) => {
const config = parseConfig({ GHIDRA_INSTALL_DIR: "/opt/ghidra" });
if (!config.ok) throw config.error;
const calls: string[] = [];
@@ -113,7 +114,13 @@ const fixture = (version = "12.1.4") => {
return ok(
operation === "procedure_pseudo_code" ||
operation === "analyze_function"
? { value, limitations: [] }
? {
value,
limitations: [],
...(operation === "analyze_function" && coverage !== undefined
? { relationship_coverage: coverage }
: {}),
}
: value,
);
},
@@ -133,42 +140,119 @@ const open = async (session: ReturnType<typeof fixture>["session"]) => {
expect((await session.openResolvedTarget(preview.value)).ok).toBe(true);
};
describe("Ghidra snapshot cache", () => {
it("refuses snapshots whose profile predates typed incoming references", async () => {
const initial = fixture();
await open(initial.session);
const snapshot = initial.session.exportAnalysisSnapshot();
if (!snapshot.ok) throw snapshot.error;
await initial.session.close();
const profile = snapshot.value.binding.analysis_profile;
expect(profile.parameters.instruction_references).toBe(
"exact-address-incoming-v1",
);
const legacy = {
...snapshot.value,
binding: {
...snapshot.value.binding,
analysis_profile: createAnalysisProfile(
profile.provider,
Object.fromEntries(
Object.entries(profile.parameters).filter(
([key]) => key !== "instruction_references",
),
),
),
},
};
const replay = fixture();
const imported = replay.session.importAnalysisSnapshot(legacy);
if (!imported.ok) throw imported.error;
expect(await replay.session.previewTarget(TARGET)).toMatchObject({
ok: false,
error: { _tag: "EvidenceIntegrityError" },
});
expect(replay.calls).toEqual([]);
await replay.session.close();
});
describe("Ghidra relationship coverage replay", () => {
it.each([
undefined,
{ status: "complete", diagnostics: ["Resolved static references only."] },
{ status: "partial", limitations: ["Instruction coverage was truncated."] },
{ status: "unknown", reason: "Relationship inventory was unavailable." },
{
status: "partial",
limitations: ["Missing callers."],
limited_directions: ["callers"],
},
] satisfies (RelationshipCoverage | undefined)[])(
"preserves relationship coverage %j through live execution, cache and snapshot replay",
async (coverage) => {
const initial = fixture("12.1.4", coverage);
await open(initial.session);
const parameters = { procedure: "0x1000" };
for (let i = 0; i < 2; i++) {
const execution = await initial.session.execute(
"analyze_function",
parameters,
);
if (!execution.ok) throw execution.error;
expect(execution.value.relationshipCoverage).toEqual(coverage);
}
expect(initial.calls).toEqual(["analyze_function"]);
const snapshot = initial.session.exportAnalysisSnapshot();
if (!snapshot.ok) throw snapshot.error;
await initial.session.close();
const replay = fixture();
expect(replay.session.importAnalysisSnapshot(snapshot.value).ok).toBe(
true,
);
await open(replay.session);
const execution = await replay.session.execute(
"analyze_function",
parameters,
);
if (!execution.ok) throw execution.error;
expect(execution.value.relationshipCoverage ?? undefined).toEqual(
coverage,
);
expect(replay.calls).toEqual([]);
await replay.session.close();
},
);
});
describe("Ghidra snapshot profile admission", () => {
it.each([
{
parameter: "instruction_references",
current: "exact-address-incoming-v1",
legacy: undefined,
},
{
parameter: "call_relationship_coverage",
current: "function-body-call-coverage-v5",
legacy: "resolved-call-reference-diagnostics-v1",
},
{
parameter: "call_relationship_coverage",
current: "function-body-call-coverage-v5",
legacy: "decoded-call-target-coverage-v2",
},
{
parameter: "call_relationship_coverage",
current: "function-body-call-coverage-v5",
legacy: "decoded-call-target-coverage-v3",
},
{
parameter: "call_relationship_coverage",
current: "function-body-call-coverage-v5",
legacy: "function-body-call-coverage-v4",
},
])(
"refuses snapshots with outdated $parameter semantics",
async ({ parameter, current, legacy: prior }) => {
const initial = fixture();
await open(initial.session);
const snapshot = initial.session.exportAnalysisSnapshot();
if (!snapshot.ok) throw snapshot.error;
await initial.session.close();
const profile = snapshot.value.binding.analysis_profile;
expect(profile.parameters[parameter]).toBe(current);
const legacy = {
...snapshot.value,
binding: {
...snapshot.value.binding,
analysis_profile: createAnalysisProfile(profile.provider, {
...Object.fromEntries(
Object.entries(profile.parameters).filter(
([key]) => key !== parameter,
),
),
...(prior === undefined ? {} : { [parameter]: prior }),
}),
},
};
const replay = fixture();
const imported = replay.session.importAnalysisSnapshot(legacy);
if (!imported.ok) throw imported.error;
expect(await replay.session.previewTarget(TARGET)).toMatchObject({
ok: false,
error: { _tag: "EvidenceIntegrityError" },
});
expect(replay.calls).toEqual([]);
await replay.session.close();
},
);
});
describe("Ghidra snapshot cache", () => {
it.each([
{ operation: "analyze_function", parameters: { procedure: "0x1000" } },
{
+5 -12
View File
@@ -14,7 +14,6 @@ import type { BinarySessionPort } from "../application/binary/BinarySessionPort.
import { EnhancedTools } from "../application/EnhancedTools.js";
import type { ValidatedEnhancedCall } from "../application/EnhancedToolTypes.js";
import {
executeComposedWorkflow,
createWorkflowEvidence,
workflowSnapshotRecord,
recordWorkflowUnknowns,
@@ -232,20 +231,15 @@ const executeEnhancedTool = async (
context,
parameters,
});
const analysis: AnalysisOperationPort = {
const services = new EnhancedTools({
execute: (operation, operationParameters, executionOptions) =>
admittedAnalysis.execute(operation, operationParameters, {
...executionOptions,
progress,
}),
};
});
const result = await logToolExecution(registration.logger, name, () =>
executeComposedWorkflow(analysis, (observed) =>
new EnhancedTools(observed).executeValidated(
validatedCall,
context.mcpReq.signal,
),
),
services.executeValidated(validatedCall, context.mcpReq.signal),
);
await progress.report({
phase: name,
@@ -259,9 +253,8 @@ const executeEnhancedTool = async (
target: registration.activeTarget?.(),
operation: name,
parameters,
result: result.value.result,
result: result.value,
upstreamProfile: registration.analysisProfile?.(),
upstreamLimitations: result.value.limitations,
});
const recorded = registration.recordEvidence?.(evidence);
if (recorded !== undefined && !recorded.ok)
@@ -280,7 +273,7 @@ const executeEnhancedTool = async (
}
const unknowns = recordWorkflowUnknowns({
name,
result: result.value.result,
result: result.value,
evidenceId: evidence.evidence_id,
recordUnknown: registration.recordUnknown,
});
@@ -67,6 +67,7 @@ describe("client configuration filesystem lifecycle", () => {
"kimi",
"vscode",
"trae",
"workbuddy",
"grok_build",
"omp",
"pi",
@@ -136,7 +136,6 @@ interface SnapshotProviderOptions {
readonly profile?: ReturnType<typeof createAnalysisProfile>;
readonly identity?: ProviderIdentity;
readonly supportedOperations?: readonly CapabilityDescriptor["operation"][];
readonly limitations?: readonly string[];
}
const makeProvider = (
@@ -210,7 +209,6 @@ const makeProvider = (
createAnalysisExecution(result, profile.provider, {
analysisProfile: profile,
rawResult: result,
limitations: options.limitations ?? [],
}),
);
},
@@ -255,11 +253,11 @@ const withAlternateWorkflowProfile = (
snapshot: AnalysisSnapshot,
path: string,
current: Evidence,
alternateProfile = createAnalysisProfile(REA_WORKFLOW_PROVIDER, {
): AnalysisSnapshot => {
const alternateProfile = createAnalysisProfile(REA_WORKFLOW_PROVIDER, {
workflow: "binary_overview",
fixture: "alternate-profile",
}),
): AnalysisSnapshot => {
});
if (
typeof current.normalized_result !== "object" ||
current.normalized_result === null ||
@@ -468,78 +466,6 @@ const verifyWindowsSnapshotAdmission = async (
).toHaveLength(1);
};
it.each([
{ tool: "binary_overview", arguments: {}, priorRevision: 3 },
{
tool: "inspect_native_api",
arguments: { procedure: "0x1000" },
priorRevision: undefined,
},
] as const)(
"refreshes pre-coverage $tool Evidence and replays the current revision",
async (scenario) => {
const fixture = await createSnapshotReplayFixture("rea-prior-coverage-");
const limitation = "Original PDB names may be missing.";
const provider = fixture.createProvider({
supportedOperations: workflowProviderOperations,
limitations: [limitation],
});
const dependencies = fixture.dependencies(provider);
const options = { snapshotPath: fixture.snapshotPath };
const first = parseEvidence(
await runDirectAnalysis(
dependencies,
fixture.path,
scenario.tool,
scenario.arguments,
options,
),
);
const saved = await readAnalysisSnapshot(fixture.snapshotPath);
if (!saved.ok) throw saved.error;
const priorProfile = createAnalysisProfile(REA_WORKFLOW_PROVIDER, {
upstream_analysis_profile: saved.value.binding.analysis_profile,
...(scenario.priorRevision === undefined
? {}
: { workflow_revision: scenario.priorRevision }),
});
const prior = withAlternateWorkflowProfile(
saved.value,
fixture.path,
{ ...first, limitations: ["Derived by an REA composed workflow."] },
priorProfile,
);
const written = await withArtifactResourceScope((resources) =>
writeAnalysisSnapshot(prior, fixture.snapshotPath, true, resources),
);
if (!written.ok) throw written.error;
const refreshed = parseEvidence(
await runDirectAnalysis(
dependencies,
fixture.path,
scenario.tool,
scenario.arguments,
options,
),
);
expect(refreshed.limitations).toContain(limitation);
expect(refreshed.analysis_profile?.digest).not.toBe(priorProfile.digest);
expect(refreshed.normalized_result).toEqual(first.normalized_result);
const starts = [...fixture.starts];
const calls = [...fixture.calls];
const replayed = await runDirectAnalysis(
dependencies,
fixture.path,
scenario.tool,
scenario.arguments,
options,
);
expect(replayed).toEqual(refreshed);
expect(fixture.starts).toEqual(starts);
expect(fixture.calls).toEqual(calls);
},
);
describe("direct analysis composed snapshot replay", () => {
it.skipIf(process.platform !== "win32")(
"does not replay a CLI snapshot through a Windows junction",
@@ -0,0 +1,159 @@
import { mkdir, readFile, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { readClientRegistrationStatuses } from "../../../src/application/ClientRegistrationStatus.js";
import { npxRegistrationCommand } from "../../../src/application/ClientRegistrationIdentity.js";
import { configureClientConfiguration } from "../../../src/application/SetupClientConfiguration.js";
import { skillDestinations } from "../../../src/application/SetupSkill.js";
import { supportedClients } from "../../../src/application/SupportedClients.js";
import { systemUninstallHost } from "../../../src/application/Uninstall.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { clearClientLocationEnvironment } from "../../fixtures/clientEnvironment.js";
/**
* The registration command setup writes on the host running the tests. Windows
* wraps it in `cmd.exe /d /c`, so the expected entry follows the host too.
*/
const command = npxRegistrationCommand(process.platform);
beforeEach(clearClientLocationEnvironment);
afterEach(() => vi.unstubAllEnvs());
const workbuddyPath = (
home: string,
platform: NodeJS.Platform = process.platform,
env: Parameters<typeof supportedClients>[2] = {},
) =>
supportedClients(home, platform, env).find(
({ name }) => name === "workbuddy",
);
describe("WorkBuddy configuration paths", () => {
it("resolves the global mcp.json under the home root", () => {
expect(workbuddyPath("/home/a")?.configPath).toBe(
join("/home/a", ".workbuddy", "mcp.json"),
);
expect(workbuddyPath("/Users/a")?.markerPath).toBe(
join("/Users/a", ".workbuddy"),
);
expect(workbuddyPath("C:\\Users\\a", "win32")?.configPath).toBe(
join("C:\\Users\\a", ".workbuddy", "mcp.json"),
);
});
it("resolves the registration format as the plain mcpServers JSON dialect", () => {
expect(workbuddyPath("/home/a")?.format).toBe("json");
});
it("resolves the personal skills directory under the home root", () => {
expect(
skillDestinations("/home/a", ["workbuddy"]).map(({ path }) => path),
).toEqual([
join("/home/a", ".workbuddy", "skills", "reverse-engineer-anything"),
]);
expect(
skillDestinations("C:\\Users\\a", ["workbuddy"], {}, "win32").map(
({ path }) => path,
),
).toEqual([
join("C:\\Users\\a", ".workbuddy", "skills", "reverse-engineer-anything"),
]);
});
});
describe("WorkBuddy JSON registration", () => {
const workbuddyClient = async () => {
const home = await createTestTempDirectory("rea-workbuddy-");
const client = workbuddyPath(home, process.platform);
if (client?.markerPath === undefined)
throw new Error("missing WorkBuddy client");
await mkdir(client.markerPath, { recursive: true });
return { home, client };
};
it("upserts rea into mcpServers while preserving unrelated settings", async () => {
const { home, client } = await workbuddyClient();
await writeFile(
client.configPath,
`{
"mcpServers": {
"other": { "command": "other-server" }
}
}`,
);
expect(
(await configureClientConfiguration(client, {}, command)).status,
).toBe("configured");
const parsed = JSON.parse(await readFile(client.configPath, "utf8"));
expect(parsed.mcpServers.other).toEqual({ command: "other-server" });
expect(parsed.mcpServers.rea).toEqual({
command: command[0],
args: command.slice(1),
});
expect(await configureClientConfiguration(client, {}, command)).toEqual({
status: "unchanged",
});
expect(
(
await readClientRegistrationStatuses(home, undefined, {
platform: process.platform,
environment: {},
})
).find(({ client }) => client === "workbuddy"),
).toMatchObject({ state: "aligned", command });
});
it("creates the mcpServers table in an existing mcp.json without servers", async () => {
const { client } = await workbuddyClient();
await writeFile(
client.configPath,
JSON.stringify({ unrelated: { setting: true } }, null, 2),
);
expect(
(await configureClientConfiguration(client, {}, command)).status,
).toBe("configured");
const parsed = JSON.parse(await readFile(client.configPath, "utf8"));
expect(parsed.unrelated).toEqual({ setting: true });
expect(parsed.mcpServers.rea).toEqual({
command: command[0],
args: command.slice(1),
});
});
it("registers into an empty mcp.json", async () => {
const { client } = await workbuddyClient();
await writeFile(client.configPath, "");
expect(
(await configureClientConfiguration(client, {}, command)).status,
).toBe("configured");
const parsed = JSON.parse(await readFile(client.configPath, "utf8"));
expect(parsed.mcpServers.rea).toEqual({
command: command[0],
args: command.slice(1),
});
});
it("removes only the rea entry on uninstall", async () => {
const { home, client } = await workbuddyClient();
await writeFile(
client.configPath,
JSON.stringify(
{ mcpServers: { other: { command: "other-server" } } },
null,
2,
),
);
await configureClientConfiguration(client, {}, command);
expect((await systemUninstallHost(home).removeClient(client)).status).toBe(
"removed",
);
const parsed = JSON.parse(await readFile(client.configPath, "utf8"));
expect(parsed.mcpServers.other).toEqual({ command: "other-server" });
expect(parsed.mcpServers.rea).toBeUndefined();
});
});
@@ -9,7 +9,6 @@ import {
AnalysisOutputError,
} from "../../../src/domain/analysisErrorCore.js";
import type { RelationshipCoverage } from "../../../src/domain/relationshipCoverage.js";
import { parseEvidence } from "../../../src/domain/evidence.js";
import { err, ok as resultOk } from "../../../src/domain/result.js";
import { closeEnhancedToolResources, connect } from "./enhancedToolsHarness.js";
@@ -126,13 +125,9 @@ describe("enhanced MCP tools", () => {
limitations: [caveat],
},
});
const evidence = parseEvidence(graph.structuredContent);
expect(
JSON.stringify(evidence.normalized_result).split(caveat),
).toHaveLength(2);
expect(
evidence.limitations.filter((limit) => limit === caveat),
).toHaveLength(1);
expect(JSON.stringify(graph.structuredContent)?.split(caveat)).toHaveLength(
2,
);
const trace = await client.callTool({
name: "trace_call_path",
@@ -1,42 +0,0 @@
import { afterEach, expect, it } from "vitest";
import { createAnalysisExecution } from "../../../src/application/AnalysisProvider.js";
import { ok } from "../../../src/domain/result.js";
import { parseEvidence } from "../../../src/domain/evidence.js";
import { connect, closeEnhancedToolResources } from "./enhancedToolsHarness.js";
afterEach(closeEnhancedToolResources);
it("retains provider coverage caveats in a composed MCP overview without repeated reads", async () => {
const calls: string[] = [];
const limitation = "Companion PDB symbol coverage is unverified.";
const client = await connect({
execute: async (operation) => {
calls.push(operation);
const value = operation === "current_document" ? "fixture" : [];
return ok(
createAnalysisExecution(
value,
{ id: "fixture", name: "Fixture", version: "1" },
{ limitations: [limitation] },
),
);
},
});
const response = await client.callTool({
name: "binary_overview",
arguments: {},
});
expect(response.isError).not.toBe(true);
const evidence = parseEvidence(response.structuredContent);
expect(evidence.limitations).toEqual([
"Artifact identity is unavailable for this observation.",
"Derived by an REA composed workflow.",
limitation,
]);
expect(evidence.normalized_result).toMatchObject({
document: "fixture",
procedure_count: 0,
string_count: 0,
});
expect(calls.length).toBe(new Set(calls).size);
});
@@ -0,0 +1,138 @@
import { Client, InMemoryTransport } from "@modelcontextprotocol/client";
import { Ajv2020 } from "ajv/dist/2020.js";
import { expect, it } from "vitest";
import { FUNCTION_COMPARISON_EXAMPLE } from "../../../src/contracts/functionComparisonExample.js";
import { createEvidence } from "../../../src/domain/evidence.js";
import { functionDossierSchema } from "../../../src/domain/hopperValues.js";
import type { RelationshipCoverage } from "../../../src/domain/relationshipCoverage.js";
import { createServer } from "../../../src/server/createServer.js";
import { createTestBinarySession } from "../../fixtures/binarySession.js";
const observe = (relationshipCoverage: RelationshipCoverage) => {
const template = FUNCTION_COMPARISON_EXAMPLE.left;
if (template.subject === null)
throw new Error("Expected artifact-bound fixture");
const dossier = functionDossierSchema.parse(template.normalized_result);
return createEvidence(
{
path: template.subject.local_path,
sha256: template.subject.digest.sha256,
format: template.subject.format,
},
template.provider,
{
operation: "analyze_function",
parameters: template.parameters,
result: { ...dossier, callees: [], unresolved_calls: [] },
confidence: template.confidence,
authority: template.authority,
relationshipCoverage,
},
);
};
it("preserves directional gaps through advertised schemas and public path reconstruction", async () => {
const session = createTestBinarySession(() => {
throw new Error("Offline call paths must not launch a provider");
});
const server = createServer({ kind: "session", session });
const client = new Client({ name: "function-call-directions", version: "1" });
const [clientTransport, serverTransport] =
InMemoryTransport.createLinkedPair();
try {
await server.connect(serverTransport);
await client.connect(clientTransport);
const advertised = (await client.listTools()).tools.find(
({ name }) => name === "build_call_path",
);
if (advertised === undefined)
throw new Error("Missing build_call_path contract");
const ajv = new Ajv2020({ strict: false, validateFormats: false });
expect(ajv.validateSchema(advertised.inputSchema)).toBe(true);
const validate = ajv.compile(advertised.inputSchema);
for (const { directions, status, exhaustive } of [
{
directions: ["callers"] as ["callers"],
status: "not_found",
exhaustive: true,
},
{
directions: ["callees"] as ["callees"],
status: "unknown",
exhaustive: false,
},
{
directions: ["callers", "callees"] as ["callers", "callees"],
status: "unknown",
exhaustive: false,
},
]) {
const limitation =
"An observed call reference is absent from the inventory.";
const coverage: RelationshipCoverage = {
status: "partial",
limitations: [limitation],
limited_directions: directions,
};
const observation = observe(coverage);
expect(session.recordEvidence(observation).ok).toBe(true);
const input = {
functions: [observation],
start: { address: "0x1000" },
goal: { address: "0x9999" },
};
expect(validate(input), JSON.stringify(validate.errors)).toBe(true);
for (const limited_directions of [
[],
["callers", "callers"],
["callees", "callees"],
["callers", "callees", "callers"],
["incoming"],
])
expect(
validate({
...input,
functions: [
{
...observation,
relationship_coverage: {
...coverage,
limited_directions,
},
},
],
}),
).toBe(false);
const result = await client.callTool({
name: "build_call_path",
arguments: input,
});
if (result.isError) throw new Error(JSON.stringify(result));
expect(result.structuredContent).toMatchObject({
normalized_result: {
status,
search_scope: { exhaustive },
limitations: expect.arrayContaining([limitation]),
},
});
const comparison = await client.callTool({
name: "compare_functions",
arguments: { left: observation, right: observation },
});
if (comparison.isError) throw new Error(JSON.stringify(comparison));
expect(comparison.structuredContent).toMatchObject({
normalized_result: {
dimensions: expect.arrayContaining([
expect.objectContaining({ dimension: "calls", status: "unknown" }),
]),
},
});
// Each scenario owns its Evidence and residual-unknown ledger.
const closed = await session.close();
if (!closed.ok) throw closed.error;
}
} finally {
await Promise.allSettled([session.close(), client.close(), server.close()]);
}
});
+21 -11
View File
@@ -15,9 +15,17 @@ import {
type GhidraFunctionOperation,
} from "../../../src/ghidra/GhidraFunctionValues.js";
it.each([true, false])(
"compares observed empty Ghidra facets while preserving pseudocode availability (%s)",
async (pseudocodeAvailable) => {
it.each([
{ pseudocodeAvailable: true, reportsCoverage: true },
{ pseudocodeAvailable: false, reportsCoverage: true },
{ pseudocodeAvailable: true, reportsCoverage: false },
{ pseudocodeAvailable: false, reportsCoverage: false },
])(
"compares observed empty Ghidra facets with pseudocode=$pseudocodeAvailable and coverage=$reportsCoverage",
async ({ pseudocodeAvailable, reportsCoverage }) => {
const coverage = reportsCoverage
? { status: "complete" as const }
: undefined;
const dossier = functionDossierSchema.parse(ghidraFunctionDossier());
Object.assign(dossier, {
comments: [],
@@ -47,7 +55,13 @@ it.each([true, false])(
if (operation !== "analyze_function")
throw new Error(`Unexpected operation: ${operation}`);
return Promise.resolve(
ok({ value: jsonValueSchema.parse(dossier), limitations: [] }),
ok({
value: jsonValueSchema.parse(dossier),
limitations: [],
...(coverage === undefined
? {}
: { relationship_coverage: coverage }),
}),
);
},
);
@@ -61,9 +75,7 @@ it.each([true, false])(
harness.session,
analyzed.structuredContent,
);
expect(evidence.relationship_coverage).toMatchObject({
status: "complete",
});
expect(evidence.relationship_coverage).toEqual(coverage ?? null);
expect(evidence.normalized_result).toEqual(dossier);
const reply = await harness.mcp.callTool({
name: "compare_functions",
@@ -86,7 +98,7 @@ it.each([true, false])(
pseudocode: pseudocodeAvailable ? "unchanged" : "unknown",
assembly: "unchanged",
comments: "unchanged",
calls: "unchanged",
calls: reportsCoverage ? "unchanged" : "unknown",
references: "unchanged",
strings_names: "unchanged",
cfg: "unchanged",
@@ -96,9 +108,7 @@ it.each([true, false])(
evidence.parameters,
);
if (!replay.ok) throw replay.error;
expect(replay.value.relationshipCoverage).toEqual(
evidence.relationship_coverage,
);
expect(replay.value.relationshipCoverage).toEqual(coverage);
expect(harness.calls).toEqual(["analyze_function"]);
} finally {
await harness.close();
@@ -315,7 +315,7 @@ it("retains missing PDB coverage in empty PE name-search Evidence and snapshot r
const evidence = sessionEvidence(harness.session, reply.structuredContent);
expect(evidence.normalized_result).toEqual([]);
expect(evidence.analysis_profile?.parameters.pdb_observations).toBe(
"program-info-v4",
"program-info-v1",
);
const limitation = evidence.limitations.find((value) =>
value.includes(pdb.file),
@@ -1,54 +0,0 @@
import { expect, it } from "vitest";
import { parseConfig } from "../../src/config/parseConfig.js";
import { runDirectAnalysis } from "../../src/application/DirectAnalysis.js";
import { createAnalysisExecution } from "../../src/application/AnalysisProvider.js";
import { ok } from "../../src/domain/result.js";
import {
createBinarySessionTargets,
createTestBinarySession,
} from "../fixtures/binarySession.js";
it("retains provider coverage caveats in a composed CLI result and closes the session", async () => {
const [path] = await createBinarySessionTargets();
const limitation = "Companion PDB symbol coverage is unverified.";
let reads = 0;
let closed = 0;
const createSession = () =>
createTestBinarySession(() => ({
execute: async (operation) => {
reads++;
return ok(
createAnalysisExecution(
operation === "current_document" ? "fixture" : [],
{ id: "fixture", name: "Fixture", version: "1" },
{ limitations: [limitation] },
),
);
},
close: async () => {
closed++;
return ok(null);
},
}));
const output = await runDirectAnalysis(
{
readConfiguration: () => parseConfig({}),
createBinarySession: createSession,
createManagedBinarySession: createSession,
},
path,
"binary_overview",
{},
);
expect(output).toMatchObject({
limitations: ["Derived by an REA composed workflow.", limitation],
normalized_result: {
document: "fixture",
segment_count: 0,
procedure_count: 0,
string_count: 0,
},
});
expect(reads).toBe(5);
expect(closed).toBe(1);
});
@@ -12,6 +12,8 @@ import ghidra.program.model.listing.FlowOverride;
import ghidra.program.model.listing.Function;
import ghidra.program.model.listing.Instruction;
import ghidra.program.model.symbol.SourceType;
import ghidra.program.model.symbol.Reference;
import ghidra.program.model.symbol.RefType;
import java.io.File;
import java.lang.reflect.Method;
import java.nio.file.Files;
@@ -20,9 +22,12 @@ import java.util.HexFormat;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.ArrayList;
/** Injects reported analysis flags into imported ELF/Mach-O functions; never executes target code. */
public class ReaNoReturnProbe extends GhidraScript {
private record MemoryCallReference(Address source, Address target, RefType type,
SourceType sourceType, int operandIndex, boolean primary) {}
private final JsonArray checks = new JsonArray();
private final Map<Function, Address> originalEnds = new LinkedHashMap<>();
private final Map<Function, String> originalSignatures = new LinkedHashMap<>();
@@ -85,6 +90,114 @@ public class ReaNoReturnProbe extends GhidraScript {
return instance;
}
private void knownInteriorCall(GhidraScript bridge) throws Exception {
Function caller = function("rea_tls_chain");
Function target = function("rea_tls");
Address interior = target.getEntryPoint().add(1);
require("interior_target_is_owned", currentProgram.getFunctionManager().getFunctionAt(interior) == null &&
target.equals(currentProgram.getFunctionManager().getFunctionContaining(interior)));
Instruction call = null;
for (Instruction instruction : currentProgram.getListing().getInstructions(caller.getBody(), true)) {
if (instruction.getFlowType().isCall()) { call = instruction; break; }
}
if (call == null) throw new IllegalStateException("Missing real call instruction");
var references = currentProgram.getReferenceManager();
List<Reference> originals = new ArrayList<>();
List<MemoryCallReference> saved = new ArrayList<>();
for (Reference reference : call.getReferencesFrom()) {
if (!reference.getReferenceType().isCall()) continue;
originals.add(reference);
saved.add(new MemoryCallReference(reference.getFromAddress(), reference.getToAddress(),
reference.getReferenceType(), reference.getSource(), reference.getOperandIndex(), reference.isPrimary()));
}
require("original_call_reference_present", !originals.isEmpty());
Reference changed = null;
try {
for (Reference reference : originals) references.delete(reference);
changed = references.addMemoryReference(call.getAddress(), interior,
RefType.UNCONDITIONAL_CALL, SourceType.USER_DEFINED, 0);
require("interior_target_omitted_by_callee_getter", !caller.getCalledFunctions(monitor).contains(target));
JsonObject query = new JsonObject();
query.addProperty("document", currentProgram.getName());
query.addProperty("procedure", caller.getName());
query.addProperty("direction", "outgoing");
Method read = bridge.getClass().getDeclaredMethod("procedureReferences", JsonObject.class);
read.setAccessible(true);
JsonObject output = (JsonObject) read.invoke(bridge, query);
require("interior_reference_keeps_known_procedure", output.getAsJsonArray("references").asList().stream().anyMatch(
edge -> !edge.getAsJsonObject().get("target_procedure").isJsonNull() &&
edge.getAsJsonObject().getAsJsonObject("target_procedure").get("name").getAsString().equals(target.getName())));
require("known_reference_is_not_also_unresolved", output.getAsJsonArray("unresolved_calls").isEmpty());
query.remove("direction");
Method callees = bridge.getClass().getDeclaredMethod("procedureCalls", JsonObject.class, boolean.class);
callees.setAccessible(true);
JsonObject inventory = (JsonObject) callees.invoke(bridge, query, false);
require("omitted_callee_inventory_stays_partial", inventory.getAsJsonObject("relationship_coverage").get("status").getAsString().equals("partial"));
require("omitted_callee_direction_is_explicit",
inventory.getAsJsonObject("relationship_coverage").getAsJsonArray("limited_directions").toString().equals("[\"callees\"]"));
require("interior_caller_omitted_by_entry_getter", !target.getCallingFunctions(monitor).contains(caller));
query.addProperty("procedure", target.getName());
JsonObject incoming = (JsonObject) callees.invoke(bridge, query, true);
require("off_entry_caller_inventory_stays_partial",
incoming.getAsJsonObject("relationship_coverage").get("status").getAsString().equals("partial"));
require("off_entry_caller_direction_is_explicit",
incoming.getAsJsonObject("relationship_coverage").getAsJsonArray("limited_directions").toString().equals("[\"callers\"]"));
JsonObject dossierQuery = new JsonObject();
dossierQuery.addProperty("procedure", target.getName());
Method analyze = bridge.getClass().getDeclaredMethod("analyzeFunctionObservation", JsonObject.class);
analyze.setAccessible(true);
Object observation = analyze.invoke(bridge, dossierQuery);
Method coverage = observation.getClass().getDeclaredMethod("relationshipCoverage");
coverage.setAccessible(true);
require("dossier_preserves_caller_only_gap",
((JsonObject) coverage.invoke(observation)).getAsJsonArray("limited_directions").toString().equals("[\"callers\"]"));
} finally {
if (changed != null) references.delete(changed);
for (MemoryCallReference reference : saved) {
Reference restored = references.addMemoryReference(reference.source(), reference.target(),
reference.type(), reference.sourceType(), reference.operandIndex());
references.setPrimary(restored, reference.primary());
}
}
}
private void nonCallInstructionReference(GhidraScript bridge) throws Exception {
Function caller = function("rea_tls_chain");
Function target = function("rea_memcpy");
Instruction source = null;
for (Instruction instruction : currentProgram.getListing().getInstructions(caller.getBody(), true)) {
if (!instruction.getFlowType().isCall()) { source = instruction; break; }
}
if (source == null) throw new IllegalStateException("Missing non-call instruction");
var references = currentProgram.getReferenceManager();
Reference added = references.addMemoryReference(source.getAddress(), target.getEntryPoint().add(1),
RefType.UNCONDITIONAL_CALL, SourceType.USER_DEFINED, -1);
try {
require("non_call_reference_target_omitted", !caller.getCalledFunctions(monitor).contains(target));
JsonObject query = new JsonObject();
query.addProperty("document", currentProgram.getName());
query.addProperty("procedure", caller.getName());
Method read = bridge.getClass().getDeclaredMethod("procedureCalls", JsonObject.class, boolean.class);
read.setAccessible(true);
JsonObject output = (JsonObject) read.invoke(bridge, query, false);
require("non_call_instruction_reference_stays_partial",
output.getAsJsonObject("relationship_coverage").get("status").getAsString().equals("partial"));
query.addProperty("direction", "outgoing");
Method edges = bridge.getClass().getDeclaredMethod("procedureReferences", JsonObject.class);
edges.setAccessible(true);
JsonObject known = (JsonObject) edges.invoke(bridge, query);
require("non_call_known_reference_is_not_unresolved", known.getAsJsonArray("unresolved_calls").isEmpty());
references.delete(added);
Address unknown = currentProgram.getMemory().getMaxAddress().add(0x1000);
added = references.addMemoryReference(source.getAddress(), unknown,
RefType.UNCONDITIONAL_CALL, SourceType.USER_DEFINED, -1);
JsonObject unresolved = (JsonObject) edges.invoke(bridge, query);
require("non_call_unknown_reference_keeps_site", unresolved.getAsJsonArray("unresolved_calls").size() == 1);
} finally {
references.delete(added);
}
}
@Override
public void run() throws Exception {
String[] args = getScriptArgs();
@@ -164,6 +277,8 @@ public class ReaNoReturnProbe extends GhidraScript {
Object output = pseudocode.invoke(bridge, query);
require("recovered_pseudocode", output instanceof JsonObject value &&
value.get("value").getAsString().contains("rea_after_call"));
nonCallInstructionReference(bridge);
knownInteriorCall(bridge);
} finally {
((ghidra.app.decompiler.DecompInterface) field.get(bridge)).dispose();
}
@@ -34,6 +34,10 @@ it
const procRoot = join(directory, "proc");
await mkdir(procRoot);
const alternateMode = mode === "alternate launcher";
let orphanPid: number | undefined;
const isolatedProcessList = async () => ({
stdout: alternateMode || orphanPid === undefined ? "" : "Xvfb",
});
if (alternateMode) {
await copyFile("/usr/bin/sleep", launcher);
await chmod(launcher, 0o700);
@@ -47,6 +51,7 @@ it
await assertCleanLinuxHopperRuntime(
alternateMode ? [launcher] : [],
procRoot,
isolatedProcessList,
);
const command = alternateMode ? alternate : process.execPath;
const arguments_ = alternateMode ? ["30"] : ["-e", childCode];
@@ -69,7 +74,6 @@ it
const parentPid = parent.child.pid;
if (parentPid === undefined)
throw new Error("Fixture parent did not spawn");
let orphanPid: number | undefined;
try {
const { stdout } = await parent;
const { pid } = JSON.parse(stdout) as { pid: number };
@@ -115,6 +119,7 @@ it
assertCleanLinuxHopperRuntime(
alternateMode ? [launcher] : [],
procRoot,
isolatedProcessList,
),
).rejects.toThrow("Clean Linux runner retained a Hopper demo process");
// Observation must not terminate the fixture, even though it resembles Xvfb.