Compare commits

..
Author SHA1 Message Date
Kaoru0822andClaude Opus 5.5 110151e523 fix(capture): keep URLs that WHATWG rejects instead of redacting userinfo
File URLs cannot carry userinfo (an "@" in a file host fails the parse),
and a special authority whose host or port WHATWG rejects fails the whole
URL. Neither has credentials, so the bridge now keeps their source text
and byte sidecar. Checked against Node's URL over 30,000 spellings: no
under-redaction, over-redaction 20,540 -> 5,865 (remaining: empty
userinfo, IPv4 range, IDNA, non-special and baseless references).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:26:29 +00:00
Kaoru0822 b93e585d68 Merge remote-tracking branch 'origin/main' into fix/issue-2671 2026-10-14 00:22:25 +00:00
Xiangfang ChenandGitHub e049ef70f7 feat(setup): add WorkBuddy client support (#3041)
* feat(setup): add WorkBuddy client support

Register rea in the global ~/.workbuddy/mcp.json through a new workbuddy
entry in SUPPORTED_CLIENT_DEFINITIONS (json dialect, ~/.workbuddy marker,
~/.workbuddy/skills skill directory; WorkBuddy AI shares the file).
Document the client, extend the unreleased-client note, and cover
registration and uninstall with boundary tests.

* docs(setup): record verified WorkBuddy tool calls

Real-client verification completed: WorkBuddy loads the registration and
real REA tool calls execute through the registered server.
2026-10-14 08:12:33 +08:00
bbf0638e88 fix(javascript): report EventSource and sendBeacon URLs as network endpoints (#3044)
Co-authored-by: xpeng5278-web <330743750+xpeng5278-web@users.noreply.github.com>
2026-10-14 08:10:04 +08:00
242e2b0001 fix(ghidra): preserve function facet and call coverage (#3002)
* fix(ghidra): preserve observed function facet and call coverage

Declare successful query availability in the Java producer, including explicit unavailable pseudocode reasons. Carry observed static call-set coverage through the parsed provider result and Evidence without inventing facts for legacy omissions. Use Ghidra indexed comment addresses over the function body and entry, retaining cancellation checks and sparse collection.

Closes #2951

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

* fix(ghidra): retain unresolved call coverage in observations

Share decoded unresolved-call facts between dossiers, callee coverage and reference observations. Combine terminal and unresolved observations in one instruction traversal and retain site records only when requested. Keep missing procedure identities incomplete and validate unresolved locations at the provider boundary.

Bind snapshots to the corrected call coverage profile so older complete claims cannot replay under the new semantics. Add real public MCP callback trace/comparison regressions.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

* fix(ghidra): separate reference targets from callee coverage

Reuse containing-function ownership for resolved reference identities and independently compare those identities with the actual returned callee set. Keep known interior routes out of unresolved site records while retaining partial inventory coverage.

Advance the coverage profile to v3 and reject interim v2 cache bindings. Add a real Ghidra producer regression with restored reference metadata; the previous bridge fails the known-route assertion.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

* fix(ghidra): measure incoming call coverage across function bodies

Compare incoming reference source identities across the owned function body and entry with the actual returned caller set. Stream caller-only checks and reuse dossier reference observations. Combine caller/callee coverage from observed counts without reinterpreting serialized metadata, and share the body scope owner with comments and references.

Bind snapshots to function-body coverage semantics v4 and preserve the real off-entry caller regression.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

* fix(ghidra): preserve whole-scope directional call coverage

Check body-and-entry call references independently of decoded instruction flow. Reuse collected dossier references and stream inventory-only coverage checks. Keep caller-only gaps distinct from incomplete outgoing coverage in offline path reconstruction, and invalidate prior snapshot profiles.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

* fix(mcp): retain compact Evidence field types

Share optional diagnostics across the three tagged relationship coverage states instead of advertising duplicate branches. Preserve coverage validation while keeping compact Evidence properties within the existing reference expansion budget. Verify invalid direction cardinalities against the actual SDK-advertised schema.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

---------

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-14 07:52:14 +08:00
morluto 929aa67336 chore(main): merge pinned CI checkpoint 2895a18 2026-10-14 03:12:47 +08:00
morluto 4e8475e965 test(ci): isolate alternate launcher process fixture 2026-10-14 03:02:01 +08:00
morluto 557148ed3f fix(ci): declare process inventory callback 2026-10-14 02:25:19 +08:00
morluto f839046776 chore(merge): refresh PR #2767 with latest main 2026-10-14 02:23:57 +08:00
Morluto 281c4b8ec5 test(ci): isolate Hopper process inventory fixtures 2026-10-14 02:15:33 +08:00
Morluto a0da18d7fa test(capture): cover empty URL userinfo markers 2026-10-14 02:14:51 +08:00
Morluto 2c6e074cac fix(capture): remove empty URL userinfo markers 2026-10-14 02:14:51 +08:00
morluto 29c871e51b chore(merge): refresh PR #2767 with main 2026-10-14 02:10:19 +08:00
REA MaintenanceandKaoru0822 c0018ca403 fix(ci): isolate Linux Hopper process fixture inventory 2026-10-14 01:23:30 +08:00
morlutoandKaoru0822 ad16d250e3 fix(android): make DEX descriptor pattern portable 2026-10-14 01:23:25 +08:00
Lucienandmorluto af2f55cda9 fix(comparison): recheck historical calls-only changes (#2223)
* fix(comparison): preserve incomplete function call evidence

Keep missing, unknown, partial and unresolved call relationships unknown in function comparisons and reconstruction claims. Aggregate IDA caller and callee coverage without overwriting uncertainty.

Validate with 144 focused tests, typecheck/lint, docs checks and test discovery. Real providers were not verified; the full headless IDA session suite requires unavailable Windows native controls.

* fix(comparison): recheck historical calls-only changes

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-14 01:23:19 +08:00
morluto 4d5d1f5daa fix(capture): preserve file-relative path evidence after separators 2026-10-13 23:02:00 +08:00
morluto 0aff997b31 fix(capture): distinguish native paths and URL base authorities 2026-10-13 21:35:04 +08:00
morluto e8361b7715 fix(capture): parse URL controls and mixed network authorities 2026-10-13 20:31:54 +08:00
morluto 72e931f3a3 chore(ci): retain contributor updates alongside verification fixes 2026-10-13 20:21:10 +08:00
morluto 0266e6552b fix(capture): preserve same-scheme relative redirect paths 2026-10-13 20:19:00 +08:00
Kaoru0822 7d972a7e85 Merge remote-tracking branch 'origin/main' into fix/issue-2671 2026-10-13 12:04:14 +00:00
Kaoru0822andClaude Opus 5.5 e1fb40460d fix(capture): skip special-scheme leading slashes before userinfo
Ending the authority at a backslash exposed a credential: for
`https://\user:pass@host/path`, WHATWG parsing skips every leading "/"
or "\" after a special scheme (and in scheme-relative references), so
`user:pass` is userinfo, but the new boundary stopped at the first
backslash and found no "@". The same applied to `https:\\user:pass@host`
and `https:user:pass@host`.

Start the authority after that whole slash run for non-file special
schemes and scheme-relative references. File URLs keep their "//"
boundary because they carry no userinfo. Load the adapter in tests with
fresh mitmproxy stand-ins scoped to its import instead of leaving them
in sys.modules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 11:25:05 +00:00
Kaoru0822andClaude Opus 5.5 553cd6e358 fix(capture): end special-scheme URL authorities at a backslash
The native mitmproxy adapter removed transport userinfo with
`^(...//)[^/?#]*@`, so the authority ran to the last "@" before "/", "?"
or "#". For `https://example.test\part@tail` it treated
`example.test\part` as userinfo, reported `https://tail` as the
Location, and withdrew the original bytes as a transport credential.
WHATWG URL parsing, used by the shared TypeScript owner
(recordedTransportUrl.ts) and by HAR captures, ends a special-scheme or
scheme-relative authority at a backslash, so that "@" is ordinary path
text.

Locate the authority with the same boundary rule before removing
userinfo. Bytes are handled as Latin-1 text so unrelated bytes stay
identical. Userinfo before the backslash is still removed and redacted.
Add source unit tests for the adapter projection to test:python.

Fixes #2671

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 11:06:25 +00:00
37 changed files with 1670 additions and 284 deletions
+282 -97
View File
@@ -81,6 +81,7 @@ import ghidra.program.model.listing.Function;
import ghidra.program.model.listing.FunctionIterator;
import ghidra.program.model.listing.Instruction;
import ghidra.program.model.listing.InstructionIterator;
import ghidra.program.model.listing.Listing;
import ghidra.program.model.listing.Variable;
import ghidra.program.model.mem.MemoryBlock;
import ghidra.program.model.mem.MemoryBlockSourceInfo;
@@ -405,7 +406,9 @@ public final class ReaGhidraBridge extends HeadlessScript {
case "xrefs" -> xrefs(request.params);
case "analyze_function" -> {
FunctionAnalysisObservation observation = analyzeFunctionObservation(request.params);
yield functionObservation(observation.dossier(), observation.limitations());
JsonObject result = functionObservation(observation.dossier(), observation.limitations());
result.add("relationship_coverage", observation.relationshipCoverage());
yield result;
}
case "set_address_name", "set_addresses_names" -> {
if (!descriptor.transport.equals("unix-socket"))
@@ -1016,12 +1019,19 @@ public final class ReaGhidraBridge extends HeadlessScript {
for (Function item : ordered) {
result.add(canonicalAddress(item.getEntryPoint()));
}
return functionObservation(
result,
noReturnCallLimitations(
if (callers) {
JsonObject observation = functionObservation(result, noReturnCallLimitations(
currentProgram.getListing().getInstructions(function.getBody(), true)
)
));
observation.add("relationship_coverage", callRelationshipCoverage(null,
unrepresentedIncomingCalls(function, observed)));
monitor.checkCancelled();
return observation;
}
CallBoundaryObservation calls = callBoundaryObservation(
function, currentProgram.getListing().getInstructions(function.getBody(), true), observed, false, null
);
return callRelationshipObservation(result, calls);
}
private JsonObject procedureInfo(JsonObject params) throws Exception {
@@ -1071,18 +1081,10 @@ public final class ReaGhidraBridge extends HeadlessScript {
result.addProperty("direction", direction);
result.add("references", edges);
result.addProperty("reference_kinds_available", true);
JsonArray unresolvedCalls = new JsonArray();
if (direction.equals("outgoing")) for (Instruction instruction : scanInstructions(function, Integer.MAX_VALUE).instructions) {
if (!instruction.getFlowType().isCall()) continue;
boolean resolved = false;
for (Reference reference : instruction.getReferencesFrom()) if (reference.getReferenceType().isCall()) resolved = true;
if (!resolved) {
JsonObject site = new JsonObject();
site.addProperty("address", canonicalAddress(instruction.getAddress()));
site.addProperty("reason", "Instruction decoder identifies a call but ReferenceManager supplies no call destination");
unresolvedCalls.add(site);
}
}
JsonArray unresolvedCalls = direction.equals("outgoing")
? callBoundaryObservation(
function, currentProgram.getListing().getInstructions(function.getBody(), true), null, true, references
).unresolvedCalls() : new JsonArray();
result.add("unresolved_calls", unresolvedCalls);
return result;
}
@@ -1490,16 +1492,28 @@ public final class ReaGhidraBridge extends HeadlessScript {
InstructionScan scan = scanInstructions(function, Integer.MAX_VALUE);
DecompileResults decompilation = decompile(function);
String pseudocode = decompiledText(decompilation);
List<Reference> incomingReferences = collectReferences(function, "incoming")
.stream()
JsonObject availability = new JsonObject();
availability.add("pseudocode", facetAvailability(
pseudocode != null,
function.isExternal() ? "Ghidra does not decompile external functions."
: function.getBody().isEmpty() ? "Ghidra has no function body to decompile."
: "Ghidra completed decompilation without returning pseudocode."
));
if (pseudocode == null) {
pseudocode = "";
}
List<Reference> bodyIncoming = collectReferences(function, "incoming");
List<Reference> incomingReferences = bodyIncoming.stream()
.filter(reference -> !function.getBody().contains(reference.getFromAddress()))
.toList();
List<Reference> outgoingReferences = collectReferences(function, "outgoing");
JsonArray incoming = referenceEdges(incomingReferences);
JsonArray outgoing = referenceEdges(outgoingReferences);
JsonArray comments = comments(scan.instructions);
JsonArray callers = procedureIdentities(function.getCallingFunctions(monitor));
JsonArray callees = procedureIdentities(function.getCalledFunctions(monitor));
JsonArray comments = comments(function);
Set<Function> callerFunctions = function.getCallingFunctions(monitor);
JsonArray callers = procedureIdentities(callerFunctions);
Set<Function> calleeFunctions = function.getCalledFunctions(monitor);
JsonArray callees = procedureIdentities(calleeFunctions);
JsonArray referencedStrings = referencedStrings(outgoingReferences);
JsonArray referencedNames = referencedNames(outgoingReferences);
JsonArray blocks = basicBlockValues(function);
@@ -1511,7 +1525,7 @@ public final class ReaGhidraBridge extends HeadlessScript {
procedure.add("locals", functionLocals(function));
JsonObject result = new JsonObject();
result.add("procedure", procedure);
result.addProperty("pseudocode", pseudocode == null ? "" : pseudocode);
result.addProperty("pseudocode", pseudocode);
result.add("assembly", assembly);
result.add("comments", comments);
result.add("callers", callers);
@@ -1520,20 +1534,6 @@ public final class ReaGhidraBridge extends HeadlessScript {
result.add("outgoing_references", outgoing);
result.add("referenced_strings", referencedStrings);
result.add("referenced_names", referencedNames);
JsonObject facets = new JsonObject();
JsonObject pseudocodeAvailability = new JsonObject();
pseudocodeAvailability.addProperty("available", pseudocode != null);
if (pseudocode == null) {
pseudocodeAvailability.addProperty("reason", "Ghidra did not supply pseudocode.");
}
facets.add("pseudocode", pseudocodeAvailability);
for (String facet : List.of("assembly", "comments", "callers", "callees",
"references", "strings_names", "cfg")) {
JsonObject available = new JsonObject();
available.addProperty("available", true);
facets.add(facet, available);
}
result.add("facet_availability", facets);
boolean swift = swiftProgram();
boolean aarch64 = currentProgram.getLanguageID().getIdAsString().startsWith("AARCH64:");
SwiftStringLiteralDecode swiftLiterals = swift && aarch64 ? swiftStringLiterals(function) : null;
@@ -1547,6 +1547,15 @@ public final class ReaGhidraBridge extends HeadlessScript {
nativeApiBoundary(function, decompilation)
);
result.add("native_value_flow", nativeValueFlow(decompilation));
// Successful full queries distinguish collected empty sets from missing facts.
// Availability does not claim that Ghidra resolved every runtime relationship.
for (String facet : List.of("comments", "callers", "callees", "references", "strings_names", "cfg")) {
availability.add(facet, facetAvailability(true, null));
}
availability.add("assembly", facetAvailability(
!scan.truncated(), "The function instruction scan was truncated."
));
result.add("facet_availability", availability);
JsonArray limitations = new JsonArray();
limitations.add(
"Unresolved computed or indirect flows without target addresses are not represented as reference edges."
@@ -1574,8 +1583,10 @@ public final class ReaGhidraBridge extends HeadlessScript {
"Swift String literal decoding is implemented for AArch64 register pairs only; no Swift literals were decoded for this architecture."
);
}
List<String> boundaryLimitations = noReturnCallLimitations(scan.instructions);
for (String limitation : boundaryLimitations) {
CallBoundaryObservation calls = callBoundaryObservation(
function, scan.instructions, calleeFunctions, true, outgoingReferences);
result.add("unresolved_calls", calls.unresolvedCalls());
for (String limitation : calls.limitations()) {
limitations.add(limitation);
}
if (decompilation != null && decompilation.getErrorMessage() != null &&
@@ -1583,7 +1594,14 @@ public final class ReaGhidraBridge extends HeadlessScript {
limitations.add("Ghidra decompiler diagnostic: " + decompilation.getErrorMessage());
}
result.add("limitations", limitations);
return new FunctionAnalysisObservation(result, boundaryLimitations);
monitor.checkCancelled();
int missingCallers = 0;
for (Reference reference : bodyIncoming) {
monitor.checkCancelled();
if (unrepresentedCaller(reference, callerFunctions)) missingCallers++;
}
return new FunctionAnalysisObservation(result, calls.limitations(),
callRelationshipCoverage(calls, missingCallers));
}
private JsonObject nativeValueFlow(DecompileResults decompilation) throws Exception {
@@ -3287,54 +3305,202 @@ public final class ReaGhidraBridge extends HeadlessScript {
return result;
}
private static JsonObject facetAvailability(boolean available, String reason) {
JsonObject result = new JsonObject();
result.addProperty("available", available);
if (!available) result.addProperty("reason", reason);
return result;
}
private JsonObject callRelationshipObservation(JsonElement value, CallBoundaryObservation calls)
throws Exception {
monitor.checkCancelled();
JsonObject result = functionObservation(value, calls.limitations());
result.add("relationship_coverage", callRelationshipCoverage(calls, null));
return result;
}
private static JsonObject callRelationshipCoverage(
CallBoundaryObservation calls, Integer missingCallers) {
JsonObject coverage = completeResolvedCallCoverage();
Integer omitted = calls == null ? null : calls.unrepresentedCalleeCalls();
JsonArray directions = new JsonArray();
if (missingCallers != null && missingCallers > 0) directions.add("callers");
String callerLimitation = missingCallers != null && missingCallers > 0
? "Ghidra's caller inventory omits sources of " + missingCallers +
" call reference(s) into this function's body." : null;
if ((calls != null && omitted == null) || (calls == null && missingCallers == null)) {
coverage.addProperty("status", "unknown");
String reason = calls == null ? "No call inventory was examined."
: "The callee inventory was not examined alongside decoded call targets.";
coverage.addProperty("reason", reason + (callerLimitation == null ? "" : " " + callerLimitation));
if (calls == null && missingCallers == null) directions.add("callers");
directions.add("callees");
coverage.add("limited_directions", directions);
return coverage;
}
JsonArray limitations = new JsonArray();
if (callerLimitation != null) limitations.add(callerLimitation);
if (omitted != null && omitted > 0) {
limitations.add("Ghidra's callee inventory omits targets of " + omitted +
" call reference(s) or decoded call(s) without destinations.");
directions.add("callees");
}
if (!limitations.isEmpty()) {
coverage.addProperty("status", "partial");
coverage.add("limitations", limitations);
coverage.add("limited_directions", directions);
}
return coverage;
}
private static JsonObject completeResolvedCallCoverage() {
JsonObject coverage = new JsonObject();
coverage.addProperty("status", "complete");
JsonArray diagnostics = new JsonArray();
diagnostics.add(
"Only resolved Ghidra call references are returned; unresolved computed or indirect calls remain unknown, while function classifications distinguish thunks and externals."
);
coverage.add("diagnostics", diagnostics);
return coverage;
}
private boolean unrepresentedCaller(Reference reference, Set<Function> callers) {
if (!reference.getReferenceType().isCall()) return false;
Function caller = containingFunction(reference.getFromAddress());
return caller == null || !callers.contains(caller);
}
private int unrepresentedIncomingCalls(Function function, Set<Function> callers)
throws Exception {
int[] omitted = { 0 };
visitReferences(function, "incoming", reference -> {
if (unrepresentedCaller(reference, callers)) omitted[0]++;
});
return omitted[0];
}
private List<String> noReturnCallLimitations(Iterable<Instruction> instructions)
throws Exception {
List<String> limitations = new ArrayList<>();
for (Instruction instruction : instructions) {
monitor.checkCancelled();
if (!instruction.getFlowType().isCall() || !instruction.getFlowType().isTerminal()) {
continue;
}
Address[] destinations = instruction.getFlows();
StringBuilder limitation = new StringBuilder();
limitation.append("Ghidra Listing reports a terminal call at ")
.append(canonicalAddress(instruction.getAddress()));
if (destinations.length == 0) {
limitation.append(" with no resolved callee; its no-return basis is unknown.");
}
else {
limitation.append(" to ");
for (int index = 0; index < destinations.length; index++) {
if (index > 0) limitation.append(", ");
Address destination = destinations[index];
limitation.append(canonicalAddress(destination));
Function target = currentProgram.getFunctionManager().getFunctionAt(destination);
if (target == null) {
limitation.append(" (callee identity unknown)");
}
else {
limitation.append(" ").append(procedureName(target));
limitation.append(" (external=").append(target.isExternal());
limitation.append(", thunk=").append(target.isThunk());
limitation.append(", hasNoReturn=").append(target.hasNoReturn()).append(")");
}
}
limitation.append("; fallthrough is excluded by Ghidra's flow model.");
Function target = destinations.length == 1
? currentProgram.getFunctionManager().getFunctionAt(destinations[0])
: null;
if (target != null && target.hasNoReturn()) {
limitation.append(" This records Ghidra's FunctionManager flag, not an independent verification that the callee cannot return.");
}
else {
limitation.append(" The reason for the terminal flow is not established by this observation.");
}
}
limitations.add(limitation.toString());
String limitation = terminalCallLimitation(instruction);
if (limitation != null) limitations.add(limitation);
}
return limitations;
}
private CallBoundaryObservation callBoundaryObservation(
Function function, Iterable<Instruction> instructions, Set<Function> returnedCallees,
boolean includeUnresolvedSites, List<Reference> observedReferences)
throws Exception {
List<String> limitations = new ArrayList<>();
CallBoundaryReferences calls = new CallBoundaryReferences(returnedCallees, includeUnresolvedSites);
if (observedReferences == null) visitReferences(function, "outgoing", calls::observe);
else for (Reference reference : observedReferences) {
monitor.checkCancelled();
calls.observe(reference);
}
for (Instruction instruction : instructions) {
monitor.checkCancelled();
String limitation = terminalCallLimitation(instruction);
if (limitation != null) limitations.add(limitation);
if (!instruction.getFlowType().isCall()) continue;
boolean destination = false;
for (Reference reference : instruction.getReferencesFrom()) {
if (!reference.getReferenceType().isCall()) continue;
destination = true;
break;
}
if (!destination) {
calls.unrepresented++;
calls.unresolved(instruction.getAddress(),
"Instruction decoder identifies a call but ReferenceManager supplies no call destination");
}
}
return new CallBoundaryObservation(List.copyOf(limitations),
returnedCallees == null ? null : calls.unrepresented, calls.unresolvedCalls());
}
private final class CallBoundaryReferences {
private final Set<Function> returnedCallees;
private final TreeMap<Address, String> unresolved;
private int unrepresented;
CallBoundaryReferences(Set<Function> returnedCallees, boolean includeUnresolvedSites) {
this.returnedCallees = returnedCallees;
unresolved = includeUnresolvedSites ? new TreeMap<>() : null;
}
void observe(Reference reference) {
if (!reference.getReferenceType().isCall()) return;
// Reference identity follows the same containing-function owner as referenceEdge.
Function target = containingFunction(reference.getToAddress());
if (returnedCallees != null && (target == null || !returnedCallees.contains(target)))
unrepresented++;
if (target == null) unresolved(reference.getFromAddress(),
"A call destination has no known containing Ghidra procedure");
}
void unresolved(Address address, String reason) {
if (unresolved != null) unresolved.put(address, reason);
}
JsonArray unresolvedCalls() {
if (unresolved == null) return null;
JsonArray result = new JsonArray();
for (var item : unresolved.entrySet()) {
JsonObject site = new JsonObject();
site.addProperty("address", canonicalAddress(item.getKey()));
site.addProperty("reason", item.getValue());
result.add(site);
}
return result;
}
}
private String terminalCallLimitation(Instruction instruction) {
if (!instruction.getFlowType().isCall() || !instruction.getFlowType().isTerminal())
return null;
Address[] destinations = instruction.getFlows();
StringBuilder limitation = new StringBuilder();
limitation.append("Ghidra Listing reports a terminal call at ")
.append(canonicalAddress(instruction.getAddress()));
if (destinations.length == 0) {
limitation.append(" with no resolved callee; its no-return basis is unknown.");
}
else {
limitation.append(" to ");
for (int index = 0; index < destinations.length; index++) {
if (index > 0) limitation.append(", ");
Address destination = destinations[index];
limitation.append(canonicalAddress(destination));
Function target = currentProgram.getFunctionManager().getFunctionAt(destination);
if (target == null) {
limitation.append(" (callee identity unknown)");
}
else {
limitation.append(" ").append(procedureName(target));
limitation.append(" (external=").append(target.isExternal());
limitation.append(", thunk=").append(target.isThunk());
limitation.append(", hasNoReturn=").append(target.hasNoReturn()).append(")");
}
}
limitation.append("; fallthrough is excluded by Ghidra's flow model.");
Function target = destinations.length == 1
? currentProgram.getFunctionManager().getFunctionAt(destinations[0])
: null;
if (target != null && target.hasNoReturn()) {
limitation.append(" This records Ghidra's FunctionManager flag, not an independent verification that the callee cannot return.");
}
else {
limitation.append(" The reason for the terminal flow is not established by this observation.");
}
}
return limitation.toString();
}
private static String renderAssembly(List<Instruction> instructions) {
return String.join("\n", renderAssemblyLines(instructions));
}
@@ -3358,15 +3524,27 @@ public final class ReaGhidraBridge extends HeadlessScript {
private List<Reference> collectReferences(
Function function,
String direction) throws Exception {
TreeMap<String, Reference> observed = new TreeMap<>();
visitReferences(function, direction, reference -> addReference(observed, reference));
List<Reference> result = new ArrayList<>(observed.values());
result.sort(REFERENCE_ORDER);
return result;
}
@FunctionalInterface
private interface ReferenceVisitor {
void accept(Reference reference) throws Exception;
}
private void visitReferences(Function function, String direction, ReferenceVisitor visitor)
throws Exception {
// References can target instruction interiors or originate in embedded
// data. Use actual body ownership, including an external's exact entry.
AddressSet scope = new AddressSet(function.getBody());
scope.add(function.getEntryPoint());
AddressSet scope = functionObservationScope(function);
boolean outgoing = direction.equals("outgoing");
AddressIterator addresses = outgoing
? currentProgram.getReferenceManager().getReferenceSourceIterator(scope, true)
: currentProgram.getReferenceManager().getReferenceDestinationIterator(scope, true);
TreeMap<String, Reference> observed = new TreeMap<>();
while (addresses.hasNext()) {
monitor.checkCancelled();
Address address = addresses.next();
@@ -3374,7 +3552,7 @@ public final class ReaGhidraBridge extends HeadlessScript {
for (Reference reference : currentProgram.getReferenceManager()
.getReferencesFrom(address)) {
monitor.checkCancelled();
addReference(observed, reference);
if (!reference.isEntryPointReference()) visitor.accept(reference);
}
}
else {
@@ -3382,19 +3560,14 @@ public final class ReaGhidraBridge extends HeadlessScript {
.getReferencesTo(address);
while (iterator.hasNext()) {
monitor.checkCancelled();
addReference(observed, iterator.next());
Reference reference = iterator.next();
if (!reference.isEntryPointReference()) visitor.accept(reference);
}
}
}
List<Reference> result = new ArrayList<>(observed.values());
result.sort(REFERENCE_ORDER);
return result;
}
private static void addReference(TreeMap<String, Reference> observed, Reference reference) {
if (reference.isEntryPointReference()) {
return;
}
String key = canonicalAddress(reference.getFromAddress()) + "\u0000" +
canonicalAddress(reference.getToAddress()) + "\u0000" +
reference.getReferenceType().getName() + "\u0000" +
@@ -3403,6 +3576,12 @@ public final class ReaGhidraBridge extends HeadlessScript {
observed.putIfAbsent(key, reference);
}
private static AddressSet functionObservationScope(Function function) {
AddressSet scope = new AddressSet(function.getBody());
scope.add(function.getEntryPoint());
return scope;
}
private JsonArray referenceEdges(List<Reference> references) {
JsonArray result = new JsonArray();
for (Reference reference : references) {
@@ -3556,23 +3735,28 @@ public final class ReaGhidraBridge extends HeadlessScript {
return result;
}
private static JsonArray comments(List<Instruction> instructions) {
private JsonArray comments(Function function) throws Exception {
Listing listing = currentProgram.getListing();
AddressSet scope = functionObservationScope(function);
AddressIterator addresses = listing.getCommentAddressIterator(scope, true);
TreeMap<String, JsonObject> observed = new TreeMap<>();
for (Instruction instruction : instructions) {
while (addresses.hasNext()) {
monitor.checkCancelled();
Address address = addresses.next();
for (CommentType type : CommentType.values()) {
String text = instruction.getComment(type);
String text = listing.getComment(type, address);
if (text == null || text.isEmpty()) {
continue;
}
String kind = type == CommentType.EOL ? "inline" : "comment";
JsonObject item = new JsonObject();
item.addProperty("address", canonicalAddress(instruction.getAddress()));
item.addProperty("address", canonicalAddress(address));
item.addProperty("kind", kind);
item.addProperty("text", text);
String key = canonicalAddress(instruction.getAddress()) + "\u0000" +
String key = canonicalAddress(address) + "\u0000" +
kind + "\u0000" + type.name() + "\u0000" + text;
observed.put(key, item);
}
}
}
JsonArray result = new JsonArray();
observed.values().forEach(result::add);
@@ -4508,7 +4692,8 @@ public final class ReaGhidraBridge extends HeadlessScript {
private record InventoryItem(Address address, String value, JsonObject facts) {}
private record AddressNameRequest(String inputAddress, Address address, String name) {}
private record FunctionEntry(Function function, InventoryItem item) {}
private record FunctionAnalysisObservation(JsonObject dossier, List<String> limitations) {}
private record FunctionAnalysisObservation(JsonObject dossier, List<String> limitations, JsonObject relationshipCoverage) {}
private record CallBoundaryObservation(List<String> limitations, Integer unrepresentedCalleeCalls, JsonArray unresolvedCalls) {}
private record SwiftStringLiteralDecode(JsonArray literals, JsonArray omissions) {}
private record SwiftLiteralLayout(
String representation,
+133 -12
View File
@@ -11,6 +11,7 @@ import math
import re
import resource
from pathlib import Path
from urllib.parse import unquote_to_bytes
from mitmproxy import ctx, version
from mitmproxy.io import tnetstring
@@ -41,6 +42,113 @@ class BoundedReader:
return self.handle.read(count)
SPECIAL_SCHEMES = {"ftp", "file", "http", "https", "ws", "wss"}
SCHEME_PREFIX = r"^[\x00-\x20]*([a-z][a-z0-9+.-]*):"
NETWORK_PREFIX = r"^[\x00-\x20]*[/\\]{2}"
URL_IGNORABLE_CONTROLS = "\t\n\r"
URL_CONTROL_TRANSLATION = {ord(char): None for char in URL_IGNORABLE_CONTROLS}
FORBIDDEN_DOMAIN_CODE_POINTS = {chr(code) for code in range(0x21)} | set("#%/:<>?@[\\]^|\x7f")
def authority_start(text, base_scheme=None):
"""Return where an authority begins and whether it is special, or None.
Follows WHATWG URL parsing as src/domain/recordedTransportUrl.ts does:
special schemes (and scheme-relative references) skip every leading "/"
or "\\" before the authority, so `https://\\user:pass@host` still has
userinfo. File URLs never carry userinfo: an "@" in a file host makes
WHATWG reject the whole URL, so there are no credentials to remove.
"""
scheme = re.match(SCHEME_PREFIX, text, flags=re.I)
name = scheme.group(1).lower() if scheme is not None else None
if name == "file" or (name is None and base_scheme == "file"):
return None
if name is not None and name in SPECIAL_SCHEMES:
# A same-scheme Location without two slash-equivalent separators is
# relative to the recorded request, so its @ belongs to the path.
following = text[scheme.end():scheme.end() + 2]
if name == base_scheme and (len(following) < 2 or any(char not in "/\\" for char in following)):
return None
start = scheme.end()
else:
if scheme is not None:
if text[scheme.end():scheme.end() + 2] != "//":
return None
return scheme.end() + 2, False
# A non-special base treats backslashes as authority data and only
# recognizes two forward slashes as a network reference.
if base_scheme is not None and base_scheme not in SPECIAL_SCHEMES:
prefix = re.match(r"^[\x00-\x20]*//", text)
return None if prefix is None else (prefix.end(), False)
prefix = re.match(NETWORK_PREFIX, text)
if prefix is None:
return None
start = prefix.end()
while start < len(text) and text[start] in "/\\":
start += 1
return start, True
def special_host_parses(host_port):
"""Whether WHATWG could accept a special authority's host and port.
A rejected host fails the whole URL, so it has no credentials to remove.
This mirrors the forbidden-code-point, empty-host and port rules; IDNA,
IPv4 range and IPv6 syntax failures are not modeled and still redact.
"""
if host_port.startswith("["):
host_end = host_port.find("]") + 1
if host_end == 0:
return False
else:
colon = host_port.find(":")
host_end = len(host_port) if colon < 0 else colon
host = unquote_to_bytes(host_port[:host_end]).decode("utf-8", errors="replace")
if host == "" or any(char in FORBIDDEN_DOMAIN_CODE_POINTS for char in host):
return False
port = host_port[host_end:]
if port == "":
return True
digits = port[1:]
return port[0] == ":" and (digits == "" or (digits.isascii() and digits.isdigit() and int(digits) <= 65535))
def without_userinfo(value, base_scheme=None):
"""Remove an authority's userinfo using the WHATWG authority boundary.
Special schemes and scheme-relative references also end the authority
at a backslash, so later path text containing "@" is not userinfo.
"""
binary = isinstance(value, bytes)
text = value.decode("latin-1") if binary else value
# WHATWG removes these controls before interpreting scheme/separators.
# Keep that parse view separate from the exact source spelling and bytes.
parsed_text = text.translate(URL_CONTROL_TRANSLATION)
located = authority_start(parsed_text, base_scheme)
if located is None:
return value
start, special = located
terminators = "/?#\\" if special else "/?#"
end = start
while end < len(parsed_text) and parsed_text[end] not in terminators:
end += 1
at = parsed_text.rfind("@", start, end)
if at < start or (special and not special_host_parses(parsed_text[at + 1 : end])):
return value
parsed_offset = 0
raw_start = 0
for raw_offset, char in enumerate(text):
if char in URL_IGNORABLE_CONTROLS:
continue
if parsed_offset == start:
raw_start = raw_offset
if parsed_offset == at:
safe = text[:raw_start] + text[raw_offset + 1 :]
return safe.encode("latin-1") if binary else safe
parsed_offset += 1
raise CaptureFailure("decoder", "Normalized URL authority could not be bound to its source.")
def project_record(state, secrets):
binaries, numbers, redactions = [], [], []
byte_secrets = []
@@ -58,16 +166,23 @@ def project_record(state, secrets):
return None
return text
def transport_value(value, path, transport_url):
def transport_value(value, path, transport_url, base_scheme):
if not (transport_url or re.fullmatch(r"/(?:backup/)*request/(?:path|authority)", path)):
return value, False
binary = isinstance(value, bytes)
pattern = rb"^([\x00-\x20]*(?:[a-z][a-z0-9+.-]*:)?//)[^/?#]*@" if binary else r"^([\x00-\x20]*(?:[a-z][a-z0-9+.-]*:)?//)[^/?#]*@"
replacement = rb"\1" if binary else r"\1"
separator = b"@" if binary else "@"
safe_url = re.sub(pattern, replacement, value, flags=re.I)
if path.endswith("/authority") and separator in safe_url:
safe_url = safe_url.rsplit(separator, 1)[1]
if transport_url:
safe_url = without_userinfo(value, base_scheme)
elif path.endswith("/authority"):
safe_url = value.rsplit(separator, 1)[-1]
else:
# Native request.path is origin-form unless it explicitly carries
# a scheme (absolute-form). Leading separators are path evidence,
# not a header URL reference's network authority.
text = value.decode("latin-1") if binary else value
safe_url = without_userinfo(value) if re.match(
SCHEME_PREFIX, text.translate(URL_CONTROL_TRANSLATION), flags=re.I
) else value
changed = safe_url != value
if changed:
redactions.append({"pointer": path, "reason": "transport-credential"})
@@ -80,7 +195,7 @@ def project_record(state, secrets):
del numbers[before[1]:]
del redactions[before[2]:]
def visit(value, path, depth=0, credential=False, transport_url=False):
def visit(value, path, depth=0, credential=False, transport_url=False, base_scheme=None):
if depth > 64:
raise CaptureFailure("input-limit", "Native capture exceeds the 64-level evidence nesting budget.", path)
if credential:
@@ -90,7 +205,7 @@ def project_record(state, secrets):
binaries.append({"pointer": path, "representation": "producer-bytes", "state": "redacted", "content_base64": None, "bytes": None, "sha256": None})
return None
if isinstance(value, bytes):
safe_bytes, credential_url = transport_value(value, path, transport_url)
safe_bytes, credential_url = transport_value(value, path, transport_url, base_scheme)
content_base64 = base64.b64encode(value).decode("ascii")
digest = hashlib.sha256(value).hexdigest()
hidden = any(secret in value or secret in safe_bytes for secret in byte_secrets) or any(secret in content_base64 or secret in digest or secret in str(len(value)) for secret in secrets)
@@ -107,7 +222,7 @@ def project_record(state, secrets):
return None
return safe_text
if isinstance(value, str):
safe_text, _ = transport_value(value, path, transport_url)
safe_text, _ = transport_value(value, path, transport_url, base_scheme)
return redact(safe_text, path, value)
if isinstance(value, bool) or value is None:
return value
@@ -125,10 +240,16 @@ def project_record(state, secrets):
for index, field in enumerate(value):
if not isinstance(field, (list, tuple)) or len(field) != 2 or not isinstance(field[0], bytes):
raise CaptureFailure("format", "Native header/trailer is not an ordered byte pair.", pointer(path, index))
result.append([visit(field[0], pointer(pointer(path, index), 0), depth + 2), visit(field[1], pointer(pointer(path, index), 1), depth + 2, field[0].lower() in CREDENTIALS, field[0].lower() in {b"location", b"referer", b"origin"})])
result.append([visit(field[0], pointer(pointer(path, index), 0), depth + 2), visit(field[1], pointer(pointer(path, index), 1), depth + 2, field[0].lower() in CREDENTIALS, field[0].lower() in {b"location", b"referer", b"origin"}, base_scheme if field[0].lower() == b"location" else None)])
return result
return [visit(item, pointer(path, index), depth + 1) for index, item in enumerate(value)]
return [visit(item, pointer(path, index), depth + 1, base_scheme=base_scheme) for index, item in enumerate(value)]
if isinstance(value, dict):
if path == "" or re.fullmatch(r"/backup(?:/backup)*", path):
request = value.get("request")
scheme = request.get("scheme") if isinstance(request, dict) else None
if isinstance(scheme, bytes):
scheme = scheme.decode("latin-1")
base_scheme = scheme.lower() if isinstance(scheme, str) and re.fullmatch(r"[a-z][a-z0-9+.-]*", scheme, flags=re.I) else None
result = {}
for key, item in value.items():
if not isinstance(key, str):
@@ -139,7 +260,7 @@ def project_record(state, secrets):
validate_omitted(item, pointer(path, key), depth + 1)
redactions.append({"pointer": path, "reason": "explicit-sensitive-value", "scope": "property-name"})
continue
result[key] = visit(item, pointer(path, key), depth + 1)
result[key] = visit(item, pointer(path, key), depth + 1, base_scheme=base_scheme)
return result
raise CaptureFailure("decoder", "Native parser returned an unexpected value type: " + type(value).__name__ + ".", path)
+214
View File
@@ -0,0 +1,214 @@
"""Source-owned tests for the mitmproxy adapter's record projection."""
import importlib.util
import os
import sys
import types
import unittest
from unittest import mock
def load_capture():
"""Load the adapter with fresh mitmproxy stand-ins scoped to its import."""
fake_mitmproxy = types.ModuleType("mitmproxy")
fake_mitmproxy.ctx = types.SimpleNamespace()
fake_mitmproxy.version = types.SimpleNamespace(VERSION="12.2.3")
fake_io = types.ModuleType("mitmproxy.io")
fake_io.tnetstring = types.ModuleType("mitmproxy.io.tnetstring")
stand_ins = {
"mitmproxy": fake_mitmproxy,
"mitmproxy.io": fake_io,
"mitmproxy.io.tnetstring": fake_io.tnetstring,
}
spec = importlib.util.spec_from_file_location(
"rea_mitmproxy_capture", os.path.join(os.path.dirname(__file__), "capture.py")
)
module = importlib.util.module_from_spec(spec)
with mock.patch.dict(sys.modules, stand_ins):
spec.loader.exec_module(module)
return module
capture = load_capture()
def location(value, scheme=None):
state = {"response": {"headers": [[b"Location", value]]}}
if scheme is not None:
state["request"] = {"scheme": scheme}
record = capture.project_record(state, [])
sidecar = next(
field for field in record["binary_fields"] if field["pointer"] == "/response/headers/0/1"
)
return record["reported"]["response"]["headers"][0][1], sidecar, record["redactions"]
class TransportUserinfoTests(unittest.TestCase):
def test_ignorable_controls_cannot_hide_authority_credentials(self):
for control in [b"\t", b"\n", b"\r"]:
for value in [b"https://" + control + b"/user:pass@host/path", b"ht" + control + b"tps://user:" + control + b"pass@host/path"]:
with self.subTest(value=value):
text, sidecar, redactions = location(value, b"https")
self.assertNotIn("user", text)
self.assertNotIn("pass", text)
self.assertEqual(sidecar["state"], "redacted")
self.assertEqual(len(redactions), 1)
def test_mixed_network_separators_remove_credentials(self):
for prefix in [b"\\\\", b"/\\", b"\\/"]:
with self.subTest(prefix=prefix):
text, sidecar, redactions = location(prefix + b"user:pass@host/path", b"https")
self.assertEqual(text, prefix.decode("ascii") + "host/path")
self.assertEqual(sidecar["state"], "redacted")
self.assertEqual(len(redactions), 1)
def test_non_special_base_keeps_backslashes_inside_network_authorities(self):
text, sidecar, redactions = location(b"//user\\name:pass@host/x", b"foo")
self.assertEqual(text, "//host/x")
self.assertEqual(sidecar["state"], "redacted")
self.assertEqual(len(redactions), 1)
text, sidecar, redactions = location(b"/\\invoice@customer", b"foo")
self.assertEqual(text, "/\\invoice@customer")
self.assertEqual(sidecar["state"], "retained")
self.assertEqual(redactions, [])
def test_native_request_paths_preserve_origin_form_and_redact_absolute_form(self):
for value in [b"/\\invoice@customer", b"//invoice@customer", b"\\\\invoice@customer"]:
with self.subTest(value=value):
record = capture.project_record({"request": {"path": value}}, [])
self.assertEqual(record["reported"]["request"]["path"], value.decode("ascii"))
self.assertEqual(record["binary_fields"][0]["state"], "retained")
self.assertEqual(record["redactions"], [])
record = capture.project_record({"request": {"path": b"https://user:pass@host/x"}}, [])
self.assertEqual(record["reported"]["request"]["path"], "https://host/x")
self.assertEqual(record["binary_fields"][0]["state"], "redacted")
def test_same_scheme_locations_preserve_relative_path_text_and_bytes(self):
for value in [b"https:invoice@customer", b"https:/invoice@customer", b"https:\\invoice@customer"]:
with self.subTest(value=value):
text, sidecar, redactions = location(value, b"https")
self.assertEqual(text, value.decode("ascii"))
self.assertEqual(sidecar["state"], "retained")
self.assertEqual(redactions, [])
def test_location_authorities_still_remove_credentials_with_a_base_scheme(self):
for value, scheme, expected in [
(b"https://user:pass@host/path", b"https", "https://host/path"),
(b"https:\\\\user:pass@host/path", b"https", "https:\\\\host/path"),
(b"https:user:pass@host/path", b"http", "https:host/path"),
]:
with self.subTest(value=value, scheme=scheme):
text, sidecar, redactions = location(value, scheme)
self.assertEqual(text, expected)
self.assertEqual(sidecar["state"], "redacted")
self.assertEqual(len(redactions), 1)
def test_backup_location_uses_its_own_request_scheme(self):
record = capture.project_record({
"request": {"scheme": b"http"},
"backup": {
"request": {"scheme": b"https"},
"response": {"headers": [[b"Location", b"https:invoice@customer"]]},
},
}, [])
self.assertEqual(record["reported"]["backup"]["response"]["headers"][0][1], "https:invoice@customer")
self.assertEqual(record["redactions"], [])
def test_backslash_ends_a_special_scheme_authority(self):
text, sidecar, redactions = location(b"https://example.test\\part@tail")
self.assertEqual(text, "https://example.test\\part@tail")
self.assertEqual(sidecar["state"], "retained")
self.assertIsNotNone(sidecar["content_base64"])
self.assertEqual(redactions, [])
def test_userinfo_before_a_backslash_is_still_removed(self):
text, sidecar, redactions = location(b"https://user:pw@example.test\\path")
self.assertEqual(text, "https://example.test\\path")
self.assertEqual(sidecar["state"], "redacted")
self.assertEqual(
redactions,
[{"pointer": "/response/headers/0/1", "reason": "transport-credential"}],
)
def test_ordinary_userinfo_is_removed(self):
text, _sidecar, _redactions = location(b"https://user:pw@example.test/a@b")
self.assertEqual(text, "https://example.test/a@b")
def test_empty_userinfo_marker_is_removed(self):
self.assertEqual(
capture.without_userinfo("https://@host"), "https://host"
)
def test_special_schemes_skip_leading_slashes_before_userinfo(self):
for value, expected in [
(b"https://\\user:pass@host/path", b"https://\\host/path"),
(b"https:\\\\user:pass@host/path", b"https:\\\\host/path"),
(b"https:user:pass@host", b"https:host"),
(b"//\\user:pass@host", b"//\\host"),
]:
with self.subTest(value=value):
self.assertEqual(capture.without_userinfo(value), expected)
text, sidecar, redactions = location(b"https://\\user:pass@host/path")
self.assertEqual(text, "https://\\host/path")
self.assertEqual(sidecar["state"], "redacted")
self.assertEqual(len(redactions), 1)
def test_file_base_keeps_paths_after_the_fixed_network_boundary(self):
for value in [b"///u@h", b"////u@h"]:
with self.subTest(value=value):
text, sidecar, redactions = location(value, b"file")
self.assertEqual(text, value.decode("ascii"))
self.assertEqual(sidecar["state"], "retained")
self.assertIsNotNone(sidecar["content_base64"])
self.assertEqual(redactions, [])
def test_file_urls_keep_path_text(self):
self.assertEqual(capture.without_userinfo("file:///a@b"), "file:///a@b")
def test_file_urls_reject_userinfo_and_keep_their_source(self):
# WHATWG rejects an "@" in a file host, so nothing parses as credentials.
self.assertEqual(
capture.without_userinfo("file://user:pass@host/x"), "file://user:pass@host/x"
)
for value in [b"\\\\user:pass@host/x", b"/\\user:pass@host/x", b"//user:pass@host/x"]:
with self.subTest(value=value):
text, sidecar, redactions = location(value, b"file")
self.assertEqual(text, value.decode("ascii"))
self.assertEqual(sidecar["state"], "retained")
self.assertEqual(redactions, [])
def test_rejected_special_hosts_keep_their_source(self):
# A rejected host or port fails the whole URL, so it has no userinfo.
for value in [
"https:user@%",
"https://user@",
"https://user:pass@h^st/x",
"https://user@ho st/x",
"https://user@host:99999/x",
"https://user@host:12a",
"https://user@[::1",
]:
with self.subTest(value=value):
self.assertEqual(capture.without_userinfo(value), value)
text, sidecar, redactions = location(b"https://user:pass@h^st/x", b"https")
self.assertEqual(text, "https://user:pass@h^st/x")
self.assertEqual(sidecar["state"], "retained")
self.assertEqual(redactions, [])
def test_accepted_special_hosts_still_remove_userinfo(self):
for value, expected in [
("https:user@%41", "https:%41"),
("https://user@[::1]:8443/x", "https://[::1]:8443/x"),
("https://user@host:/x", "https://host:/x"),
("wss://user:pass@host:65535", "wss://host:65535"),
]:
with self.subTest(value=value):
self.assertEqual(capture.without_userinfo(value), expected)
def test_non_special_schemes_keep_backslashes_in_the_authority(self):
self.assertEqual(capture.without_userinfo("foo://a\\b@host/x"), "foo://host/x")
self.assertEqual(capture.without_userinfo(b"//a\\b@host"), b"//a\\b@host")
if __name__ == "__main__":
unittest.main()
+15
View File
@@ -265,6 +265,21 @@ The user-level MCP paths follow the editor's VS Code layout. Trae's official
documentation describes the raw JSON editor without specifying those paths;
loading this registration in Trae remains unverified.
## WorkBuddy
For [WorkBuddy](https://www.workbuddy.cn/docs/workbuddy/From-Beginner-to-Expert-Guide/Function-Description/MCP-Guide), setup writes `mcpServers.rea` into the
global `~/.workbuddy/mcp.json` (a standard `mcpServers` JSON file at the user
home root on every platform). Entries carry `command`, `args`, and `env`;
stdio is the default transport. WorkBuddy discovers personal skills from
`~/.workbuddy/skills` (`%userprofile%\.workbuddy\skills` on Windows), so
setup installs its skill copy there.
The global MCP path is a plain JSON file; setup preserves every other server
and top-level key when it upserts `rea`. WorkBuddy also reads a project-level
`<project>/.workbuddy/mcp.json`, which setup does not touch: it registers the
user-level file only. Loading this registration and real REA tool calls
through WorkBuddy are verified.
## ZCode
For [ZCode](https://zcode.z.ai/en/docs/mcp-services), setup registers `rea` in the nested `mcp.servers` table of the
+5 -7
View File
@@ -288,11 +288,12 @@ hosted connector; see [its configuration](client-configuration.md#other-client-s
| Qoder | `qoder` |
| Factory Droid | `droid` |
| Trae | `trae` |
| WorkBuddy | `workbuddy` |
| ZCode | `zcode` |
| DeepSeek Harness | `dsh` |
`dsh` and `kimi` require a release containing them. The registry check above does
not cover them.
`dsh`, `kimi`, and `workbuddy` require a release containing them. The registry
check above does not cover them.
See [client configuration and troubleshooting](client-configuration.md) for
configuration paths, profiles, trust policies, and schema-limit workarounds.
@@ -565,11 +566,8 @@ Java major range, 64-bit JDK bitness, and the presence of `javac`/`javac.exe`.
It also checks that the reported JVM `os.arch` and `os.name` map to the same native decompiler
directory validated for the host; unrecognized reported architectures or a
non-host operating system are refused. LaunchSupport overrides of either
property, including quoted helper-environment JVM options, are rejected at
launch. Indirect JVM `@argument-file` options are also refused because their
effective platform properties cannot be established before launch; provide
explicit options instead. OS-name matching follows Ghidra's operating-system
identifiers: macOS requires `Mac OS X`, not `Darwin` or a generic `mac*` alias.
property, including through helper-environment JVM option variables, are
rejected at launch.
When Java is found through `PATH`, setup records its observed JDK home so GUI
MCP clients do not depend on an incidental shell path. Setup shows every exact
environment entry in its plan, writes only after approval, and never downloads,
+1 -1
View File
@@ -138,7 +138,7 @@
"test": "npm run test:with-build",
"test:with-build": "npm run test:prepare && vitest run",
"test:run": "vitest run",
"test:python": "python3 -m unittest bridge/native/test_rea_lldb_tracer.py -v && python3 -m unittest discover -s experiments/analysis-efficiency -p 'test_*.py' -v",
"test:python": "python3 -m unittest bridge/native/test_rea_lldb_tracer.py bridge/mitmproxy/test_capture.py -v && python3 -m unittest discover -s experiments/analysis-efficiency -p 'test_*.py' -v",
"test:local": "node scripts/run-development-tests.mjs local",
"test:fast": "node scripts/ci/test-group.mjs source",
"test:built": "node scripts/ci/test-group.mjs built",
@@ -1,4 +1,5 @@
export function assertCleanLinuxHopperRuntime(
launcherPaths?: readonly string[],
procRoot?: string,
listProcesses?: () => Promise<{ stdout: string }>,
): Promise<void>;
+2 -1
View File
@@ -17,8 +17,9 @@ export const assertCleanLinuxHopperRuntime = async (
: [process.env.HOPPER_LAUNCHER_PATH]),
],
procRoot = "/proc",
listProcesses = () => execFileAsync("ps", ["-ax", "-o", "command="]),
) => {
const { stdout } = await execFileAsync("ps", ["-ax", "-o", "command="]);
const { stdout } = await listProcesses();
if (/\/opt\/hopper\/bin\/Hopper|\bXvfb\b|hopper-demo-x11\.py/u.test(stdout))
throw new Error("Clean Linux runner retained a Hopper demo process");
const launchers = [];
+82
View File
@@ -215,6 +215,27 @@ async function verifyNativeValueAdapters(
const functionEvidence = parseEvidence(analyzed.structuredContent);
assert.equal(functionEvidence.operation, "analyze_function");
assert.equal(functionEvidence.provider.id, "ghidra");
assert.deepEqual(
functionEvidence.normalized_result.facet_availability,
Object.fromEntries(
[
"pseudocode",
"assembly",
"comments",
"callers",
"callees",
"references",
"strings_names",
"cfg",
].map((facet) => [facet, { available: true }]),
),
);
assert.equal(functionEvidence.relationship_coverage?.status, "complete");
assert.ok(
functionEvidence.relationship_coverage.diagnostics.some((diagnostic) =>
diagnostic.includes("indirect calls remain unknown"),
),
);
const functionProfile = analysisProfileSchema.parse(
functionEvidence.analysis_profile,
);
@@ -247,6 +268,7 @@ async function verifyNativeValueAdapters(
assert.ok(
comparisonEvidence.evidence_links.includes(functionEvidence.evidence_id),
);
await verifyUnresolvedCallEvidence(client);
const bundle = await client.callTool({
name: "get_evidence_bundle",
arguments: {},
@@ -307,3 +329,63 @@ async function verifyNativeValueAdapters(
unknowns: cliGraph.unknowns.length,
};
}
async function verifyUnresolvedCallEvidence(client) {
const inventory = await client.callTool({
name: "list_procedures",
arguments: {},
});
assert.notEqual(inventory.isError, true, JSON.stringify(inventory));
const procedures = parseEvidence(
inventory.structuredContent,
).normalized_result;
const indirect = procedures.find((item) =>
item.value.endsWith("rea_ghidra_inventory_indirect"),
);
const leaf = procedures.find((item) =>
item.value.endsWith("rea_ghidra_inventory_leaf"),
);
assert.ok(
indirect && leaf,
"Source fixture lacks the unresolved callback and its possible target",
);
const analyzed = await client.callTool(
{ name: "analyze_function", arguments: { procedure: indirect.address } },
{ timeout: 180000 },
);
assert.notEqual(analyzed.isError, true, JSON.stringify(analyzed));
const evidence = parseEvidence(analyzed.structuredContent);
assert.equal(evidence.relationship_coverage?.status, "partial");
assert.ok(evidence.normalized_result.unresolved_calls.length > 0);
const traced = await client.callTool(
{
name: "trace_call_path",
arguments: {
start: indirect.address,
goal: leaf.address,
direction: "forward",
},
},
{ timeout: 180000 },
);
assert.notEqual(traced.isError, true, JSON.stringify(traced));
const trace = parseEvidence(traced.structuredContent).normalized_result;
assert.equal(trace.goal_status, "unknown");
assert.equal(trace.truncated, true);
assert.ok(trace.residual_unknowns.length > 0);
const compared = await client.callTool(
{
name: "compare_functions",
arguments: {
left: analyzed.structuredContent,
right: analyzed.structuredContent,
},
},
{ timeout: 180000 },
);
assert.notEqual(compared.isError, true, JSON.stringify(compared));
assert.equal(
parseEvidence(compared.structuredContent).normalized_result.status,
"unknown",
);
}
+1 -1
View File
@@ -136,7 +136,7 @@ export async function verifyGhidraBoundaries(
assert.deepEqual(externalDossier.callees, []);
assert.deepEqual(externalDossier.facet_availability.pseudocode, {
available: false,
reason: "Ghidra did not supply pseudocode.",
reason: "Ghidra does not decompile external functions.",
});
for (const facet of [
"assembly",
+8
View File
@@ -606,6 +606,14 @@ export const SUPPORTED_CLIENT_DEFINITIONS = [
markerPath: traeUserDirectory,
format: "json",
},
{
name: "workbuddy",
displayName: "WorkBuddy",
skillPath: [".workbuddy", "skills"],
configPath: [".workbuddy", "mcp.json"],
markerPath: [".workbuddy"],
format: "json",
},
{
name: "grok_build",
displayName: "Grok Build",
+36
View File
@@ -337,6 +337,42 @@ describe("call path topology and capacity", () => {
});
});
describe("directional call relationship coverage", () => {
it.each([
{
directions: ["callers"] as ["callers"],
status: "not_found",
exhaustive: true,
},
{
directions: ["callees"] as ["callees"],
status: "unknown",
exhaustive: false,
},
{
directions: ["callers", "callees"] as ["callers", "callees"],
status: "unknown",
exhaustive: false,
},
])(
"uses $directions coverage for the forward search",
({ directions, status, exhaustive }) => {
const limitation = "The inventory omitted a reference.";
const relationshipCoverage: RelationshipCoverage = {
status: "partial",
limitations: [limitation],
limited_directions: directions,
};
const result = run([
observe("0x1000", ["0x2000"], "rea-workflow", { relationshipCoverage }),
observe("0x2000", [], "rea-workflow", { relationshipCoverage }),
]);
expect(result).toMatchObject({ status, search_scope: { exhaustive } });
expect(result.limitations).toContain(limitation);
},
);
});
describe("explicit call relationship coverage", () => {
it("accumulates 150,000 partial reasons without argument-stack exhaustion", () => {
const result = run([
+2 -2
View File
@@ -10,7 +10,7 @@ import {
parseFunctionEvidence,
type FunctionSnapshot,
} from "./functionDossierEvidence.js";
import { isCallerOnlyUnknownCoverage } from "./functionCallCoverage.js";
import { isCallerOnlyIncompleteCoverage } from "./functionCallCoverage.js";
import { compareUnicodeCodePoints } from "./unicodeCodePointOrder.js";
/** Minimal directed caller-to-callee adjacency for call-path search. */
@@ -268,7 +268,7 @@ const inspectSearch = ({
coverage === null
? snapshot.limitations.length > 0
: coverage.status !== "complete" &&
!isCallerOnlyUnknownCoverage(coverage)
!isCallerOnlyIncompleteCoverage(coverage)
)
incompleteExpandedCoverage = true;
} else if (node === goal) {
+33 -7
View File
@@ -2,7 +2,7 @@ import { describe, expect, it } from "vitest";
import {
combineFunctionRelationshipCoverage,
isCallerOnlyUnknownCoverage,
isCallerOnlyIncompleteCoverage,
} from "./functionCallCoverage.js";
import type { RelationshipCoverage } from "./relationshipCoverage.js";
@@ -22,14 +22,14 @@ const cappedCallees: RelationshipCoverage = {
};
const complete: RelationshipCoverage = { status: "complete" };
describe("isCallerOnlyUnknownCoverage", () => {
describe("isCallerOnlyIncompleteCoverage", () => {
it("detects unknown callers beside complete callees", () => {
const combined = combineFunctionRelationshipCoverage(
unknownCallers,
complete,
);
expect(combined.status).toBe("unknown");
expect(isCallerOnlyUnknownCoverage(combined)).toBe(true);
expect(isCallerOnlyIncompleteCoverage(combined)).toBe(true);
});
it("rejects unknown callees beside complete callers", () => {
@@ -37,7 +37,7 @@ describe("isCallerOnlyUnknownCoverage", () => {
complete,
unknownCallees,
);
expect(isCallerOnlyUnknownCoverage(combined)).toBe(false);
expect(isCallerOnlyIncompleteCoverage(combined)).toBe(false);
});
it("rejects unknown coverage on both sides", () => {
@@ -45,7 +45,7 @@ describe("isCallerOnlyUnknownCoverage", () => {
unknownCallers,
unknownCallees,
);
expect(isCallerOnlyUnknownCoverage(combined)).toBe(false);
expect(isCallerOnlyIncompleteCoverage(combined)).toBe(false);
});
it("rejects unknown callers beside IDA-capped callees", () => {
@@ -54,11 +54,37 @@ describe("isCallerOnlyUnknownCoverage", () => {
cappedCallees,
);
expect(combined.status).toBe("unknown");
expect(isCallerOnlyUnknownCoverage(combined)).toBe(false);
expect(isCallerOnlyIncompleteCoverage(combined)).toBe(false);
});
it("rejects complete coverage", () => {
const combined = combineFunctionRelationshipCoverage(complete, complete);
expect(isCallerOnlyUnknownCoverage(combined)).toBe(false);
expect(isCallerOnlyIncompleteCoverage(combined)).toBe(false);
});
it("retains caller-only partial coverage without relying on limitation wording", () => {
const combined = combineFunctionRelationshipCoverage(
{ status: "partial", limitations: ["Missing incoming references."] },
complete,
);
expect(combined).toMatchObject({ limited_directions: ["callers"] });
expect(isCallerOnlyIncompleteCoverage(combined)).toBe(true);
});
it("uses explicit directions even when explanation text contains direction labels", () => {
expect(
isCallerOnlyIncompleteCoverage({
status: "unknown",
reason: "Callers: unavailable. Callees: diagnostic text.",
limited_directions: ["callers"],
}),
).toBe(true);
expect(
isCallerOnlyIncompleteCoverage({
status: "unknown",
reason: "Callers: unavailable.",
limited_directions: ["callees"],
}),
).toBe(false);
});
});
+27 -10
View File
@@ -22,6 +22,16 @@ export const combineFunctionRelationshipCoverage = (
),
);
const details = diagnostics.length > 0 ? { diagnostics } : {};
const callersLimited = callers?.status !== "complete";
const calleesLimited = callees?.status !== "complete";
const limited_directions: Extract<
RelationshipCoverage,
{ status: "partial" }
>["limited_directions"] = callersLimited
? calleesLimited
? ["callers", "callees"]
: ["callers"]
: ["callees"];
const reasons = reads.flatMap(({ direction, coverage }) => {
const label =
direction === "callers" ? callerCoveragePrefix : calleeCoveragePrefix;
@@ -43,30 +53,37 @@ export const combineFunctionRelationshipCoverage = (
return {
status: "unknown",
reason: uniqueSorted(reasons).join(" "),
limited_directions,
...details,
};
if (reasons.length > 0)
return {
status: "partial",
limitations: uniqueSorted(reasons),
limited_directions,
...details,
};
return { status: "complete", ...details };
};
/**
* Whether combined relationship coverage is unknown only on the caller side.
* The combine helper above prefixes each direction's reasons, so a reason
* that carries the caller prefix and no callee prefix records observed
* callees with unobservable callers. Forward (callee-edge) searches stay
* exhaustive under that caveat; the caller reason remains a limitation.
* Forward searches depend on callee coverage. Preserve the historical
* unknown-reason encoding when reading observations without explicit directions.
*/
export const isCallerOnlyUnknownCoverage = (
export const isCallerOnlyIncompleteCoverage = (
coverage: RelationshipCoverage,
): boolean =>
coverage.status === "unknown" &&
coverage.reason.startsWith(callerCoveragePrefix) &&
!coverage.reason.includes(calleeCoveragePrefix);
): boolean => {
if (coverage.status === "complete") return false;
if (coverage.limited_directions !== undefined)
return !coverage.limited_directions.some(
(direction) => direction === "callees",
);
return (
coverage.status === "unknown" &&
coverage.reason.startsWith(callerCoveragePrefix) &&
!coverage.reason.includes(calleeCoveragePrefix)
);
};
/** Explain why two resolved call sets cannot establish call-set equality. */
export const functionCallCoverageLimitations = (
@@ -0,0 +1,40 @@
import { describe, expect, it } from "vitest";
import { analyzeJavaScriptStaticSource } from "./javascriptStaticAnalysis.js";
const endpoints = (source: string) =>
analyzeJavaScriptStaticSource(source).endpoints.map(
({ kind, value, mechanism }) => ({ kind, value, mechanism }),
);
describe("server-sent event endpoints", () => {
it("reports EventSource URLs like WebSocket URLs", () => {
expect(
endpoints(`
new WebSocket("wss://chat.example/ws");
new EventSource("/api/chat/stream");
new window.EventSource("https://chat.example/events", { withCredentials: true });
`),
).toEqual([
{
kind: "network",
value: "wss://chat.example/ws",
mechanism: "call:WebSocket",
},
{
kind: "network",
value: "/api/chat/stream",
mechanism: "call:EventSource",
},
{
kind: "network",
value: "https://chat.example/events",
mechanism: "call:window.EventSource",
},
]);
});
it("keeps a dynamic EventSource URL unreported", () => {
expect(endpoints("new EventSource(url);")).toEqual([]);
});
});
@@ -0,0 +1,34 @@
import { describe, expect, it } from "vitest";
import { analyzeJavaScriptStaticSource } from "./javascriptStaticAnalysis.js";
const endpoints = (source: string) =>
analyzeJavaScriptStaticSource(source).endpoints.map(
({ kind, value, mechanism }) => ({ kind, value, mechanism }),
);
describe("beacon endpoints", () => {
it("reports navigator.sendBeacon URLs", () => {
expect(
endpoints(`
navigator.sendBeacon("/api/telemetry", payload);
window.navigator.sendBeacon("https://metrics.example/collect");
`),
).toEqual([
{
kind: "network",
value: "/api/telemetry",
mechanism: "call:navigator.sendBeacon",
},
{
kind: "network",
value: "https://metrics.example/collect",
mechanism: "call:window.navigator.sendBeacon",
},
]);
});
it("keeps a dynamic beacon URL unreported", () => {
expect(endpoints("navigator.sendBeacon(url, payload);")).toEqual([]);
});
});
@@ -554,7 +554,13 @@ export const endpointArgument = (
receiverFact?: JavaScriptOpenReceiverFact,
): string | undefined => {
if (receiverFact === "window") return undefined;
if (name === "fetch" || name.endsWith(".fetch") || name === "WebSocket")
if (
name === "fetch" ||
name.endsWith(".fetch") ||
name === "WebSocket" ||
name === "EventSource" ||
name.endsWith(".EventSource")
)
return stringValue(args[0]);
if (name.endsWith(".open"))
return xhrOpenUrl(name, args[0], args[1], receiverFact);
@@ -574,6 +580,9 @@ export const endpointArgument = (
? undefined
: value;
}
// navigator.sendBeacon(url, data) queues a POST to its first argument.
if (name === "sendBeacon" || name.endsWith(".sendBeacon"))
return stringValue(args[0]);
if (name.endsWith("loadURL")) return stringValue(args[0]);
return undefined;
};
+11
View File
@@ -0,0 +1,11 @@
import { expect, it } from "vitest";
import { parseRecordedTransportUrl } from "./recordedTransportUrl.js";
it("recognizes an empty userinfo marker at the authority start", () => {
expect(parseRecordedTransportUrl("https://@host")).toMatchObject({
origin: "https://host",
path: "/",
credentials: { start: 8, end: 9 },
});
});
+17 -6
View File
@@ -1,24 +1,35 @@
import { z } from "zod";
const diagnostics = z.array(z.string().min(1)).min(1);
const completeCoverage = z.strictObject({ status: z.literal("complete") });
const diagnostics = z.array(z.string().min(1)).min(1).exactOptional();
const completeCoverage = z.strictObject({
status: z.literal("complete"),
diagnostics,
});
const limitedDirections = z.union([
z.tuple([z.literal("callers")]).check(z.minLength(1), z.maxLength(1)),
z.tuple([z.literal("callees")]).check(z.minLength(1), z.maxLength(1)),
z
.tuple([z.literal("callers"), z.literal("callees")])
.check(z.minLength(2), z.maxLength(2)),
]);
const partialCoverage = z.strictObject({
status: z.literal("partial"),
limitations: z.array(z.string().min(1)).min(1),
limited_directions: limitedDirections.exactOptional(),
diagnostics,
});
const unknownCoverage = z.strictObject({
status: z.literal("unknown"),
reason: z.string().min(1),
limited_directions: limitedDirections.exactOptional(),
diagnostics,
});
/** Explicit coverage state for a provider's observed caller/callee set. */
export const relationshipCoverageSchema = z.union([
export const relationshipCoverageSchema = z.discriminatedUnion("status", [
completeCoverage,
completeCoverage.extend({ diagnostics }),
partialCoverage,
partialCoverage.extend({ diagnostics }),
unknownCoverage,
unknownCoverage.extend({ diagnostics }),
]);
export type RelationshipCoverage = z.infer<typeof relationshipCoverageSchema>;
+4 -2
View File
@@ -40,6 +40,8 @@ import {
NATIVE_AOT_PROFILE_CONTRACT_REVISION,
} from "../domain/native/nativeAotPe.js";
const callRelationshipCoverage = "function-body-call-coverage-v5";
/**
* The processor language REA pins for this target, or undefined when Ghidra may
* select from the header. The committed profile and the session readback both
@@ -125,7 +127,7 @@ export const resolveGhidraAnalysisProfile = async (
load_image_observations: "source-mappings-entry-context-v2",
function_body_evidence: "complete-inclusive-ranges-v1",
function_references: "complete-body-and-entry-reference-manager-v2",
call_relationship_coverage: "resolved-call-reference-diagnostics-v1",
call_relationship_coverage: callRelationshipCoverage,
function_dossier_coverage:
"observed-facets-and-call-relationships-v1",
string_inventory_evidence: "defined-data-coverage-v1",
@@ -222,7 +224,7 @@ export const resolveGhidraAnalysisProfile = async (
load_image_observations: "source-mappings-entry-context-v2",
function_body_evidence: "complete-inclusive-ranges-v1",
function_references: "complete-body-and-entry-reference-manager-v2",
call_relationship_coverage: "resolved-call-reference-diagnostics-v1",
call_relationship_coverage: callRelationshipCoverage,
function_dossier_coverage: "observed-facets-and-call-relationships-v1",
string_inventory_evidence: "defined-data-coverage-v1",
location_resolution: "explicit-address-exact-entry-symbol-first-v3",
+113
View File
@@ -43,7 +43,120 @@ const parseFunctionValueFixture = (
return parsed.ok ? { ok: true as const, value: parsed.value.value } : parsed;
};
describe("Ghidra unresolved call locations", () => {
it("rejects a noncanonical unresolved call site rather than retaining unverifiable location evidence", () => {
expect(
parseGhidraFunctionResult("analyze_function", {
value: {
...functionDossierSchema.parse(ghidraFunctionDossier()),
unresolved_calls: [
{ address: "not-an-address", reason: "No call destination." },
],
},
limitations: [],
relationship_coverage: {
status: "partial",
limitations: ["An unresolved call remains."],
},
}),
).toMatchObject({ ok: false, error: { _tag: "AnalysisOutputError" } });
});
});
describe("Ghidra function-analysis result values", () => {
it("retains unavailable pseudocode instead of treating its empty representation as observed source", () => {
const availability = {
pseudocode: {
available: false,
reason: "Ghidra does not decompile external functions.",
},
comments: { available: true },
};
const parsed = parseGhidraFunctionResult("analyze_function", {
value: {
...functionDossierSchema.parse(ghidraFunctionDossier()),
pseudocode: "",
facet_availability: availability,
},
limitations: [],
});
expect(parsed).toMatchObject({
ok: true,
value: { value: { pseudocode: "", facet_availability: availability } },
});
});
it.each([
{ status: "complete", diagnostics: ["Resolved static references only."] },
{
status: "partial",
limitations: ["Only the first instruction range was examined."],
},
{ status: "unknown", reason: "Relationship inventory was unavailable." },
{
status: "partial",
limitations: ["Missing callers."],
limited_directions: ["callers"],
},
{
status: "unknown",
reason: "Missing callees.",
limited_directions: ["callees"],
},
])("preserves observed $status relationship coverage", (coverage) => {
const parsed = parseGhidraFunctionResult("analyze_function", {
value: ghidraFunctionDossier(),
limitations: [],
relationship_coverage: coverage,
});
expect(parsed).toMatchObject({
ok: true,
value: { relationshipCoverage: coverage },
});
});
it("preserves omitted relationship coverage for legacy observations", () => {
const parsed = parseGhidraFunctionResult("analyze_function", {
value: ghidraFunctionDossier(),
limitations: [],
});
if (!parsed.ok) throw parsed.error;
expect(parsed.value).not.toHaveProperty("relationshipCoverage");
});
});
describe("Ghidra relationship metadata validation", () => {
it.each([
{ status: "partial", limitations: [] },
{ status: "unknown", reason: "" },
{ status: "complete", limitations: ["Truncated."] },
{ status: "partial", limitations: ["Truncated."], limited_directions: [] },
{
status: "partial",
limitations: ["Truncated."],
limited_directions: ["callers", "callers"],
},
{ status: "unknown", reason: "Missing.", limited_directions: ["incoming"] },
{ status: "complete", limited_directions: ["callers"] },
])("rejects malformed relationship coverage %j", (coverage) => {
expect(
parseGhidraFunctionResult("analyze_function", {
value: ghidraFunctionDossier(),
limitations: [],
relationship_coverage: coverage,
}),
).toMatchObject({ ok: false, error: { _tag: "AnalysisOutputError" } });
});
it("rejects relationship metadata on unrelated function operations", () => {
expect(
parseGhidraFunctionResult("procedure_assembly", {
value: "ret",
limitations: [],
relationship_coverage: { status: "complete" },
}),
).toMatchObject({ ok: false, error: { _tag: "AnalysisOutputError" } });
});
it("preserves file-offset resolution in Evidence locations and rejects inexact offsets", () => {
const selection = {
file_offset: 0x210,
+16 -1
View File
@@ -27,6 +27,10 @@ import {
import { nativeApiBoundarySchema } from "../domain/native/nativeApiBoundary.js";
import { nativeValueFlowSchema } from "../domain/native/nativeValueFlow.js";
import type { EvidenceLocation } from "../domain/evidence.js";
import {
relationshipCoverageSchema,
type RelationshipCoverage,
} from "../domain/relationshipCoverage.js";
import { err, ok, type Result } from "../domain/result.js";
import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js";
import {
@@ -307,6 +311,7 @@ const ghidraFunctionDossier = functionDossierSchema
...value.comments.map(({ address }) => address),
...value.callers.map(({ address }) => address),
...value.callees.map(({ address }) => address),
...(value.unresolved_calls ?? []).map(({ address }) => address),
...value.incoming_references.flatMap(referenceAddresses),
...value.outgoing_references.flatMap(referenceAddresses),
...value.referenced_strings.flatMap(({ address, source_address }) => [
@@ -419,6 +424,7 @@ const functionBoundaryObservationOperations: ReadonlySet<GhidraFunctionOperation
const functionObservationEnvelope = z.strictObject({
value: z.unknown(),
limitations: z.array(z.string()),
relationship_coverage: relationshipCoverageSchema.exactOptional(),
selection: z
.strictObject({
file_offset: z.number().int().min(0).max(Number.MAX_SAFE_INTEGER),
@@ -432,6 +438,7 @@ export interface ParsedGhidraFunctionResult {
readonly value: JsonValue;
readonly limitations: readonly string[];
readonly locations?: readonly EvidenceLocation[];
readonly relationshipCoverage?: RelationshipCoverage;
}
/** Require exact, bounded Java-bridge function output before creating Evidence. */
@@ -442,6 +449,7 @@ export const parseGhidraFunctionResult = (
let observation: {
readonly value: unknown;
readonly limitations: readonly string[];
readonly relationship_coverage?: RelationshipCoverage;
readonly selection?: z.infer<
typeof functionObservationEnvelope
>["selection"];
@@ -450,13 +458,17 @@ export const parseGhidraFunctionResult = (
const envelope = functionObservationEnvelope.safeParse(value);
if (
!envelope.success ||
(operation !== "analyze_function" &&
operation !== "procedure_callers" &&
operation !== "procedure_callees" &&
envelope.data.relationship_coverage !== undefined) ||
(operation !== "procedure_pseudo_code" &&
envelope.data.selection !== undefined)
)
return err(
new AnalysisOutputError(
operation,
"Ghidra bridge omitted the function observation envelope",
"Ghidra bridge returned an invalid function observation envelope for this operation",
envelope.success ? undefined : { cause: envelope.error },
),
);
@@ -469,6 +481,9 @@ export const parseGhidraFunctionResult = (
? ok({
value: jsonValueSchema.parse(parsed.data),
limitations: observation.limitations,
...(observation.relationship_coverage === undefined
? {}
: { relationshipCoverage: observation.relationship_coverage }),
...(observation.selection === undefined
? {}
: {
-30
View File
@@ -1041,36 +1041,6 @@ describe("Ghidra JVM native directory admission", () => {
});
describe("Ghidra JVM OS admission", () => {
it.each([
["Darwin", false],
["macOS", false],
["Macintosh", false],
["Mac OS X", true],
["MAC OS X 26", true],
["JVM on Mac OS X", true],
["Mac OS X / Windows", false],
["Mac OS X / Linux", false],
] as const)(
"matches Ghidra's ordered OS-name predicate for %s (available=%s)",
(osName, available) => {
const result = inspectGhidraInstallation(
{
environment: {},
installDir: INSTALL,
platform: "darwin",
architecture: "arm64",
},
host({ probeJava: () => ({ ...JAVA, osArch: "aarch64", osName }) }),
);
expect(result.status).toBe(available ? "available" : "unavailable");
expect(projectGhidraDoctorInspection(result).available).toBe(available);
expect(result.javaOsName).toBe(osName);
expect(
result.checks.find(({ name }) => name === "native_decompiler"),
).toMatchObject({ status: available ? "passed" : "failed" });
},
);
it("rejects a successful Java probe that omits os.name", () => {
const { osName: _observedOsName, ...withoutOsName } = JAVA;
const result = inspectGhidraInstallation(
+5 -11
View File
@@ -741,17 +741,11 @@ const jvmOsNameMatchesPlatform = (
osName: string,
): boolean => {
const normalized = osName.toLowerCase();
// Ghidra 12.1.4 OperatingSystem.findCurrentOperatingSystem uses ordered,
// case-insensitive substring matching, not Node's platform names or broad
// mac*/Darwin aliases. Preserve that order for names containing several tags.
const selected = normalized.includes("windows")
? "win32"
: normalized.includes("linux")
? "linux"
: normalized.includes("mac os x")
? "darwin"
: undefined;
return selected === platform;
if (platform === "win32") return normalized.startsWith("windows");
if (platform === "darwin")
return normalized.startsWith("mac") || normalized.startsWith("darwin");
if (platform === "linux") return normalized.startsWith("linux");
return false;
};
/**
+1 -9
View File
@@ -26,13 +26,6 @@ const jvmEnvironmentArguments = (value: string): readonly string[] => {
};
const rejectJvmPlatformOverrides = (arguments_: readonly string[]): void => {
// Indirect options can change the JVM platform after this admission check.
// Do not read arbitrary, mutable argument files and pretend their contents
// establish the later launch; require explicit inspectable options instead.
if (arguments_.some((argument) => argument.startsWith("@")))
throw new Error(
"Ghidra LaunchSupport may not supply JVM argument files; their effective platform properties cannot be validated before launch.",
);
if (
arguments_.some((argument) => /^-Dos\.(?:arch|name)(?:=|$)/u.test(argument))
)
@@ -120,8 +113,7 @@ export const ghidraJavaLaunch = async (options: {
// The merged environment becomes the analysis JVM's environment, and the
// JVM applies these options variables automatically. An injected
// -Dos.arch/-Dos.name there selects an unvalidated native directory without
// appearing in vmArguments, so reject the effective values as well. Argument
// files are refused because their expanded options are not established here.
// appearing in vmArguments, so reject the effective values as well.
for (const key of [
"JAVA_TOOL_OPTIONS",
"_JAVA_OPTIONS",
+11 -2
View File
@@ -633,7 +633,7 @@ describe("Ghidra client projection", () => {
expect(resolved.value.profile.parameters).toMatchObject({
language_id: "auto-from-header",
compiler_spec_id: "auto-default",
call_relationship_coverage: "resolved-call-reference-diagnostics-v1",
call_relationship_coverage: "function-body-call-coverage-v5",
function_dossier_coverage: "observed-facets-and-call-relationships-v1",
});
@@ -870,7 +870,16 @@ describe("Ghidra relationship coverage", () => {
Promise.resolve(
ok(
operation === "procedure_callees"
? { value: ["0x2000"], limitations: [] }
? {
value: ["0x2000"],
limitations: [],
relationship_coverage: {
status: "complete",
diagnostics: [
"Only resolved Ghidra call references are returned; unresolved computed or indirect calls remain unknown, while function classifications distinguish thunks and externals.",
],
},
}
: { value: [], limitations: [] },
),
),
-17
View File
@@ -2,7 +2,6 @@ import type {
CapabilityDescriptor,
ProviderIdentity,
} from "../application/AnalysisProvider.js";
import type { RelationshipCoverage } from "../domain/relationshipCoverage.js";
import {
hasWindowsNativeAuthority,
windowsNativeAuthorityUnavailableReason,
@@ -44,22 +43,6 @@ const stringInventoryLimitation =
const searchLimitation =
"Regex mode uses Java Pattern semantics. Matcher or compiler stack exhaustion returns a resource constraint without closing the session; literal mode avoids regex recursion and returns complete matching values.";
const resolvedCallReferenceLimitation =
"Only resolved Ghidra call references are returned; unresolved computed or indirect calls remain unknown, while function classifications distinguish thunks and externals.";
/** Relationship-only caveats belong to each observed caller/callee result. */
export const relationshipCoverageFor = (
operation: string,
): RelationshipCoverage | undefined =>
operation === "procedure_callers" ||
operation === "procedure_callees" ||
operation === "analyze_function"
? {
status: "complete",
diagnostics: [resolvedCallReferenceLimitation],
}
: undefined;
/** Limitation text for one admitted operation, including the common base. */
export const limitationsFor = (operation: string): readonly string[] => {
const common = [
+5 -2
View File
@@ -34,6 +34,7 @@ import {
} from "../domain/analysisErrorCore.js";
import type { AnalysisError } from "../domain/analysisErrorBase.js";
import type { EvidenceLocation } from "../domain/evidence.js";
import type { RelationshipCoverage } from "../domain/relationshipCoverage.js";
import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js";
import { ProviderAdapterError } from "../domain/providerAdapterError.js";
import { err, ok, type Result } from "../domain/result.js";
@@ -86,7 +87,6 @@ import {
healthLimitations,
windowsP0Limitations,
limitationsFor,
relationshipCoverageFor,
} from "./GhidraProviderCapabilities.js";
import type { GhidraSessionError } from "./GhidraSessionError.js";
import {
@@ -733,8 +733,8 @@ export const createGhidraProviderClient = (input: {
rawResult: JsonValue,
observationLimitations: readonly string[] = [],
locations: readonly EvidenceLocation[] = [],
relationshipCoverage?: RelationshipCoverage,
) => {
const relationshipCoverage = relationshipCoverageFor(operation);
return createAnalysisExecution(normalized, committedProfile.provider, {
rawResult,
locations,
@@ -838,12 +838,14 @@ export const createGhidraProviderClient = (input: {
let normalized: JsonValue;
let observationLimitations: readonly string[] = [];
let observationLocations: readonly EvidenceLocation[] = [];
let relationshipCoverage: RelationshipCoverage | undefined;
if (isGhidraFunctionOperation(operation)) {
const result = parseGhidraFunctionResult(operation, called.value);
if (!result.ok) return result;
normalized = result.value.value;
observationLimitations = result.value.limitations;
observationLocations = result.value.locations ?? [];
relationshipCoverage = result.value.relationshipCoverage;
} else {
const result = parseGhidraInventoryResult(operation, called.value);
if (!result.ok) return result;
@@ -939,6 +941,7 @@ export const createGhidraProviderClient = (input: {
called.value,
observationLimitations,
observationLocations,
relationshipCoverage,
),
);
},
+121 -37
View File
@@ -11,6 +11,7 @@ import { silentLogger } from "../logger.js";
import { GhidraProvider } from "./GhidraProvider.js";
import { GhidraSessionError } from "./GhidraSessionError.js";
import { GHIDRA_SESSION_CAPABILITIES } from "./GhidraSessionValues.js";
import type { RelationshipCoverage } from "../domain/relationshipCoverage.js";
const TARGET: BinaryTarget = {
path: "/tmp/ghidra-cache-fixture",
@@ -23,7 +24,7 @@ const TARGET: BinaryTarget = {
// Exercise the real provider descriptors, profile, input/output projection and
// session cache; only the installed Ghidra process is replaced at its seam.
const fixture = (version = "12.1.4") => {
const fixture = (version = "12.1.4", coverage?: RelationshipCoverage) => {
const config = parseConfig({ GHIDRA_INSTALL_DIR: "/opt/ghidra" });
if (!config.ok) throw config.error;
const calls: string[] = [];
@@ -113,7 +114,13 @@ const fixture = (version = "12.1.4") => {
return ok(
operation === "procedure_pseudo_code" ||
operation === "analyze_function"
? { value, limitations: [] }
? {
value,
limitations: [],
...(operation === "analyze_function" && coverage !== undefined
? { relationship_coverage: coverage }
: {}),
}
: value,
);
},
@@ -133,42 +140,119 @@ const open = async (session: ReturnType<typeof fixture>["session"]) => {
expect((await session.openResolvedTarget(preview.value)).ok).toBe(true);
};
describe("Ghidra snapshot cache", () => {
it("refuses snapshots whose profile predates typed incoming references", async () => {
const initial = fixture();
await open(initial.session);
const snapshot = initial.session.exportAnalysisSnapshot();
if (!snapshot.ok) throw snapshot.error;
await initial.session.close();
const profile = snapshot.value.binding.analysis_profile;
expect(profile.parameters.instruction_references).toBe(
"exact-address-incoming-v1",
);
const legacy = {
...snapshot.value,
binding: {
...snapshot.value.binding,
analysis_profile: createAnalysisProfile(
profile.provider,
Object.fromEntries(
Object.entries(profile.parameters).filter(
([key]) => key !== "instruction_references",
),
),
),
},
};
const replay = fixture();
const imported = replay.session.importAnalysisSnapshot(legacy);
if (!imported.ok) throw imported.error;
expect(await replay.session.previewTarget(TARGET)).toMatchObject({
ok: false,
error: { _tag: "EvidenceIntegrityError" },
});
expect(replay.calls).toEqual([]);
await replay.session.close();
});
describe("Ghidra relationship coverage replay", () => {
it.each([
undefined,
{ status: "complete", diagnostics: ["Resolved static references only."] },
{ status: "partial", limitations: ["Instruction coverage was truncated."] },
{ status: "unknown", reason: "Relationship inventory was unavailable." },
{
status: "partial",
limitations: ["Missing callers."],
limited_directions: ["callers"],
},
] satisfies (RelationshipCoverage | undefined)[])(
"preserves relationship coverage %j through live execution, cache and snapshot replay",
async (coverage) => {
const initial = fixture("12.1.4", coverage);
await open(initial.session);
const parameters = { procedure: "0x1000" };
for (let i = 0; i < 2; i++) {
const execution = await initial.session.execute(
"analyze_function",
parameters,
);
if (!execution.ok) throw execution.error;
expect(execution.value.relationshipCoverage).toEqual(coverage);
}
expect(initial.calls).toEqual(["analyze_function"]);
const snapshot = initial.session.exportAnalysisSnapshot();
if (!snapshot.ok) throw snapshot.error;
await initial.session.close();
const replay = fixture();
expect(replay.session.importAnalysisSnapshot(snapshot.value).ok).toBe(
true,
);
await open(replay.session);
const execution = await replay.session.execute(
"analyze_function",
parameters,
);
if (!execution.ok) throw execution.error;
expect(execution.value.relationshipCoverage ?? undefined).toEqual(
coverage,
);
expect(replay.calls).toEqual([]);
await replay.session.close();
},
);
});
describe("Ghidra snapshot profile admission", () => {
it.each([
{
parameter: "instruction_references",
current: "exact-address-incoming-v1",
legacy: undefined,
},
{
parameter: "call_relationship_coverage",
current: "function-body-call-coverage-v5",
legacy: "resolved-call-reference-diagnostics-v1",
},
{
parameter: "call_relationship_coverage",
current: "function-body-call-coverage-v5",
legacy: "decoded-call-target-coverage-v2",
},
{
parameter: "call_relationship_coverage",
current: "function-body-call-coverage-v5",
legacy: "decoded-call-target-coverage-v3",
},
{
parameter: "call_relationship_coverage",
current: "function-body-call-coverage-v5",
legacy: "function-body-call-coverage-v4",
},
])(
"refuses snapshots with outdated $parameter semantics",
async ({ parameter, current, legacy: prior }) => {
const initial = fixture();
await open(initial.session);
const snapshot = initial.session.exportAnalysisSnapshot();
if (!snapshot.ok) throw snapshot.error;
await initial.session.close();
const profile = snapshot.value.binding.analysis_profile;
expect(profile.parameters[parameter]).toBe(current);
const legacy = {
...snapshot.value,
binding: {
...snapshot.value.binding,
analysis_profile: createAnalysisProfile(profile.provider, {
...Object.fromEntries(
Object.entries(profile.parameters).filter(
([key]) => key !== parameter,
),
),
...(prior === undefined ? {} : { [parameter]: prior }),
}),
},
};
const replay = fixture();
const imported = replay.session.importAnalysisSnapshot(legacy);
if (!imported.ok) throw imported.error;
expect(await replay.session.previewTarget(TARGET)).toMatchObject({
ok: false,
error: { _tag: "EvidenceIntegrityError" },
});
expect(replay.calls).toEqual([]);
await replay.session.close();
},
);
});
describe("Ghidra snapshot cache", () => {
it.each([
{ operation: "analyze_function", parameters: { procedure: "0x1000" } },
{
@@ -67,6 +67,7 @@ describe("client configuration filesystem lifecycle", () => {
"kimi",
"vscode",
"trae",
"workbuddy",
"grok_build",
"omp",
"pi",
@@ -0,0 +1,159 @@
import { mkdir, readFile, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { readClientRegistrationStatuses } from "../../../src/application/ClientRegistrationStatus.js";
import { npxRegistrationCommand } from "../../../src/application/ClientRegistrationIdentity.js";
import { configureClientConfiguration } from "../../../src/application/SetupClientConfiguration.js";
import { skillDestinations } from "../../../src/application/SetupSkill.js";
import { supportedClients } from "../../../src/application/SupportedClients.js";
import { systemUninstallHost } from "../../../src/application/Uninstall.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { clearClientLocationEnvironment } from "../../fixtures/clientEnvironment.js";
/**
* The registration command setup writes on the host running the tests. Windows
* wraps it in `cmd.exe /d /c`, so the expected entry follows the host too.
*/
const command = npxRegistrationCommand(process.platform);
beforeEach(clearClientLocationEnvironment);
afterEach(() => vi.unstubAllEnvs());
const workbuddyPath = (
home: string,
platform: NodeJS.Platform = process.platform,
env: Parameters<typeof supportedClients>[2] = {},
) =>
supportedClients(home, platform, env).find(
({ name }) => name === "workbuddy",
);
describe("WorkBuddy configuration paths", () => {
it("resolves the global mcp.json under the home root", () => {
expect(workbuddyPath("/home/a")?.configPath).toBe(
join("/home/a", ".workbuddy", "mcp.json"),
);
expect(workbuddyPath("/Users/a")?.markerPath).toBe(
join("/Users/a", ".workbuddy"),
);
expect(workbuddyPath("C:\\Users\\a", "win32")?.configPath).toBe(
join("C:\\Users\\a", ".workbuddy", "mcp.json"),
);
});
it("resolves the registration format as the plain mcpServers JSON dialect", () => {
expect(workbuddyPath("/home/a")?.format).toBe("json");
});
it("resolves the personal skills directory under the home root", () => {
expect(
skillDestinations("/home/a", ["workbuddy"]).map(({ path }) => path),
).toEqual([
join("/home/a", ".workbuddy", "skills", "reverse-engineer-anything"),
]);
expect(
skillDestinations("C:\\Users\\a", ["workbuddy"], {}, "win32").map(
({ path }) => path,
),
).toEqual([
join("C:\\Users\\a", ".workbuddy", "skills", "reverse-engineer-anything"),
]);
});
});
describe("WorkBuddy JSON registration", () => {
const workbuddyClient = async () => {
const home = await createTestTempDirectory("rea-workbuddy-");
const client = workbuddyPath(home, process.platform);
if (client?.markerPath === undefined)
throw new Error("missing WorkBuddy client");
await mkdir(client.markerPath, { recursive: true });
return { home, client };
};
it("upserts rea into mcpServers while preserving unrelated settings", async () => {
const { home, client } = await workbuddyClient();
await writeFile(
client.configPath,
`{
"mcpServers": {
"other": { "command": "other-server" }
}
}`,
);
expect(
(await configureClientConfiguration(client, {}, command)).status,
).toBe("configured");
const parsed = JSON.parse(await readFile(client.configPath, "utf8"));
expect(parsed.mcpServers.other).toEqual({ command: "other-server" });
expect(parsed.mcpServers.rea).toEqual({
command: command[0],
args: command.slice(1),
});
expect(await configureClientConfiguration(client, {}, command)).toEqual({
status: "unchanged",
});
expect(
(
await readClientRegistrationStatuses(home, undefined, {
platform: process.platform,
environment: {},
})
).find(({ client }) => client === "workbuddy"),
).toMatchObject({ state: "aligned", command });
});
it("creates the mcpServers table in an existing mcp.json without servers", async () => {
const { client } = await workbuddyClient();
await writeFile(
client.configPath,
JSON.stringify({ unrelated: { setting: true } }, null, 2),
);
expect(
(await configureClientConfiguration(client, {}, command)).status,
).toBe("configured");
const parsed = JSON.parse(await readFile(client.configPath, "utf8"));
expect(parsed.unrelated).toEqual({ setting: true });
expect(parsed.mcpServers.rea).toEqual({
command: command[0],
args: command.slice(1),
});
});
it("registers into an empty mcp.json", async () => {
const { client } = await workbuddyClient();
await writeFile(client.configPath, "");
expect(
(await configureClientConfiguration(client, {}, command)).status,
).toBe("configured");
const parsed = JSON.parse(await readFile(client.configPath, "utf8"));
expect(parsed.mcpServers.rea).toEqual({
command: command[0],
args: command.slice(1),
});
});
it("removes only the rea entry on uninstall", async () => {
const { home, client } = await workbuddyClient();
await writeFile(
client.configPath,
JSON.stringify(
{ mcpServers: { other: { command: "other-server" } } },
null,
2,
),
);
await configureClientConfiguration(client, {}, command);
expect((await systemUninstallHost(home).removeClient(client)).status).toBe(
"removed",
);
const parsed = JSON.parse(await readFile(client.configPath, "utf8"));
expect(parsed.mcpServers.other).toEqual({ command: "other-server" });
expect(parsed.mcpServers.rea).toBeUndefined();
});
});
@@ -0,0 +1,138 @@
import { Client, InMemoryTransport } from "@modelcontextprotocol/client";
import { Ajv2020 } from "ajv/dist/2020.js";
import { expect, it } from "vitest";
import { FUNCTION_COMPARISON_EXAMPLE } from "../../../src/contracts/functionComparisonExample.js";
import { createEvidence } from "../../../src/domain/evidence.js";
import { functionDossierSchema } from "../../../src/domain/hopperValues.js";
import type { RelationshipCoverage } from "../../../src/domain/relationshipCoverage.js";
import { createServer } from "../../../src/server/createServer.js";
import { createTestBinarySession } from "../../fixtures/binarySession.js";
const observe = (relationshipCoverage: RelationshipCoverage) => {
const template = FUNCTION_COMPARISON_EXAMPLE.left;
if (template.subject === null)
throw new Error("Expected artifact-bound fixture");
const dossier = functionDossierSchema.parse(template.normalized_result);
return createEvidence(
{
path: template.subject.local_path,
sha256: template.subject.digest.sha256,
format: template.subject.format,
},
template.provider,
{
operation: "analyze_function",
parameters: template.parameters,
result: { ...dossier, callees: [], unresolved_calls: [] },
confidence: template.confidence,
authority: template.authority,
relationshipCoverage,
},
);
};
it("preserves directional gaps through advertised schemas and public path reconstruction", async () => {
const session = createTestBinarySession(() => {
throw new Error("Offline call paths must not launch a provider");
});
const server = createServer({ kind: "session", session });
const client = new Client({ name: "function-call-directions", version: "1" });
const [clientTransport, serverTransport] =
InMemoryTransport.createLinkedPair();
try {
await server.connect(serverTransport);
await client.connect(clientTransport);
const advertised = (await client.listTools()).tools.find(
({ name }) => name === "build_call_path",
);
if (advertised === undefined)
throw new Error("Missing build_call_path contract");
const ajv = new Ajv2020({ strict: false, validateFormats: false });
expect(ajv.validateSchema(advertised.inputSchema)).toBe(true);
const validate = ajv.compile(advertised.inputSchema);
for (const { directions, status, exhaustive } of [
{
directions: ["callers"] as ["callers"],
status: "not_found",
exhaustive: true,
},
{
directions: ["callees"] as ["callees"],
status: "unknown",
exhaustive: false,
},
{
directions: ["callers", "callees"] as ["callers", "callees"],
status: "unknown",
exhaustive: false,
},
]) {
const limitation =
"An observed call reference is absent from the inventory.";
const coverage: RelationshipCoverage = {
status: "partial",
limitations: [limitation],
limited_directions: directions,
};
const observation = observe(coverage);
expect(session.recordEvidence(observation).ok).toBe(true);
const input = {
functions: [observation],
start: { address: "0x1000" },
goal: { address: "0x9999" },
};
expect(validate(input), JSON.stringify(validate.errors)).toBe(true);
for (const limited_directions of [
[],
["callers", "callers"],
["callees", "callees"],
["callers", "callees", "callers"],
["incoming"],
])
expect(
validate({
...input,
functions: [
{
...observation,
relationship_coverage: {
...coverage,
limited_directions,
},
},
],
}),
).toBe(false);
const result = await client.callTool({
name: "build_call_path",
arguments: input,
});
if (result.isError) throw new Error(JSON.stringify(result));
expect(result.structuredContent).toMatchObject({
normalized_result: {
status,
search_scope: { exhaustive },
limitations: expect.arrayContaining([limitation]),
},
});
const comparison = await client.callTool({
name: "compare_functions",
arguments: { left: observation, right: observation },
});
if (comparison.isError) throw new Error(JSON.stringify(comparison));
expect(comparison.structuredContent).toMatchObject({
normalized_result: {
dimensions: expect.arrayContaining([
expect.objectContaining({ dimension: "calls", status: "unknown" }),
]),
},
});
// Each scenario owns its Evidence and residual-unknown ledger.
const closed = await session.close();
if (!closed.ok) throw closed.error;
}
} finally {
await Promise.allSettled([session.close(), client.close(), server.close()]);
}
});
+21 -11
View File
@@ -15,9 +15,17 @@ import {
type GhidraFunctionOperation,
} from "../../../src/ghidra/GhidraFunctionValues.js";
it.each([true, false])(
"compares observed empty Ghidra facets while preserving pseudocode availability (%s)",
async (pseudocodeAvailable) => {
it.each([
{ pseudocodeAvailable: true, reportsCoverage: true },
{ pseudocodeAvailable: false, reportsCoverage: true },
{ pseudocodeAvailable: true, reportsCoverage: false },
{ pseudocodeAvailable: false, reportsCoverage: false },
])(
"compares observed empty Ghidra facets with pseudocode=$pseudocodeAvailable and coverage=$reportsCoverage",
async ({ pseudocodeAvailable, reportsCoverage }) => {
const coverage = reportsCoverage
? { status: "complete" as const }
: undefined;
const dossier = functionDossierSchema.parse(ghidraFunctionDossier());
Object.assign(dossier, {
comments: [],
@@ -47,7 +55,13 @@ it.each([true, false])(
if (operation !== "analyze_function")
throw new Error(`Unexpected operation: ${operation}`);
return Promise.resolve(
ok({ value: jsonValueSchema.parse(dossier), limitations: [] }),
ok({
value: jsonValueSchema.parse(dossier),
limitations: [],
...(coverage === undefined
? {}
: { relationship_coverage: coverage }),
}),
);
},
);
@@ -61,9 +75,7 @@ it.each([true, false])(
harness.session,
analyzed.structuredContent,
);
expect(evidence.relationship_coverage).toMatchObject({
status: "complete",
});
expect(evidence.relationship_coverage).toEqual(coverage ?? null);
expect(evidence.normalized_result).toEqual(dossier);
const reply = await harness.mcp.callTool({
name: "compare_functions",
@@ -86,7 +98,7 @@ it.each([true, false])(
pseudocode: pseudocodeAvailable ? "unchanged" : "unknown",
assembly: "unchanged",
comments: "unchanged",
calls: "unchanged",
calls: reportsCoverage ? "unchanged" : "unknown",
references: "unchanged",
strings_names: "unchanged",
cfg: "unchanged",
@@ -96,9 +108,7 @@ it.each([true, false])(
evidence.parameters,
);
if (!replay.ok) throw replay.error;
expect(replay.value.relationshipCoverage).toEqual(
evidence.relationship_coverage,
);
expect(replay.value.relationshipCoverage).toEqual(coverage);
expect(harness.calls).toEqual(["analyze_function"]);
} finally {
await harness.close();
@@ -273,22 +273,6 @@ describe.skipIf(process.platform === "win32")(
expect(analysisArguments).toHaveLength(1);
});
it.each(["@options", '"@options with spaces"'])(
"refuses indirect JVM options (%s) instead of assuming platform alignment",
async (value) => {
const { session, launcher, analysisArguments } = await fixture(
`env-injection:JDK_JAVA_OPTIONS=${value}`,
);
const launched = await launcher.launch(session);
expect(launched.ok).toBe(false);
expect(analysisArguments).toEqual([]);
if (launched.ok) throw new Error("Expected an argument-file refusal");
expect(launched.error.cause).toMatchObject({
message: expect.stringContaining("may not supply JVM argument files"),
});
},
);
it("bounds retained UTF-8 warnings and reports original byte counts", async () => {
const { session, launcher } = await fixture("large");
const launched = await launcher.launch(session);
@@ -12,6 +12,8 @@ import ghidra.program.model.listing.FlowOverride;
import ghidra.program.model.listing.Function;
import ghidra.program.model.listing.Instruction;
import ghidra.program.model.symbol.SourceType;
import ghidra.program.model.symbol.Reference;
import ghidra.program.model.symbol.RefType;
import java.io.File;
import java.lang.reflect.Method;
import java.nio.file.Files;
@@ -20,9 +22,12 @@ import java.util.HexFormat;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.ArrayList;
/** Injects reported analysis flags into imported ELF/Mach-O functions; never executes target code. */
public class ReaNoReturnProbe extends GhidraScript {
private record MemoryCallReference(Address source, Address target, RefType type,
SourceType sourceType, int operandIndex, boolean primary) {}
private final JsonArray checks = new JsonArray();
private final Map<Function, Address> originalEnds = new LinkedHashMap<>();
private final Map<Function, String> originalSignatures = new LinkedHashMap<>();
@@ -85,6 +90,114 @@ public class ReaNoReturnProbe extends GhidraScript {
return instance;
}
private void knownInteriorCall(GhidraScript bridge) throws Exception {
Function caller = function("rea_tls_chain");
Function target = function("rea_tls");
Address interior = target.getEntryPoint().add(1);
require("interior_target_is_owned", currentProgram.getFunctionManager().getFunctionAt(interior) == null &&
target.equals(currentProgram.getFunctionManager().getFunctionContaining(interior)));
Instruction call = null;
for (Instruction instruction : currentProgram.getListing().getInstructions(caller.getBody(), true)) {
if (instruction.getFlowType().isCall()) { call = instruction; break; }
}
if (call == null) throw new IllegalStateException("Missing real call instruction");
var references = currentProgram.getReferenceManager();
List<Reference> originals = new ArrayList<>();
List<MemoryCallReference> saved = new ArrayList<>();
for (Reference reference : call.getReferencesFrom()) {
if (!reference.getReferenceType().isCall()) continue;
originals.add(reference);
saved.add(new MemoryCallReference(reference.getFromAddress(), reference.getToAddress(),
reference.getReferenceType(), reference.getSource(), reference.getOperandIndex(), reference.isPrimary()));
}
require("original_call_reference_present", !originals.isEmpty());
Reference changed = null;
try {
for (Reference reference : originals) references.delete(reference);
changed = references.addMemoryReference(call.getAddress(), interior,
RefType.UNCONDITIONAL_CALL, SourceType.USER_DEFINED, 0);
require("interior_target_omitted_by_callee_getter", !caller.getCalledFunctions(monitor).contains(target));
JsonObject query = new JsonObject();
query.addProperty("document", currentProgram.getName());
query.addProperty("procedure", caller.getName());
query.addProperty("direction", "outgoing");
Method read = bridge.getClass().getDeclaredMethod("procedureReferences", JsonObject.class);
read.setAccessible(true);
JsonObject output = (JsonObject) read.invoke(bridge, query);
require("interior_reference_keeps_known_procedure", output.getAsJsonArray("references").asList().stream().anyMatch(
edge -> !edge.getAsJsonObject().get("target_procedure").isJsonNull() &&
edge.getAsJsonObject().getAsJsonObject("target_procedure").get("name").getAsString().equals(target.getName())));
require("known_reference_is_not_also_unresolved", output.getAsJsonArray("unresolved_calls").isEmpty());
query.remove("direction");
Method callees = bridge.getClass().getDeclaredMethod("procedureCalls", JsonObject.class, boolean.class);
callees.setAccessible(true);
JsonObject inventory = (JsonObject) callees.invoke(bridge, query, false);
require("omitted_callee_inventory_stays_partial", inventory.getAsJsonObject("relationship_coverage").get("status").getAsString().equals("partial"));
require("omitted_callee_direction_is_explicit",
inventory.getAsJsonObject("relationship_coverage").getAsJsonArray("limited_directions").toString().equals("[\"callees\"]"));
require("interior_caller_omitted_by_entry_getter", !target.getCallingFunctions(monitor).contains(caller));
query.addProperty("procedure", target.getName());
JsonObject incoming = (JsonObject) callees.invoke(bridge, query, true);
require("off_entry_caller_inventory_stays_partial",
incoming.getAsJsonObject("relationship_coverage").get("status").getAsString().equals("partial"));
require("off_entry_caller_direction_is_explicit",
incoming.getAsJsonObject("relationship_coverage").getAsJsonArray("limited_directions").toString().equals("[\"callers\"]"));
JsonObject dossierQuery = new JsonObject();
dossierQuery.addProperty("procedure", target.getName());
Method analyze = bridge.getClass().getDeclaredMethod("analyzeFunctionObservation", JsonObject.class);
analyze.setAccessible(true);
Object observation = analyze.invoke(bridge, dossierQuery);
Method coverage = observation.getClass().getDeclaredMethod("relationshipCoverage");
coverage.setAccessible(true);
require("dossier_preserves_caller_only_gap",
((JsonObject) coverage.invoke(observation)).getAsJsonArray("limited_directions").toString().equals("[\"callers\"]"));
} finally {
if (changed != null) references.delete(changed);
for (MemoryCallReference reference : saved) {
Reference restored = references.addMemoryReference(reference.source(), reference.target(),
reference.type(), reference.sourceType(), reference.operandIndex());
references.setPrimary(restored, reference.primary());
}
}
}
private void nonCallInstructionReference(GhidraScript bridge) throws Exception {
Function caller = function("rea_tls_chain");
Function target = function("rea_memcpy");
Instruction source = null;
for (Instruction instruction : currentProgram.getListing().getInstructions(caller.getBody(), true)) {
if (!instruction.getFlowType().isCall()) { source = instruction; break; }
}
if (source == null) throw new IllegalStateException("Missing non-call instruction");
var references = currentProgram.getReferenceManager();
Reference added = references.addMemoryReference(source.getAddress(), target.getEntryPoint().add(1),
RefType.UNCONDITIONAL_CALL, SourceType.USER_DEFINED, -1);
try {
require("non_call_reference_target_omitted", !caller.getCalledFunctions(monitor).contains(target));
JsonObject query = new JsonObject();
query.addProperty("document", currentProgram.getName());
query.addProperty("procedure", caller.getName());
Method read = bridge.getClass().getDeclaredMethod("procedureCalls", JsonObject.class, boolean.class);
read.setAccessible(true);
JsonObject output = (JsonObject) read.invoke(bridge, query, false);
require("non_call_instruction_reference_stays_partial",
output.getAsJsonObject("relationship_coverage").get("status").getAsString().equals("partial"));
query.addProperty("direction", "outgoing");
Method edges = bridge.getClass().getDeclaredMethod("procedureReferences", JsonObject.class);
edges.setAccessible(true);
JsonObject known = (JsonObject) edges.invoke(bridge, query);
require("non_call_known_reference_is_not_unresolved", known.getAsJsonArray("unresolved_calls").isEmpty());
references.delete(added);
Address unknown = currentProgram.getMemory().getMaxAddress().add(0x1000);
added = references.addMemoryReference(source.getAddress(), unknown,
RefType.UNCONDITIONAL_CALL, SourceType.USER_DEFINED, -1);
JsonObject unresolved = (JsonObject) edges.invoke(bridge, query);
require("non_call_unknown_reference_keeps_site", unresolved.getAsJsonArray("unresolved_calls").size() == 1);
} finally {
references.delete(added);
}
}
@Override
public void run() throws Exception {
String[] args = getScriptArgs();
@@ -164,6 +277,8 @@ public class ReaNoReturnProbe extends GhidraScript {
Object output = pseudocode.invoke(bridge, query);
require("recovered_pseudocode", output instanceof JsonObject value &&
value.get("value").getAsString().contains("rea_after_call"));
nonCallInstructionReference(bridge);
knownInteriorCall(bridge);
} finally {
((ghidra.app.decompiler.DecompInterface) field.get(bridge)).dispose();
}
@@ -34,6 +34,10 @@ it
const procRoot = join(directory, "proc");
await mkdir(procRoot);
const alternateMode = mode === "alternate launcher";
let orphanPid: number | undefined;
const isolatedProcessList = async () => ({
stdout: alternateMode || orphanPid === undefined ? "" : "Xvfb",
});
if (alternateMode) {
await copyFile("/usr/bin/sleep", launcher);
await chmod(launcher, 0o700);
@@ -47,6 +51,7 @@ it
await assertCleanLinuxHopperRuntime(
alternateMode ? [launcher] : [],
procRoot,
isolatedProcessList,
);
const command = alternateMode ? alternate : process.execPath;
const arguments_ = alternateMode ? ["30"] : ["-e", childCode];
@@ -69,7 +74,6 @@ it
const parentPid = parent.child.pid;
if (parentPid === undefined)
throw new Error("Fixture parent did not spawn");
let orphanPid: number | undefined;
try {
const { stdout } = await parent;
const { pid } = JSON.parse(stdout) as { pid: number };
@@ -115,6 +119,7 @@ it
assertCleanLinuxHopperRuntime(
alternateMode ? [launcher] : [],
procRoot,
isolatedProcessList,
),
).rejects.toThrow("Clean Linux runner retained a Hopper demo process");
// Observation must not terminate the fixture, even though it resembles Xvfb.