* feat(native): search analyzed memory for masked byte patterns The diff adds search_bytes to the CLI, the shared MCP contract, Ghidra inventory, and the headless bridge, and advertises the same operation on Hopper as unavailable until that provider is verified. Patterns are hexadecimal byte pairs with optional ?? wildcards and an optional equal-length bit mask; the TypeScript boundary and the bridge both require a nonempty pattern, equal lengths, and at least one constrained bit. Ghidra walks initialized blocks in address order within inclusive bounds, keeps overlapping matches inside a single block, and returns each hit with its segment, unique source file offset, and containing procedure, continuing through a cursor bound to the target, profile, pattern, mask, and bounds. Each page reports the inclusive ranges examined and the skipped uninitialized blocks; cancellation returns a typed error and discards the active Ghidra database. address_to_file_offset now uses the same unique-offset helper, which rejects a mapping that is missing, ambiguous, or larger than a safe integer. * fix(native): limit unbounded byte search to loaded memory Without start or end, search_bytes walked every Ghidra block, including non-loaded file parts such as .symtab and overlay blocks, which the docs call out of scope. Unbounded searches now skip those blocks and list them in searched.skipped_not_loaded; explicit bounds in their address space still search them. Document that a match must end at or before end. * fix(native): preserve implicit address spaces in range consumers * fix(native): reject present malformed byte search state --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
230 lines
22 KiB
Markdown
230 lines
22 KiB
Markdown
# Ghidra provider semantics
|
|
|
|
REA imports one target into an ephemeral Ghidra database. Linux/macOS support
|
|
read-only analysis and atomic function-name/entry-comment edits; Windows P0 is
|
|
read-only. GUI controls and persistent database mutation are unavailable.
|
|
Executable bytes remain unchanged.
|
|
|
|
Use [installation](installation.md#ghidra) for supported hosts, prerequisites,
|
|
configuration, startup deadlines, resource controls, and cleanup recovery.
|
|
[Windows P0](windows-ghidra-p0.md) has its own admission and lifecycle boundary;
|
|
[NativeAOT observations](ghidra-nativeaot.md) include a built-in read-only PE
|
|
snapshot parser and a separate optional annotation extension. Check the
|
|
[release boundary](installation.md#released-package-and-main) when using npm.
|
|
|
|
See [import language and analysis seeds](installation.md#ghidra-import-language-and-analysis-seeds)
|
|
to override the header-detected language or seed analysis.
|
|
|
|
Provider selection and profile identity follow
|
|
[MCP contracts](mcp-contracts.md#identity-and-discovery). One deep
|
|
provider stays bound to the target; runtime failure never selects another
|
|
provider automatically. The contracts below describe Ghidra observations,
|
|
not equivalence with another engine's pseudocode or analysis database.
|
|
|
|
## Admitted inventory semantics
|
|
|
|
| Concern | Ghidra contract |
|
|
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| Program identity | One `analyzeHeadless` import produces exactly one Program; `list_documents` therefore returns exactly one name. |
|
|
| Addresses | Default memory uses lowercase `0x` hexadecimal. Non-default and external spaces use `<percent-encoded-space>:0x<hex>` and remain round-trippable. The handshake commits image base and default address-space name. |
|
|
| Symbols | `list_names` includes address-bearing memory and external symbols, including dynamic symbols, while excluding variable and no-address namespace records. Each item reports primary, dynamic, external, symbol type, and source facts. |
|
|
| Procedures | Both non-external and external functions are listed. A local thunk remains distinct from its resolved target; exact and qualified name lookup fails on ambiguity rather than guessing. |
|
|
| Strings | Only Ghidra-defined string `Data` is observed. Items report charset, byte length, and whether a required null terminator is missing. The API cannot distinguish a present terminator from fixed/Pascal layouts when no terminator is missing, so that state is named `present_or_not_required`. |
|
|
| Memory | Memory-block end addresses are exclusive. Read/write/execute, initialization, overlay, address space, and image base are direct Ghidra observations. |
|
|
| Inventory | Procedure, symbol, and string listings plus searches return the complete matching collection in one response; callers do not provide offsets or result-count limits. |
|
|
| Search | Literal and Java-regex searches return every match in the immutable provider-defined inventory; string search does not cover every text sequence in artifact bytes. Search waits for a reply or caller cancellation; there are no caller-supplied offsets or result-count limits. |
|
|
| Analysis state | The socket is exposed only after default auto-analysis completes. Established operations wait for their reply or caller cancellation; there is no fixed response-size ceiling. |
|
|
|
|
String-search completeness applies only to Ghidra-defined string `Data`. Packed
|
|
or undefined literals, including Rust strings, may be absent; an empty result
|
|
does not prove text is absent. Both list and search Evidence carry this
|
|
limitation. When an address is known, use `read_bytes` and `xrefs` to inspect
|
|
bytes and references.
|
|
|
|
## PE debug-symbol coverage
|
|
|
|
The post-analysis handshake reports Ghidra's PDB Program Information attributes:
|
|
the referenced file, GUID, hexadecimal age, legacy signature, version, and
|
|
`PDB Loaded` state. Absent attributes remain null. PE Evidence carries a
|
|
limitation when a PDB reference is present but Ghidra reports it as not loaded,
|
|
or its load state is unknown. A bridge without this report leaves PDB coverage
|
|
unknown. A reported loaded state is a Ghidra observation, not an independent
|
|
PDB identity or complete symbol-coverage attestation, and retains an inline
|
|
coverage caveat. Handshake PDB strings share a 4096 UTF-16-unit serialization
|
|
budget (at most 24 KiB after JSON escaping); whole attributes that do not fit
|
|
are omitted with an explicit resource limitation, rather than truncated.
|
|
|
|
Original names and types may be missing even when auto-analysis completes.
|
|
An empty name search does not establish that a function is absent; use
|
|
`list_procedures` and address-based analysis to inspect recovered functions.
|
|
REA imports a private executable copy and does not admit a caller-selected
|
|
PDB companion. This reporting does not search symbol servers, copy a sibling
|
|
PDB, or change analyzer settings.
|
|
|
|
## Loaded addresses and file offsets
|
|
|
|
`rea analyze <path> --provider ghidra --json` and MCP `binary_overview` report
|
|
the observed loaded `image_base` in `normalized_result`. They retain each
|
|
segment's reported `image_base` and `address_space`. A provider that does not
|
|
report a base, or whose segments disagree, omits the overview field, following
|
|
the existing contract on main. REA does not substitute zero or the first
|
|
segment's address. All reported bases and address spaces remain visible in the
|
|
complete segment inventory, including conflicting observations.
|
|
|
|
`function` and most MCP procedure selectors use procedure names or
|
|
provider analysis addresses. Ghidra `decompile` / MCP `procedure_pseudo_code`
|
|
also accept source-file byte offsets. Existing symbols and analysis addresses
|
|
take precedence; a `0x`-prefixed number that resolves to no procedure falls back
|
|
to its observed source mapping. Use `--file-offset=0x...` on the CLI or
|
|
`procedure: "file-offset:0x..."` through MCP to select file coordinates explicitly.
|
|
Multiple mapping candidates fail as ambiguous rather than selecting a bank or
|
|
address space. A successful conversion retains the file offset, mapped address
|
|
and procedure entry in raw Evidence and `locations`.
|
|
|
|
If an offset maps to loaded target bytes but no procedure starts at or contains
|
|
the mapped address, the failure preserves both the original file offset and
|
|
analysis address. This differs from an unmapped offset and does not recommend
|
|
retrying the same offset syntax.
|
|
If the function resolves but the backend fails or cancels decompilation, the
|
|
failure retains the selector, mapped address and procedure entry without
|
|
changing the backend failure code.
|
|
|
|
Both mapping directions require a complete original `FileBytes` source whose
|
|
SHA-256 matches the admitted target. Auxiliary files and partial source objects
|
|
remain visible in the load-image inventory but do not establish offsets in the
|
|
original target. Ghidra's object-local offsets are not silently treated as
|
|
original-file coordinates. Original-source identity is checked once per source
|
|
object per session; analysis-memory edits do not change the original bytes.
|
|
|
|
ELF virtual addresses and file offsets are
|
|
different coordinates. Ghidra can rebase a PIE ELF during import: in the
|
|
source-owned verification fixture, ELF entry `0x1013` becomes Ghidra procedure
|
|
`0x101013` with image base `0x100000`. That example is not a general conversion
|
|
rule.
|
|
|
|
Use `list_procedures` to select an observed procedure address. To relate file
|
|
bytes to loaded memory, call MCP `inspect_native_load_image` or:
|
|
|
|
```sh
|
|
rea search <path> . --kind procedures --mode regex --provider ghidra --json
|
|
rea inspect-native-load-image <path> --provider ghidra --json
|
|
rea decompile <path> <observed-procedure-address> --provider ghidra --json
|
|
rea decompile <path> --file-offset=0x1013 --provider ghidra --json
|
|
rea address-to-file-offset <path> <observed-procedure-address> --provider ghidra --json
|
|
```
|
|
|
|
The load-image result preserves source-file indexes, file offsets, address
|
|
spaces, and inclusive mapping ranges. Follow the mapping for the selected
|
|
source file and region; adding `image_base` to an arbitrary file offset is not
|
|
valid. For a linear byte-addressed, file-backed mapping, a file offset `F` in
|
|
`[mapping.file_offset, mapping.file_offset + mapping.length)` maps to
|
|
`mapping.start + (F - mapping.file_offset)` in the reported `address_space`.
|
|
If several mappings cover the same file offset, keep those candidates distinct
|
|
and select the procedure from the observed inventory. Uninitialized regions may
|
|
have no file backing. ELF mappings are observed
|
|
without independent format-specific load-image verification, as the result's
|
|
status and limitations state.
|
|
|
|
An unsuccessful numeric procedure lookup reports the current Ghidra image base
|
|
and gives the CLI commands above alongside the corresponding MCP operations.
|
|
Repeat any target-format, architecture, and entry-point options used to open the
|
|
target. Other procedure operations keep their analysis-address semantics;
|
|
file-offset fallback applies only to decompilation.
|
|
|
|
Overview workflow revision 3 binds this output to snapshot identity. Older
|
|
serialized overview Evidence remains readable; it is not replayed as a current
|
|
overview merely because its source artifact is unchanged.
|
|
|
|
## Admitted function-analysis semantics
|
|
|
|
| Concern | Ghidra contract |
|
|
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| Decompiler lifetime | One persistent `DecompInterface` is opened for the imported Program and disposed during bridge shutdown. Native decompilation has no fixed per-function deadline. External functions or functions without bodies return `null`; cancellation and native failure remain distinct. |
|
|
| Serialization | A FIFO sends one Program request at a time. Caller cancellation removes queued work promptly and is passed to active socket waits. This is an adapter safety commitment, not a claim that every Ghidra API is thread-safe. |
|
|
| Function identity | Every function result carries the entry address and Ghidra FunctionManager classification for external, thunk, and resolved thunk target. These are observations, not proof that unresolved targetless calls have been recovered. |
|
|
| Assembly and pseudocode | Assembly is complete Ghidra Listing text; pseudocode is Ghidra decompiler output. Neither is original source, and cross-provider comparison never treats Hopper and Ghidra text as equal or unequal semantic facts. |
|
|
| Instruction fast path | `read_function_instructions` returns every raw Listing instruction for the requested function without invoking the decompiler or whole-program name/string inventories. |
|
|
| Calls and references | Callers/callees contain only resolved functions. Reference edges preserve exact ReferenceManager type and call/jump/data/read/write/indirect/computed/conditional/terminal/external facts. Targetless computed flow remains unknown. Synthetic entry-point references without actionable memory sources are omitted explicitly. |
|
|
| CFG | Dossiers use `BasicBlockModel` and retain only non-call successors inside the function body. CFG topology is address-normalized for comparison; provider-specific block construction remains a declared difference. |
|
|
| P-code value flow | Function dossiers include up to 3,000 high-p-code operations, at most 64 inputs per operation, and 12,000 def-use edges. Results identify memory reads/writes, branches, and calls by p-code opcode, and mark decompiler-dead operations. Large results report truncation, an omitted-operation lower bound, and known omitted-input/edge counts. The dossier is an intra-function graph; `trace_native_values` composes derived parameter/argument and return/output dependencies across resolved calls. It does not resolve memory aliasing, call side effects, runtime behavior, persistent state, or business meaning. |
|
|
| Result extent | Function instruction scans and native API boundary observations remain complete. P-code flow is the explicit bounded exception; its count fields distinguish exact retained counts from lower-bound or known-omitted counts. Caller cancellation and provider failures remain distinct. |
|
|
|
|
Function extents and instruction completeness refer to Ghidra's analyzed body.
|
|
An inferred no-return call can terminate that body before subsequent bytes.
|
|
Pseudocode, assembly, procedure info, instruction windows, callee and caller
|
|
reads, and dossiers report observed terminal call sites in Evidence limitations,
|
|
including resolved callee names, addresses, and FunctionManager flags when
|
|
available. Instruction windows
|
|
and dossiers also retain these warnings in their result limitations. An
|
|
unresolved callee or unestablished terminal-flow basis remains explicit. These
|
|
observations qualify the analysis; they do not prove that a no-return flag is
|
|
incorrect or recover excluded code. Analysis profiles version this observation
|
|
contract so older snapshots cannot replay unqualified results.
|
|
|
|
On non-Windows hosts, the ephemeral import repair clears no-return flags on the
|
|
known returning external ABI names `malloc`, `__tls_get_addr`, `memcpy`, `memset`,
|
|
`strlen`, and `__errno_location`. This relies on their expected ABI behavior;
|
|
external implementations are not inspected. The repair re-disassembles affected
|
|
terminal calls and refits their owning functions, preserving independent function
|
|
entries. If a recovered caller reveals a decoded return, its propagated no-return
|
|
flag and affected callers are repaired in the same pass. Unknown imports and
|
|
genuine no-return imports keep their flags. Analysis profiles version this repair
|
|
so snapshots produced before it are not reused. Windows P0 does not run it.
|
|
|
|
`npm run verify:ghidra` compiles the versioned C oracle into debug and stripped
|
|
host-native targets (x86-64/AArch64 ELF on Linux or Mach-O on macOS), plus a native
|
|
DWARF 4 type-layout object. It proves the admitted operations, external functions, resolved thunks, exports, stripped-name
|
|
behavior, direct and targetless indirect calls, typed references, strings/xrefs,
|
|
multi-block CFG, bounded p-code def-use/effect links, semantic enhanced workflows, cancellation, startup deadlines,
|
|
serialized concurrency, malformed-target rejection, profile identity, and
|
|
process/project cleanup against real Ghidra 12.1.4. Unit fixtures separately
|
|
cover startup deadlines, process exit, queued cancellation, and malformed wire
|
|
output.
|
|
|
|
`npm run verify:ghidra:cross-format` adds AArch64 ELF, x86-64 PE, and x86-64
|
|
Mach-O target coverage. It requires `clang`, LLD, and `lld-link`; set
|
|
`REA_CLANG`, `REA_LLD`, or `REA_LLD_LINK` to select them. The verifier checks
|
|
these tools before compilation. Keeping this matrix separate lets Linux
|
|
host/provider acceptance run with only the host compiler.
|
|
|
|
`npm run verify:ghidra:windows` checks source-owned native x86-64 PE applications;
|
|
`-- --x86` selects x86 and `-- --dll` selects DLLs, and the flags can be combined.
|
|
These lanes check all 25 read-only operations, target/snapshot/import SHA-256 linkage,
|
|
and project, endpoint, process, and runtime cleanup. The independent native
|
|
lane checks DACLs, handle admission, cancellation, and Job Object lifecycle.
|
|
`npm run verify:ghidra:windows:package` packs and installs REA into an isolated
|
|
prefix, then checks ordinary-user CLI/MCP operations against canonical contracts.
|
|
It accepts the same architecture and DLL flags and verifies the selected PE role
|
|
in the analysis profile.
|
|
All lanes require matching native controls; package startup alone establishes
|
|
only package compatibility.
|
|
|
|
## Masked byte search
|
|
|
|
`search_bytes` (`rea search-bytes PATH '48 8B ?? ?? 89' --provider ghidra`)
|
|
searches initialized analyzed memory, including loader fixups. Supply hexadecimal
|
|
byte pairs, with `??` for any byte. An optional equal-length hexadecimal `mask`
|
|
selects bits: `pattern: "a0"`, `mask: "f0"` matches bytes from `a0` through `af`.
|
|
Wildcards remain unconstrained even with an explicit mask. At least one bit
|
|
must be constrained.
|
|
|
|
Optional `start` and `end` addresses are inclusive and select one address space.
|
|
A match must end at or before `end`, so `end` set inside a match excludes it.
|
|
Without bounds, loaded memory blocks are eligible, in Ghidra address order.
|
|
Blocks that are not loaded (file parts such as `.symtab`) and overlay blocks are
|
|
listed in `searched.skipped_not_loaded` and are not searched; pass `start` or
|
|
`end` in their address space to search them.
|
|
Overlapping matches are included; matches never cross block boundaries.
|
|
Each match reports its address, segment, unique source file offset (or null),
|
|
and containing procedure (or null).
|
|
|
|
Pages default to 100 matches and accept 1 to 1000. Pass `next_cursor` as `cursor`
|
|
with the same target, profile, pattern, mask and bounds to continue. `complete`
|
|
means no further matches remain. `searched.blocks` and `searched.bytes` describe
|
|
the inclusive memory ranges examined for this page, including lookahead for the
|
|
next match. Coverage can overlap between pages; do not sum it as unique memory.
|
|
`searched.skipped_uninitialized` lists eligible uninitialized blocks on every
|
|
page. These blocks are never searched. Cancellation returns a typed error,
|
|
discards the active Ghidra database, and does not return a successful partial page.
|
|
Hopper byte search remains unavailable until its implementation is verified.
|