Skip to content
BranchesTags

Repository files navigation

PatchPOC

Binary patch diffing and vulnerability analysis with multi-agent AI

Python 3.12+ CrewAI License: MIT

Features

  • Two pipeline variants - sequential 4-agent crew + conditional 5-agent pipeline with CVE-mode branching
  • Autonomous analysis loop - Sisyphus-style iteration that re-runs the crew until quality converges
  • Deterministic preprocessing - BinDiff candidate selection, CodeQL context extraction, and CVE enrichment run before any LLM call
  • Docker-based binary analysis - IDA + BinExport + BinDiff + TRAP + CodeQL in a container
  • MCP integration - query-only servers for BinDiff results and CodeQL databases
  • Multi-LLM support - OpenAI, Google Gemini, Anthropic Claude, DeepSeek with per-agent model selection
  • Structured outputs - Pydantic models for validated TRAP reports (Triage → Research → Analysis → PoC)

Architecture

Pipeline Overview

INPUT: V_vuln + V_patched (ELF x86_64 binaries)
          │
          ▼
┌─────────────────────────────────────────────┐
│  Docker Container - Preprocessing            │
│                                              │
│  IDA Headless Analysis (both binaries)       │
│         │                                    │
│   ┌─────┴─────┐                              │
│   ▼           ▼           (parallel)         │
│ BinExport   TRAP Extraction                  │
│   → BinDiff   → CodeQL DB Construction       │
└─────┬──────────┬────────────────────────────┘
      │          │
      ▼          ▼
┌─────────────────────────────────────────────┐
│  Deterministic Seed Layer (no LLM)           │
│                                              │
│  • BinDiff candidate ranking                 │
│  • CodeQL semantic context extraction        │
│  • CVE intelligence pre-fetch (NVD, EPSS,    │
│    CISA KEV, GitHub PoC Index, ExploitDB)    │
│  • Web search for feature context            │
└──────────────────┬──────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────┐
│  AnalysisLoop - Autonomous Iteration         │
│                                              │
│  for each iteration:                         │
│    1. Create crew (sequential or conditional) │
│    2. Inject pre-built context + learnings   │
│    3. Run kickoff → score quality            │
│    4. Accumulate findings across iterations  │
│                                              │
│  Stop when:                                  │
│    • Quality ≥ threshold (default 0.7)       │
│    • Findings converge (<5% improvement)     │
│    • Max iterations reached                  │
└──────────────────┬──────────────────────────┘
                   ▼
         TRAPReport (structured output)

Agent Architecture

Sequential Pipeline (agents/crew.py)

The 4-agent sequential crew:

Agent Role Tools Output
① CodeQL Query CodeQL Query Specialist BinQL MCP CodeQLQueryResult
② Code Analysis Deep Code Analyst None (reasoning) DeepAnalysisResult
③ Vuln Analysis Vulnerability Classifier CVE intelligence VulnAnalysisResult
④ Report Writer Security Report Writer None (synthesis) VulnerabilityReport

BinDiff candidate selection runs deterministically in the CLI before the crew starts. Results are injected as context into the first agent.

Conditional Pipeline (pipeline/crew.py) - with CVE branching

The 5-agent conditional pipeline inserts a CVE correlation agent when a CVE ID is provided:

Discovery mode (no CVE):

DiffTriage → FunctionAnalyst → VulnClassifier → ExploitSynthesis

Correlation mode (CVE provided):

DiffTriage → FunctionAnalyst → VulnClassifier → CVECorrelation → ExploitSynthesis

The CVE correlation agent links discovered findings to known advisory data, enriching the report with CVSS, CWE, EPSS, CISA KEV status, and public exploit references.

Module Structure

src/patchpoc/
├── cli.py                     # CLI entry point & orchestration
├── agents/                    # Sequential pipeline
│   ├── crew.py                # 4-agent crew assembly
│   ├── analysis_loop.py       # Autonomous iteration loop (AnalysisLoop)
│   ├── llm_hooks.py           # LLM call logging & tool-call counters
│   └── tasks/                 # Task definitions for sequential agents
├── pipeline/                  # Conditional pipeline
│   ├── crew.py                # 5-agent crew with CVE-mode branching
│   ├── loop.py                # Pipeline iteration driver
│   └── context.py             # Pipeline context management
├── tools/                     # Shared tool modules
│   ├── cve.py                 # CVE intelligence (NVD, EPSS, KEV, GitHub PoCs)
│   ├── web_search.py          # DuckDuckGo search + page fetch
│   ├── mcp.py                 # MCP client (Diff :8891, BinQL :8892)
│   ├── workspace.py           # Sandboxed file I/O for agent memory
│   ├── codeql_builder.py      # Deterministic CodeQL context extraction
│   └── diff_builder.py        # Deterministic BinDiff candidate seeding
├── preprocessing/             # Docker preprocessing orchestration
│   ├── artifacts.py           # Locate & validate BinExport/CodeQL/BinDiff outputs
│   ├── cache.py               # SHA-256 hash-based binary pair caching
│   └── docker.py              # Container execution & docker-compose control
├── ui/                        # Console display
│   └── display.py             # Rich formatting for task progress & results
├── schemas/
│   └── models.py              # Pydantic output models (TRAPReport, etc.)
└── config/
    └── llm_config.py          # Multi-backend LLM configuration

Installation

git clone https://github.com/anhhung04/patchpoc.git
cd patchpoc
git submodule update --init --recursive

# Install with uv (recommended)
uv pip install -e .

# Or with pip
pip install -e .

# Configure
cp config.example.yml ~/.patchpoc/config.yml
# Edit config with your API keys and preferences

Quick Start

# Full pipeline: preprocessing + multi-agent analysis
patchpoc --vuln ./bins/vulnerable --patched ./bins/patched --cve CVE-2024-1234

# JSON output
patchpoc --vuln ./bins/vulnerable --patched ./bins/patched --cve CVE-2024-1234 --output-json report.json

# Analyze more candidate functions
patchpoc --vuln ./bins/vulnerable --patched ./bins/patched --cve CVE-2024-1234 --count 20

# Skip preprocessing (use pre-computed outputs)
patchpoc --vuln ./bins/vulnerable --patched ./bins/patched --cve CVE-2024-1234 \
  --skip-preprocessing --output ./precomputed/

# Using Make
make run VULN=./bins/vulnerable PATCHED=./bins/patched CVE=CVE-2024-1234
make preprocess VULN=./bins/vulnerable PATCHED=./bins/patched
make analyze OUTPUT=./output CVE=CVE-2024-1234

Configuration

LLM Configuration

Set your preferred provider via environment or config:

# Environment variables
export PATCHPOC_LLM_PROVIDER=openai
export PATCHPOC_LLM_MODEL=gpt-4o
export OPENAI_API_KEY=sk-...

# Or in config.yml with per-agent overrides:
agent_models:
  code_analysis: anthropic/claude-sonnet-4-20250514
  vuln_analysis: anthropic/claude-sonnet-4-20250514

Supported providers: OpenAI, Anthropic Claude, Google Gemini, DeepSeek.

Docker Services

# Build preprocessing image
make docker-build-preprocessor

# Start MCP servers
make docker-up

# View logs
make docker-logs

MCP servers:

  • Diff MCP (:8891) - BinDiff results query
  • BinQL MCP (:8892) - CodeQL database queries

Models & Output

TRAP Report

The final output is a TRAPReport (Triage, Research, Analysis, PoC):

Section Content
Metadata Binary hashes, tool versions, duration, iteration count, quality score
DiffCheckResult Triage: ranked candidate functions from BinDiff
VulnerabilityReport Research + Analysis: classified findings with CWE, CVSS, root cause, exploitability
CVEMatch CVE correlation (when CVE ID provided)
ExploitArtifacts PoC exploits with trigger steps, crash signatures, proof-of-concept code

Each vulnerability finding includes: function name, CWE ID, severity, CVSS estimate, root cause, patch description, source→sink paths, call chains, exploitation primitives, and trigger analysis.

Vulnerability Detection

Detects and classifies:

  • Buffer Overflow (CWE-120, CWE-121, CWE-122)
  • Use-After-Free (CWE-416)
  • Integer Overflow (CWE-190)
  • Null Pointer Dereference (CWE-476)
  • Out-of-Bounds Read/Write (CWE-125, CWE-787)
  • Format String (CWE-134)
  • Double Free (CWE-415)
  • Race Condition (CWE-362)
  • And other CWE classifications

Tech Stack

  • CrewAI - Multi-agent orchestration
  • FastMCP - Model Context Protocol servers
  • Pydantic - Structured output schemas
  • IDA Pro - Binary disassembly / decompilation
  • BinDiff - Binary diffing
  • CodeQL - Semantic code analysis
  • Docker - Preprocessing containerization
  • DuckDuckGo Search - Feature context discovery
  • Python 3.12+ - Core runtime

Benchmark Results

Evaluated on 52 real CVEs across 8 open-source C projects, with CVE selection guided by prior binary analysis literature and NVD records.

Metric Value
Dataset 52 CVEs, 8 projects (curl, openssl, libtiff, libxml2, openjpeg, ffmpeg, php-src, binutils)
VIR (detection) 98.1% (51/52)
VCA-exact (CWE match, normalized) 94.2% (49/52)
VCA-hierarchy (CWE hierarchy-aware) 94.2% (49/52)
Mean CVSS delta 1.37
Mean confidence 0.88
Mean analysis time 232.9s per binary pair

Per-project breakdown:

Project N VIR% VCA-exact% VCA-hier% CVSS delta Conf
binutils 6 100.0 66.7 100.0 1.02 0.87
curl 6 100.0 100.0 100.0 1.87 0.87
ffmpeg 8 100.0 50.0 100.0 1.51 0.84
libtiff 7 100.0 57.1 85.7 1.16 0.89
libxml2 7 85.7 71.4 71.4 1.30 0.88
openjpeg 5 100.0 60.0 100.0 1.14 0.91
openssl 8 100.0 50.0 87.5 1.54 0.91
php-src 5 100.0 60.0 100.0 1.36 0.89

CWE hierarchy-aware matching uses a taxonomy of 110 CWEs with parent-child relationships. A detected CWE scores 1.0 for exact match, 0.8 for parent/child, 0.6 for grandparent, and 0.4 for same-category. VCA-hierarchy counts any match scoring >= 0.6.

Run the evaluation:

PYTHONPATH=src python3 benchmark/scripts/evaluate.py

Development

make dev-install
make test       # 648 tests
make lint
make format

License

MIT License

About

No description, website, or topics provided.

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages