feat: Export Chart Relationships to Graphviz DOT Format

[mits87]

Summary

Add support for extracting chart/diagram structures (nodes, edges, labels, and styling) into the Graphviz DOT format - graphiz_rust.

This would allow users to programmatically reconstruct diagrams, visualize relationships, and integrate with graph-based tooling.

Problem

Currently, when processing documents with diagrams (e.g. org charts, flowcharts, architecture diagrams), the extracted output lacks a structured representation of relationships between elements.

This makes it difficult to:

• Reconstruct diagrams programmatically
• Analyze relationships between entities
• Visualize or export diagrams into other systems
• Perform graph-based reasoning or transformations

Proposed Solution

Extend the extraction pipeline to detect and output diagram structures in DOT format.

The system should:

1. Detect Nodes

• Identify all diagram elements (boxes, shapes, entities) and node colors (if applicable).
• Extract texts (eg. using OCR)

2. Detect Edges (Connections)

• Identify relationships between nodes (arrows, lines)
• Extract:
  • Source node
  • Target node
  • Direction (if applicable)
  • Edge labels (if present)

3. Capture Styling (Optional but valuable)

• Node color / fill
• Border styles
• Edge styles (dashed, solid, etc.)

4. Preserve Text Context

• Text inside nodes
• Text between nodes (e.g. relationship labels)

Expected Behavior

When a document contains a diagram:

• The system outputs a DOT representation alongside existing extraction results

The output should be:

• Deterministic
• Clean (no duplicates)
• Referencing consistent node IDs

Use Cases

• Org chart extraction
• System architecture diagrams
• Process flows
• Dependency graphs
• Legal/entity relationship mapping

Why This Matters

This unlocks:

• Graph-based processing (Neo4j, NetworkX, etc.)
• Visualization pipelines (Graphviz, D3, Mermaid conversion)
• Better explainability (traceable relationships vs flat text)
• Stronger downstream automation (agents operating on graphs)

Example diagram output

digraph G {
  rankdir=TB;

  node1 [label="Node A" shape=box fillcolor="#a6cee3"];
  node2 [label="Node B" shape=diamond fillcolor="#b2df8a"];
  node3 [label="Node C" shape=box fillcolor="#fb9a99"];
  node4 [label="Node D" shape=ellipse fillcolor="#fdbf6f"];

  note1 [label="Some annotation" shape=note];

  node1 -> node2 [label="50%"];
  node2 -> node3 [label="relates"];
  node3 -> node4;
  node2 -> note1 [style=dashed arrowhead=none];
}

Scope / Notes

• Should support extraction from both PDFs and images
• The diagram output should be an additional structured extraction, included under content.diagram (without affecting existing outputs)
• All detected nodes and edges must be traceable to source, including at minimum page reference and bounding box (for validation and debugging)

Examples

Observation

Most of the Kreuzberg pipeline is there (rasterize pdf pages, OCR and text recognition, text coordinates etc.), the missing part is diagram / charts recognition (nodes and edges + text connection).

[Goldziher]

Thanks for the detailed feature request — this is a compelling use case and we'd love to support it.

We've done extensive research into what's available today for diagram element detection (nodes, edges/arrows, labels) and want to share our findings transparently.

Current State of Available Models

The core challenge is Stage 2 — parsing the internals of a diagram. Kreuzberg already handles Stage 1 (detecting diagram/figure regions in documents via our layout detection models) and Stage 3 (OCR for text within detected regions). What's missing is a model that can detect individual shapes (boxes, diamonds, ovals), arrows/connections, and their relationships.

Here's what we found:

• No permissively-licensed ONNX model exists that detects diagram elements (nodes, arrows, shapes) at production quality. The closest candidate (ELCA-SA/BPMN_Detection, F1=0.96 objects / 0.92 arrows) is MPL-2.0 licensed, which is incompatible with our codebase.
• Grounding DINO (Apache 2.0, ONNX available) can do zero-shot text-prompted detection ("arrow", "rectangle", "diamond"), but it was trained entirely on photographs — performance on technical diagrams is untested and likely insufficient.
• google/deplot (Apache 2.0) only handles data charts (bar, line, pie) → table conversion. It cannot process flowcharts or relational diagrams.
• diagram2graph (Apache 2.0, Qwen2.5-VL-3B) does end-to-end diagram → JSON extraction but is a 3B parameter VLM — far too large for ONNX deployment in our pipeline.

Permissive Datasets That Exist

The training data situation is better:

• hdBPMN (CC-BY 4.0) — 502 hand-drawn BPMN diagrams, 25 element types, COCO format
• FloCo (MIT) — 11,884 flowchart images
• Roboflow flowchart datasets (CC-BY 4.0) — ~774 annotated images
• DIDI (Google Research) — 58K+ digital ink diagrams

Our Plan

We're adding this to our roadmap as a later-phase feature. The path forward is:

1. Fine-tune our existing RT-DETRv2 or YOLO model on the permissive datasets above, with an added keypoint head for arrow start/end detection
2. Export to ONNX and integrate into the existing kreuzberg layout detection pipeline as a second-stage model
3. Build the graph construction post-processing: match arrow endpoints to node bounding boxes, OCR text within nodes, emit DOT/JSON output

This keeps everything MIT-licensed and within our existing ONNX inference pipeline. We'll update this issue as we make progress.

The observation in the issue is exactly right — most of the pipeline is already there. The missing piece is the diagram element detection model, and we'll need to train one ourselves on permissive data.

[mits87]

@Goldziher Hi! Just following up on this feature request — do you have any rough timeline for when this could be implemented?

This functionality is quite important on our side, so it would really help us plan accordingly if there’s any estimate (even a rough one).

If this isn’t something that can be delivered in the near term, would it be possible to expose at least a lightweight signal indicating whether a document likely contains a diagram / org chart / ownership structure? For example:

{
  "diagram": {
    "confidence": 0.8
  }
}

Even a simple confidence score (or boolean flag) per chunk / page would already allow us to improve routing and handling on our side.

Thanks a lot!

[Goldziher]

@mits87 -- so realilstically this will probably have to be done via hybrid extraction. Namely, we will need to extract diagrams and send them into VLM extraction. It will be included in our commercial offering for certain (our SAAS / self-hosted solution). If this is interesting for you guys, lets have a chat (via either discord or email -- we can set a meeting as wel).

For extracting confidence info -- this is doable now. I'll try to make it in the next minor version. Which will come next week or even earlier.

[Goldziher]

@mits87 -- api is extended, and in the next verison you will be able to get the data. BUT the layout model does not distinguish between "picture" and "diagram" - its all "picture".

[mits87]

@Goldziher this is already very helpful, thanks!

One follow-up: would it be possible to add an option to plug in a custom detection mechanism (e.g. a vision LLM classifier), so I can determine what’s on the page on my side?

[Goldziher]

@Goldziher this is already very helpful, thanks!

One follow-up: would it be possible to add an option to plug in a custom detection mechanism (e.g. a vision LLM classifier), so I can determine what’s on the page on my side?

Open a separate issue please