Proposal: Extensibility changes to unblock ML and OCR integrations

[luisquintanilla]

Following up on this discussion about the ONNX layout detection and data ingestion packages. @BobLd mentioned PdfPig can help by "making classes and constructors public, adding new interfaces, etc.". Here are the specific things I've been working around.

I ran into three spots where PdfPig's core types don't quite bend far enough for ML model integration. The RapidOcrNet discussion hits the same walls from the OCR side, so these changes would unblock both projects.

1. TextBlock has no way to carry metadata

When an ONNX layout model classifies a region as "table" or "picture" or "section_header," that label needs to travel with the TextBlock. Right now there's no place to put it.

What I did: I subclass TextBlock to carry the label and confidence score. Full source: AnnotatedTextBlock.cs

public class AnnotatedTextBlock : TextBlock
{
    public string Label { get; }
    public float Confidence { get; }

    public AnnotatedTextBlock(IReadOnlyList<TextLine> lines, string label, float confidence, string separator = "\n")
        : base(lines, separator)
    {
        Label = label;
        Confidence = confidence;
    }
}

The ONNX segmenter creates these when mapping detected regions to text blocks:

blocks.Add(new AnnotatedTextBlock(lines, detection.Label, detection.Confidence));

The problem: IPageSegmenter.GetBlocks() returns IReadOnlyList<TextBlock>, so the label gets erased at the interface boundary. Every consumer has to cast back to AnnotatedTextBlock to read the metadata.

In my data ingestion reader, I ended up adding a Func<TextBlock, string?> delegate parameter just so callers can tell the reader how to pull the label out of whatever TextBlock subclass they're using:

// PdfPigReader constructor takes this delegate
private readonly Func<TextBlock, string?>? elementTypeResolver;

Then the caller has to wire the cast (03-full-pipeline.cs line 117):

var reader = new PdfPigReader(segmenter, PdfReadingMode.TextOnly,
    elementTypeResolver: block => (block as AnnotatedTextBlock)?.Label);

And the reader uses the delegate to get the label back:

var elementType = elementTypeResolver?.Invoke(block);
if (elementType is not null)
{
    paragraph.Metadata["element_type"] = elementType;
}

It works, but it's a lot of plumbing for something that should just be on the block. It also depends on TextBlock not being sealed, which isn't something the API guarantees.

What would fix it: A metadata dictionary on TextBlock.

public class TextBlock : IBoundingBox
{
    // existing properties unchanged

    public IDictionary<string, object>? Metadata { get; set; }
}

Null by default, so there's zero cost for code that doesn't use it. With this in place, I'd delete AnnotatedTextBlock entirely and the segmenter would just do:

var block = new TextBlock(lines);
block.Metadata = new Dictionary<string, object>
{
    ["label"] = detection.Label,
    ["confidence"] = detection.Confidence
};
blocks.Add(block);

The elementTypeResolver delegate, the as AnnotatedTextBlock cast in the demo app, all of that goes away. Different integrations can attach whatever metadata they need (ONNX labels, OCR confidence, language detection, table structure) without fighting over a single subclass hierarchy.

2. Word can only be created from Letter objects

Word has one constructor:

public Word(IReadOnlyList<Letter> letters)

Letter objects come from PdfPig's PDF parsing internals. But OCR engines and ML text recognizers produce text + bounding box + orientation. They don't have Letter-level data.

This shows up in the RapidOcrNet discussion where the developer creates words from OCR output and uses a LayoutAnalysis.Pdf.Word type that doesn't exist in PdfPig yet. They have the text, orientation, and bounding box from RapidOcrNet's detection, but there's no way to create a real PdfPig Word from that.

Without an alternative, you're stuck either fabricating synthetic Letter objects (need font metadata that OCR doesn't have) or creating a parallel Word-like type that won't work with existing IPageSegmenter implementations, reading order detectors, etc.

What would fix it: A second constructor.

public Word(string text, TextOrientation textOrientation, PdfRectangle boundingBox, string? fontName = null)
{
    Text = text;
    TextOrientation = textOrientation;
    BoundingBox = boundingBox;
    FontName = fontName;
    Letters = Array.Empty<Letter>();
}

Existing constructor stays the same. The new one creates a Word with empty Letters, which is accurate because external sources don't have letter-level data. RapidOcrNet's pipeline simplifies to:

var word = new Word(wordText, orientation,
    new PdfRectangle(topLeft, topRight, bottomLeft, bottomRight));
words.Add(word);

// Now these feed directly into any IPageSegmenter
var blocks = segmenter.GetBlocks(words);

3. TextBlock properties aren't virtual (lower priority)

TextBlock.Text, BoundingBox, and TextOrientation are non-virtual get-only properties set in the constructor. If the metadata dictionary from #1 lands, this becomes less important because most use cases just need to attach data. But for scenarios like lazy-loaded text (OCR that hasn't run yet) or adjusted bounding boxes (merging overlapping detections), virtual properties would help.

This one is lower priority. Just flagging it since I'm listing the spots where I hit the type system.

What I'd do with these changes

If you're open to them, here's how my packages would adapt:

Change	What goes away	What gets simpler
Metadata on TextBlock	AnnotatedTextBlock class, elementTypeResolver delegate	Segmenter writes metadata, reader reads metadata. No casting.
Word constructor	(doesn't affect my packages directly)	Unblocks RapidOcrNet + future OCR integrations feeding into IPageSegmenter
Virtual properties	(nice to have)	Lazy text loading for VisionOnly reading mode

None of these are breaking. All additive. Happy to send a PR if the direction looks good.

[luisquintanilla]

cc: @malv007