Garbage extracted from pdf instead of text

[kobruleht]

To reproduce, extract text from attached pdf using

      var p = new ParsingOptions() {
        UseLenientParsing = true,
        SkipMissingFonts = true,
        FilterProvider = SkiaRenderingFilterProvider.Instance,
        UseActualText = true
      };
        using PdfDocument document = PdfDocument.Open(stream,p); 
        document.AddSkiaPageFactory();

Observed:

Garbage characters extracted instead of pdf text.

dgm-woodexpo-garbage.pdf

[BobLd]

@kobruleht thanks for opening the issue. Can you provide the name of a pdf reader or library that is able to extract the correct text?

If you can't find one, it might be because the document is corrupted and you'd need to use OCR

[kobruleht]

Chrome shows it properly in browser window. Opening it in pdfpig does not throw exception. So it is assumed to be correct.

There should be propery or exception in pdfpig which indicates that pdf cannot converted to text using pdfpig.
GetPageAsPng() should convert it to image.

[BobLd]

@kobruleht several aspects to the problem:

• When I said 'corrupted', in this case I should have said 'malformed', i.e. the pdf document is valid, but the mapping of the rendered letter to its actual value is not correct (renders correctly, but mapped text value is not correct). This wrong mapping would not throw an exception, this is just how the pdf document was built.
• If this is the case, pdfpig has no way to know the mapping is not correct, no exception thrown possible.
• Can GetPageAsPng() properly render the page? If not, what error / output are you getting?

To check if point 1 is what's going on, open the pdf in a pdf reader and try to copy/paste the text. What output are you getting? Different reader might behave differently, so maybe try several. If none can copy/paste the text correctly, then chances are that the letter mapping is wrong

[kobruleht]

iTextSharp allows to fix this Mojibake issue using code

using System;
using System.IO;
using System.Text;
using iTextSharp.text.pdf;

public class PdfCmapRepairer
{
    public static void ReplaceCorruptedCMap(string sourcePdf, string outputPdf)
    {
        // 1. Initialize the reader to open the corrupt document
        using (PdfReader reader = new PdfReader(sourcePdf))
        {
            // 2. Loop through all indirect objects in the PDF file
            for (int i = 1; i <= reader.XrefSize; i++)
            {
                PdfObject pdfObj = reader.GetPdfObject(i);
                
                // Look specifically for Dictionary objects (Fonts are structured as dictionaries)
                if (pdfObj != null && pdfObj.IsDictionary())
                {
                    PdfDictionary dict = (PdfDictionary)pdfObj;
                    
                    // 3. Identify if the dictionary represents a Font
                    if (PdfName.FONT.Equals(dict.GetAsName(PdfName.SUBTYPE)) || 
                        PdfName.FONT.Equals(dict.GetAsName(PdfName.TYPE)))
                    {
                        // Generate a clean, basic ToUnicode CMap template
                        byte[] cleanCMapBytes = Encoding.ASCII.GetBytes(GetValidCMapTemplate());

                        // 4. Create a fresh, uncorrupted PDF Stream object containing the new CMap
                        PdfStream newCMapStream = new PdfStream(cleanCMapBytes);
                        newCMapStream.FlateCompress(); // Compress it to comply with standard PDF structures

                        // 5. Inject/Overwrite the existing /ToUnicode key with our healthy stream
                        dict.Put(PdfName.TOUNICODE, newCMapStream);
                    }
                }
            }

            // 6. Persist the modifications to a brand new PDF file via PdfStamper
            using (FileStream fs = new FileStream(outputPdf, FileMode.Create, FileAccess.Write))
            {
                using (PdfStamper stamper = new PdfStamper(reader, fs))
                {
                    // PdfStamper automatically rewrites the cross-reference tables cleanly
                }
            }
        }
    }

    /// <summary>
    /// Generates a valid standard PostScript CMap template mapping generic character codes sequentially.
    /// Modify the character ranges below if your corrupt document utilizes custom multi-byte CID mappings.
    /// </summary>
    private static string GetValidCMapTemplate()
    {
        StringBuilder cmap = new StringBuilder();
        cmap.AppendLine("/CIDInit /ProcSet findresource begin");
        cmap.AppendLine("12 dict begin");
        cmap.AppendLine("begincmap");
        cmap.AppendLine("/CIDSystemInfo << /Registry (Adobe) /Ordering (UCS) /Supplement 0 >> def");
        cmap.AppendLine("/CMapName /Custom-ToUnicode def");
        cmap.AppendLine("/CMapType 2 def");
        cmap.AppendLine("1 begincodespacerange");
        cmap.AppendLine("<0000> <FFFF>");
        cmap.AppendLine("endcodespacerange");
        cmap.AppendLine("1 beginbfrange");
        cmap.AppendLine("<0000> <FFFF> <0000>"); // Straight 1:1 sequence mapping fallback
        cmap.AppendLine("endbfrange");
        cmap.AppendLine("endcmap");
        cmap.AppendLine("CMapName currentdict /CMap defineresource pop");
        cmap.AppendLine("end");
        cmap.AppendLine("end");
        return cmap.ToString();
    }
}

Pdfpig does not allow to modify or replace the internal ToUnicode CMap array directly.
Letter objects should pre-processed fro this during processing:

using System;
using System.Text;
using UglyToad.PdfPig;
using UglyToad.PdfPig.Content;

public static class PdfCMapRepairer
{
    public static string ExtractAndRepairText(string filePath)
    {
        StringBuilder repairedText = new StringBuilder();

        using (PdfDocument document = PdfDocument.Open(filePath))
        {
            foreach (Page page in document.GetPages())
            {
                foreach (Word word in page.GetWords())
                {
                    string cleanWord = RepairWordCMap(word);
                    repairedText.Append(cleanWord + " ");
                }
                repairedText.AppendLine();
            }
        }

        return repairedText.ToString();
    }

    private static string RepairWordCMap(Word word)
    {
        StringBuilder fixedWord = new StringBuilder();

        foreach (Letter letter in word.Letters)
        {
         TODO. how to apply standard mapping like in iTextSharp example
                fixedWord.Append( ??? ); // Inject fixed mapping manually 
        }

        return fixedWord.ToString();
    }
}

It looks like it is not possible to find standard mapping for every character separately in Pdfpig.
PdfPdi should allow to replace whole cmap with standard mapping like iTextSharp allows:

                        // Generate a clean, basic ToUnicode CMap template
                        byte[] cleanCMapBytes = Encoding.ASCII.GetBytes(GetValidCMapTemplate());

                        // 4. Create a fresh, uncorrupted PDF Stream object containing the new CMap
                        PdfStream newCMapStream = new PdfStream(cleanCMapBytes);
                        newCMapStream.FlateCompress(); // Compress it to comply with standard PDF structures

                        // 5. Inject/Overwrite the existing /ToUnicode key with our healthy stream
                        dict.Put(PdfName.TOUNICODE, newCMapStream);

GetPageAsPng() renders it correctly but there should be direct solution.