📄Fixing Unprintable Scanned PDFs Without Breaking the Invisible Text

This article was co-authored with generative AI. Facts have been checked against public documentation where feasible, but errors may remain. Please verify primary sources before relying on this for important decisions.

For a set of PDFs from a historical-source collection (32 files, about 870MB total) that overlaid invisible OCR text on scanned images, I was asked to address two things.

1. Page sizes differed from page to page
2. Underpowered PCs sometimes couldn't print them

I carried out the work to fix these in bulk, but partway through I hit a pitfall: "when you helpfully re-compress the images, the invisible text (OCR layer) is slightly chipped away." This article records the diagnosis and the conversion method that got the job done without breaking a single character of text.

First, Check the State

You can learn a PDF's basic properties with pdfinfo and pdfimages (poppler). For variation in page sizes, list all page sizes and count the distinct kinds.

# Count the distinct kinds of page sizes
pdfinfo -f 1 -l 9999 input.pdf | grep "Page.*size:" \
  | awk '{print $4" x "$6}' | sort | uniq -c

 338 380.16 x 576
  24 396.72 x 594.72
  13 515.52 x 729.36
   1 419.58 x 595.38

Four distinct sizes were mixed within a single PDF. The body, front matter, the errata at the end, and pages added at the very end each seem to have a different history of original scanning and editing.

The image compression method can be found with pdfimages -list.

pdfimages -list input.pdf | tail -n +3 \
  | awk '{print $6, $9, $13"dpi"}' | sort | uniq -c | sort -rn

 338 gray jbig2 400dpi
  35 gray ccitt 200dpi
   2 gray jpeg 200dpi

Most of the body was compressed with JBIG2. JBIG2 is a method that can store bilevel images at a very high compression ratio, and it makes files smaller (this book is 376 pages at about 16MB). On the other hand, its decoding computation cost is high, and depending on the environment a printer driver or viewer cannot fully process it and printing stalls. The reported reason that "it can't print because the file is large" was, I suspected, not the file's byte size but this decoding load.

Surveying all 32 files revealed a clear pattern.

• Only some files use JBIG2 (9 of 32)
• The rest use image (Flate), jpeg, or ccitt, with no JBIG2
• Nearly every file mixes 2–5 page sizes (one file had 138 kinds)
• Every file has invisible text (an OCR layer)

The plan settled on two things: "unify page sizes across all files" and "re-compress only the 9 JBIG2 files to a more printer-friendly method." However, because every file has an OCR layer, not breaking the text was an absolute requirement.

Pitfall: Ghostscript Also Rebuilds the Text

At first I tried to do both the size change and the image re-compression at once with Ghostscript's pdfwrite. It is a commonly used method.

gs -o output.pdf -sDEVICE=pdfwrite -dCompatibilityLevel=1.6 \
  -dDEVICEWIDTHPOINTS=595 -dDEVICEHEIGHTPOINTS=842 \
  -dFIXEDMEDIA -dPDFFitPage input.pdf

The pages line up and JBIG2 changes to CCITT. However, comparing the invisible text's character count before and after conversion, it had visibly decreased.

# Compare character counts with whitespace removed
pdftotext input.pdf  - | tr -d '[:space:]' | wc -m   # 498335
pdftotext output.pdf - | tr -d '[:space:]' | wc -m   # 484935

About 2.7% (13,000 characters) were lost. What decreased was mainly words with abbreviation marks seen in older Western-language spelling (ligatures and special glyphs). Because Ghostscript's pdfwrite rebuilds the entire PDF, it seems that during the font subsetting and re-encoding process, some information for special glyphs not contained in the embedded font is dropped.

Even if the visible image becomes cleaner, having the text layer used for search and citation altered is not acceptable for a historical source. I diagnose which step is the cause.

# Page-size unification only (done with pypdf, not Ghostscript)
pdftotext normalized_by_pypdf.pdf - | tr -d '[:space:]' | wc -m   # 498336

With size unification alone via pypdf, the character count almost perfectly matched the original (498335) (a difference of 1 character). In other words, what breaks the text is Ghostscript's rebuild, and page scaling itself is harmless.

The reasoning is simple: scaling a page only inserts a single coordinate transformation matrix, so it does not touch the text-drawing instructions (which characters go where). Ghostscript, on the other hand, rebuilds both the content stream and the fonts.

Re-Compress Only the Images Without Touching the Text

So I settled on the following two-stage structure. Neither touches the text layer.

flowchart LR
  A[Original PDF] --> B{Has JBIG2 images?}
  B -->|Yes| C["pikepdf<br/>re-compress only images to CCITT"]
  B -->|No| D[As-is]
  C --> E["pypdf<br/>unify page sizes"]
  D --> E
  E --> F[Converted PDF]

1. Replace Only the Image Objects with pikepdf

pikepdf (the Python bindings for qpdf) can rewrite individual objects inside a PDF without touching the others. It decodes only the JBIG2 images and re-encodes them to CCITT G4 (a FAX-derived bilevel compression; mature and lightweight to decode in nearly any environment), writing them back. It never reads or writes the text content streams at all.

Decoding JBIG2 requires jbig2dec (brew install jbig2dec). The raw CCITT G4 stream is obtained by first saving to TIFF (group4) with Pillow and extracting its strip. At this point, if the G4 stream is split into multiple strips, a naive concatenation breaks it, so the key point is to enlarge the strip size so it is always a single strip.

import io
import pikepdf
from pikepdf import Name, Dictionary, PdfImage
from PIL import Image, TiffImagePlugin
from PIL.TiffImagePlugin import STRIPOFFSETS, STRIPBYTECOUNTS

# G4 breaks on concatenation if split into multiple strips. Force a single strip.
TiffImagePlugin.STRIP_SIZE = 2 ** 31


def pil_1bit_to_ccitt_g4(im):
    if im.mode != "1":
        im = im.convert("1")
    buf = io.BytesIO()
    im.save(buf, format="TIFF", compression="group4")
    data = buf.getvalue()
    tif = Image.open(io.BytesIO(data))
    offsets = tif.tag_v2[STRIPOFFSETS]
    counts = tif.tag_v2[STRIPBYTECOUNTS]
    assert len(offsets) == 1, f"strip split: {len(offsets)}"
    return data[offsets[0]:offsets[0] + counts[0]]


def recompress_jbig2(src, dst):
    pdf = pikepdf.open(src)
    n = 0
    for page in pdf.pages:
        xobjs = (page.get("/Resources") or {}).get("/XObject") or {}
        for _, xobj in list(xobjs.items()):
            if xobj.get("/Subtype") != Name("/Image"):
                continue
            if "JBIG2" not in str(xobj.get("/Filter")):
                continue
            w, h = int(xobj.Width), int(xobj.Height)
            im = PdfImage(xobj).as_pil_image()          # decode with jbig2dec
            raw = pil_1bit_to_ccitt_g4(im)
            xobj.write(
                raw,
                filter=Name("/CCITTFaxDecode"),
                decode_parms=Dictionary(K=-1, Columns=w, Rows=h, BlackIs1=True),
            )
            n += 1
    pdf.save(dst)
    pdf.close()
    return n

K=-1 denotes G4 (two-dimensional encoding). Polarity (BlackIs1) requires care. CCITTFaxDecode's default BlackIs1=False means PDF's usual convention "0 bit = black, 1 bit = white." However, the raw stream obtained by encoding the bilevel image from PdfImage with Pillow's group4 came out with black and white inverted under this convention. So I specify BlackIs1=True on the CCITTFaxDecode side to align the orientation.

Polarity can invert depending on the combination of the decode path and encoding library used, so do not decide the value by assumption. To be safe, always render a few pages after conversion and confirm mechanically that they are black text on a white background (that the average brightness is on the white side). In fact, in the first draft of this article I got the polarity wrong, and every scanned page of the output was inverted to white on black. It is easy to miss with visual inspection alone, and I only caught it by also using a numeric brightness check.

2. Unify Page Sizes with pypdf

Scale each page to the target size (e.g., A4), preserving the aspect ratio, and center it. Because add_transformation only inserts a coordinate transformation matrix, the image and the invisible text move together at the same scale, and their positional relationship (i.e., where the OCR lands) is preserved.

from pypdf import PdfReader, PdfWriter, Transformation
from pypdf.generic import RectangleObject


def normalize(src, dst, W, H):
    reader, writer = PdfReader(src), PdfWriter()
    for page in reader.pages:
        box = page.mediabox
        w, h = float(box.width), float(box.height)
        s = min(W / w, H / h)                      # preserve aspect ratio
        tx = (W - w * s) / 2 - float(box.left) * s  # center
        ty = (H - h * s) / 2 - float(box.bottom) * s
        page.add_transformation(Transformation().scale(s, s).translate(tx, ty))
        rect = RectangleObject([0, 0, W, H])
        page.mediabox = page.cropbox = rect
        writer.add_page(page)
    with open(dst, "wb") as f:
        writer.write(f)

Confirm the Text Really Is Intact

"Same character count" alone is not enough. When a page shape changes, pdftotext changes where lines wrap, and word breaks and how whitespace appears change too. In fact, taking a word-level diff before and after conversion shows many differences, but their substance was apparent differences due to wrapping — like (bronckados)金 being split into (bronck and ados)金.

So I compare the frequency of each individual character rather than words. If the text-drawing instructions have not been touched, the multiset of characters (which character appears how many times) should match almost perfectly.

import subprocess, collections

def char_freq(path):
    out = subprocess.run(["pdftotext", path, "-"], capture_output=True).stdout
    text = "".join(out.decode("utf-8", "replace").split())  # remove all whitespace
    return collections.Counter(text)

a, b = char_freq("input.pdf"), char_freq("output.pdf")
diff = {c: (a[c], b[c]) for c in set(a) | set(b) if a[c] != b[c]}
print("character types with differences:", len(diff))
print("total of differences:", sum(abs(x - y) for x, y in diff.values()))

Even for the file with the largest difference, out of about 280,000 characters, the differences were only 26 character types totaling 29 characters (0.010%). Moreover, they were scattered across the whole text by ±1 per character, such as "の" going from 13004 to 13003, which is within the error range of pdftotext reconstructing coordinates from a changed page layout (the file with the most text — about 550,000 characters — had a difference of 0). This is a completely different pattern from the systematic dropout seen in the Ghostscript version, where "specific special glyphs disappear in a lump of 2.7%." With this level of difference, the text layer can be judged effectively intact.

For reference, here is a comparison of the two methods.

	Page unification	JBIG2→CCITT	OCR text
Ghostscript pdfwrite	possible	possible	about 2.7% systematically dropped
pikepdf + pypdf	possible	possible (9 files)	effectively unchanged (at most 0.010% reconstruction error, ±1 per character)

Summary

• A scanned PDF's "can't print" is sometimes caused not by file size but by decoding load such as JBIG2. Check the compression method with pdfimages -list
• For PDFs with invisible text (an OCR layer), Ghostscript's pdfwrite also rebuilds the text, so some special glyphs may be lost
• Page scaling only inserts a coordinate transformation and does not break the text (pypdf). Image re-compression, too, leaves the text intact if you replace only the image objects with pikepdf
• Do verification not by character count or word diff but by per-character frequency, so you can distinguish apparent differences from wrapping from real dropout
• When re-compressing bilevel images, it's easy to get polarity (black/white inversion) wrong. Visual inspection can miss it, so after conversion measure each page's average brightness and confirm mechanically that it is on the white (bright) side

The same toolset (pikepdf / pypdf / poppler) can also be used for merging, splitting, rotating, and lightening while keeping the invisible text. The idea of "don't touch the contents of images or text, and replace only the objects you need" seems to be the safe-side basis when handling OCR'd historical-source PDFs.