Anything that turns a document into audio, a summary, or embeddings has the same unglamorous bottleneck: getting clean, correctly-ordered text out of the file. I underestimated this badly. Generation is a solved-ish problem you can buy. Extraction is where the engineering lives.
PDF has no concept of a paragraph, and this ruins everything
A PDF is a drawing program's output. The content stream says "place this glyph at these coordinates in this font." No paragraph, no reading order, no notion that this run is a footnote and that one a heading. Libraries handing you a page.get_text() string are guessing, usually "sort by y, then x."
That dies on the first two-column academic paper: line one of column A, then line one of column B, then line two of column A — interleaved garbage. Fed into anything narrative, it produces confident nonsense.
Detect columns geometrically before ordering anything. Project every text block's horizontal extent onto the x-axis and look for a sustained gap:
