Pdf Miner returns weird letters/characters

问题

I am using pdfminer with python 3 and I get weird letters in the text that is recovered from the pdf.

For instance, I get signiﬁcantinstead of significant (notice that the letters f and I are merged into one).

I have no idea why this is happening. This is the code I am using.

from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter
from pdfminer.converter import TextConverter
from pdfminer.layout import LAParams
from pdfminer.pdfpage import PDFPage
from io import StringIO
from nltk.tokenize import sent_tokenize


def convert_pdf_to_txt(path):
    rsrcmgr = PDFResourceManager()
    retstr = StringIO()
    codec = 'utf-8'
    laparams = LAParams()
    device = TextConverter(rsrcmgr, retstr, codec=codec, laparams=laparams)
    fp = open(path, 'rb')
    interpreter = PDFPageInterpreter(rsrcmgr, device)
    password = ""
    maxpages = 0
    caching = True
    pagenos=set()

    for page in PDFPage.get_pages(fp, pagenos, maxpages=maxpages, password=password,caching=caching, check_extractable=True):
        interpreter.process_page(page)

    text = retstr.getvalue()

    sentences = sent_tokenize(text)

    for s in sentences:
        print(s)
        print("\n\n")

My only guess so far is that it may have to do with the encoding, but it seems that there is no way to retrieve the encoding of a pdf

回答1:

PDFminer is working correctly. The character in question is the Unicode character U+FB01, the fi ligature.

Add a line to replace ﬁ with fi to your code:

for s in sentences:
    s = s.replace ('ﬁ', 'fi')
    print (s)

There is one other very common – and purely typographic(*) – ligature defined in Unicode: U+FB02, the fl ligature; treat this the same:

    s = s.replace ('ﬂ', 'fl')

and a couple of others in the Alphabetic Presentation block, which you might as well include too.

(*) Do not make the mistake to change æ to ae and œ to oe. These are not 'purely typographic ligatures' but valid characters on their own.

来源：https://stackoverflow.com/questions/52863575/pdf-miner-returns-weird-letters-characters

标签

python

python-3.x

pdf

text

pdfminer