How to read PDF files which are in asian languages (Chinese, Japanese, Thai, etc.) and store in a string in python

Question

I am using PyPDF2 to read PDF files in python. While it works well for languages in English and European languages (with alphabets in english), the library fails to read Asian languages like Japanese and Chinese. I tried encode('utf-8'), decode('utf-8') but nothing seems to work. It just prints a blank string on extraction of the text.

I have tried other libraries like textract and PDFMiner but no success yet.

When I copy the text from PDF and paste it on a notebook, the characters turn into some random format text (probably in a different encoding).

def convert_pdf_to_text(filename):
    text = ''
    pdf = PyPDF2.PdfFileReader(open(filename, "rb"))
    if pdf.isEncrypted:
        pdf.decrypt('')
    for page in pdf.pages:
        text = text + page.extractText()
    return text

Can anyone point me in the right direction?

Answer 1

I too faced similar issue. I could resolve it by using 'tika-python' library.

import tika
tika.initVM()
from tika import parser
parsed = parser.from_file('fileName.pdf')
print(parsed["metadata"])
print(parsed["content"])

You can find more information about the library over here