UnicodeEncodeError: 'charmap' codec can't encode characters

Question

I\'m trying to scrape a website, but it gives me an error.

I\'m using the following code:

import urllib.request
from bs4 import BeautifulSoup

get =          


        

Answer 1

I was getting the same UnicodeEncodeError when saving scraped web content to a file. To fix it I replaced this code:

with open(fname, "w") as f:
    f.write(html)

with this:

import io
with io.open(fname, "w", encoding="utf-8") as f:
    f.write(html)

Using io gives you backward compatibility with Python 2.

If you only need to support Python 3 you can use the builtin open function instead:

with open(fname, "w", encoding="utf-8") as f:
    f.write(html)