how to deal with unicode in mako?

问题

I constantly get this error using mako:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 6: ordinal not in range(128)

I've told mako I'm using unicode in any possible way:

    mylookup = TemplateLookup(
        directories=['plugins/stl/templates'],
        input_encoding='utf-8',
        output_encoding='utf-8',
        default_filters=['decode.utf8'],
        encoding_errors='replace')

    self.template = Template(self.getTemplate(), lookup=mylookup,
        module_directory=tempfile.gettempdir(),
        input_encoding='utf-8',
        output_encoding='utf-8',
        default_filters=['decode.utf8'],
        encoding_errors='replace')

    html = self.template.render_unicode(data=self.stuff)

All my template files starts with:

## -*- coding: utf-8 -*-

and, inside them, all costant strings are prefixed with "u". I know the self.stuff parameter contains unicode strings, but the way I instantiate the mako objects should take care of it (otherwise what those arguments are good for?). Is there anything I forgot to do?

One more question: what's the point of encoding_errors='replace'?

=EDIT= I left only a single unicode string and this is the traceback:

Traceback (most recent call last):
  File "C:\My Dropbox\src\flucso\src\plugins\stl\main.py", line 240, in updateView
    flags=self.makoflags)
  File "C:\Python26\lib\site-packages\mako-0.3.4-py2.6.egg\mako\template.py", line 198, in render_unicode
    as_unicode=True)
  File "C:\Python26\lib\site-packages\mako-0.3.4-py2.6.egg\mako\runtime.py", line 403, in _render
    _render_context(template, callable_, context, *args, **_kwargs_for_callable(callable_, data))
  File "C:\Python26\lib\site-packages\mako-0.3.4-py2.6.egg\mako\runtime.py", line 434, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "C:\Python26\lib\site-packages\mako-0.3.4-py2.6.egg\mako\runtime.py", line 457, in _exec_template
    callable_(context, *args, **kwargs)
  File "memory:0x41317f0", line 89, in render_body
  File "C:\Python26\lib\site-packages\mako-0.3.4-py2.6.egg\mako\runtime.py", line 278, in <lambda>
    return lambda *args, **kwargs:callable_(self.context, *args, **kwargs)
  File "FriendFeed_mako", line 49, in render_inlist_entry
  File "C:\Python26\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u263c' in position 8: ordinal not in range(128)

回答1:

finally I saved my templates in unicode, actually (I guess) utf-16 instead of utf-8. their size on disk doubled and mako started complaining about a "CompileException("Unicode decode operation of encoding 'utf-8' bla bla", so I changed the first line in all of them in:

## -*- coding: utf-16 -*-

and removed all the ".decode('utf-8')" - constant strings are still prefixed by "u".

the initializations in python now are:

mylookup = TemplateLookup(
    directories=['plugins/stl/templates'],
    input_encoding='utf-16',
    output_encoding='utf-16',
    encoding_errors='replace')

self.template = Template(self.getTemplate(), lookup=mylookup,
    module_directory=tempfile.gettempdir(),
    input_encoding='utf-16',
    output_encoding='utf-16',
    encoding_errors='replace')

it works now. looks like utf-8 was the wrong choice (or my inability to save the templates in utf-8), but I can't explain why it worked from eclipse/pydev.

回答2:

For the sake of googlers:

Mako raises an exception mako.exceptions.CompileException: Unicode decode operation of encoding 'ascii' failed in file, etc., when your template file comprehends non-ascii characters, and when the Unicode BOM isn't written into the file. You need to manually add the BOM (this is not made automatically, at least in my text editor), so that this:

$file test.htm
test.htm: HTML document, UTF-8 Unicode text

becomes this:

$file test.htm
test.htm: HTML document, UTF-8 Unicode (with BOM) text

回答3:

None of these suggestions (including the accepted answer) work in all cases, specifically where the mako template renders content (eg. ${value | n}) where the value contains non ascii characters.

This is because by default mako wraps unicode(foo) around any values in the generated compiled templates, which will still result in:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 0: ordinal not in range(128)

The only sure fire way of making mako handle unicode in python2 is to replace the default ('unicode') handler, like this:

def handle_unicode(value):
    if isinstance(value, basestring):
        return unicode(value.decode('ascii', errors='ignore'))
    return unicode(value)


...    

lookup = TemplateLookup(
    directories=[self._root.template_path],
    imports=['from utils.view import handle_unicode'],
    default_filters=["handle_unicode"]
)

...

template = self._lookup.get_template(self.template())
rtn = template.render(request=self.request)

来源：https://stackoverflow.com/questions/3333550/how-to-deal-with-unicode-in-mako