Unicode conversion issue using Python in Emacs

Question

I\'m trying to understand the difference in a bit of Python script behavior when run on the command line vs run as part of an Emacs elisp function.

The script looks

Answer 1

The Python wiki page, "PrintFails" says

When Python does not detect the desired character set of the output, it sets sys.stdout.encoding to None, and print will invoke the "ascii" codec.

It appears that when python is being run from an elisp function, it can not detect the desired character set, so it is defaulting to "ascii". So trying to print unicode is tacitly causing python to encode the unicode as ascii, which is reason for the error.

---

Replacing u\"Fooザ\" with u\"Foo\\u30b6\" seems to work:

(defun mytest ()
  (interactive)
  (shell-command-on-region (point)
         (point) "LC_ALL=\"en_US.UTF-8\" python -c 's = u\"Foo\\u30b6\"; print s.encode(\"utf8\",\"replace\")'" nil t))

C-x C-e M-x mytest

yields

Fooザ