How to replace Unicode characters with ASCII

Question

I have the following command to replace Unicode characters with ASCII ones.

sed -i \'s/Ã/A/g\'

The problem is Ã isn\'t recognized

Answer 1

You can use iconv:

iconv -f utf-8 -t ascii//translit

Answer 2

It is possible to use hex values in "sed".

echo "Ã" | hexdump -C
00000000  c3 83 0a                                          |...|
00000003

Ok, that character is two byte combination "c3 83". Let's replace it with single byte "A":

echo "Ã" |sed 's/\xc3\x83/A/g'
A

Explanation: \x indicates for "sed" that a hex code follows.

Answer 3

Try setting LANG=C and then run it over the Unicode range:
echo "hi ☠ there ☠" | LANG=C sed "s/[\x80-\xFF]//g"

Answer 4

There is also uconv, from ICU.

Examples:

• uconv -x "::NFD; [:Nonspacing Mark:] > ; ::NFC;": to remove accents
• uconv -x "::Latin; ::Latin-ASCII;": for a transliteration latin/ascii
• uconv -x "::Latin; ::Latin-ASCII; ([^\x00-\x7F]) > ;": for a transliteration latin/ascii and removal of remaining code points > 0x7F
• ...

echo "À l'école ☠" | uconv -x "::Latin; ::Latin-ASCII; ([^\x00-\x7F]) > ;" gives: A l'ecole