I'm importing a txt file in to an sqlite database and then outputting those values in json format using php
json_encode
fails, complaining about illegal characters. I tracked it down to the two accented characters in the string terrains à bâtir
- this string renders fine when I open the file in Sublime but in Textedit the string is shown as terrains ‡ b‚tir
Some info about the file and its contents
file -i file.txt
tells metext/plain; charset=us-ascii
mb_detect_encoding()
on a valid string tells me it isASCII
mb_detect_encoding()
on a invalid string tells me it isUTF-8
hexdump -C file.txt | grep terrains
shows the characters as dots:
00a4eb30 7c 74 65 72 72 61 69 6e 73 20 e0 20 62 e2 74 69 ||terrains . b.ti|
cat file.txt | tail -c +1671338 | head -c 20
shows the characters as � and they appear in my sqlite GUI the same way.ns � b�tir|11111|AAA
I know it's possible to use iconv to "fix" this using the TRANSLIT or IGNORE options but then I end up with something different than what it's supposed to be.
$encoding = mb_detect_encoding($row[2]);
if($encoding !== 'ASCII') {
$enc = mb_detect_encoding($row[2]);
$converted = iconv('UTF-8', 'ASCII//IGNORE', $row[2]);
print_r($converted);
}
Using IGNORE
(obviously) outputs terrains btir
and with TRANSLIT the method complains about iconv(): Detected an illegal character in input string
My goal is to revert these characters to their proper accented form, using PHP. How can I do this? I'm guessing the hexdump output provides some clues, but I can't figure out which bytes are the problematic ones or how to fix them.
来源:https://stackoverflow.com/questions/51381963/fixing-invalid-utf8-characters