Removing invalid/incomplete multibyte characters

Question

I'm having some issues using the following code on user input:

htmlentities($string, ENT_COMPAT, 'UTF-8');

When an invalid multibyte character is detected PHP throws a notice:

PHP Warning: htmlentities(): Invalid multibyte sequence in argument in /path/to/file.php on line 123

My first thought was to supress the error, but this is slow and poor practice: http://derickrethans.nl/five-reasons-why-the-shutop-operator-should-be-avoided.html

My second thought was to use the ENT_IGNORE flag, but even the PHP manual suggests not to use this:

Silently discard invalid code unit sequences instead of returning an empty string. Using this flag is discouraged as it » may have security implications.

A bit of further reason led me to the following piece of code:

    // detect encoding
$encoding =  mb_detect_encoding($query);
if($encoding != 'UTF-8') {
    $query = mb_convert_encoding($query, 'UTF-8', $encoding);
} else {
    // strip out invalid utf8 sequences
    $query = iconv('UTF-8', 'UTF-8//IGNORE', $query);
}

Unfortunately iconv also throws an E_NOTICE when it removes/ignores invalid characters:

If you append the string //TRANSLIT to out_charset transliteration is activated. This means that when a character can't be represented in the target charset, it can be approximated through one or several similarly looking characters. If you append the string //IGNORE, characters that cannot be represented in the target charset are silently discarded. Otherwise, str is cut from the first illegal character and an E_NOTICE is generated.

So I'm basically out of options here. I'd rather use a tried and tested library to handle this kind of stuff than attempting it with a few of the regular expression based solutions I've seen floated around.

So that leads me to my final question: How can I remove invalid multibyte characters, efficiently, securely, without notices/warnings/errors?

Answer 1

How can I remove invalid multibyte characters, efficiently, securely, without notices/warnings/errors?

Well, as you already have outlined in your question on your own (or at least linked), deleting the invalid byte sequence(s) is not an option.

Instead it should be probably replaced with the replacement character U+FFFD. As of PHP 5.4.0 you can make use of the ENT_SUBSTITUTE flag for htmlentities. That's probably most safe if you don't want to reject the string.

iconv will always give you warning in recent PHP versions if not even deleting the whole string. So it does not look like a good alternative for you.

Answer 2

iconv('UTF-8', "ISO-8859-1//IGNORE", $string);

worked extremely well for me. Doesn't seem to generate any notice.