strlen() and UTF-8 encoding

Question

Assuming UTF-8 encoding, and strlen() in PHP, is it possible that this string has a length of 4?

I\'m only interested to know about strlen(), not other functions

Answer 1

Many UTF-8 characters take several bytes instead of one. That's how UTF-8 is constructed (That's how you can have so many characters in a single set).

Try mb_strlen() instead.

Answer 2

It's likely that at some point between the preparation of the question and your reading of it some process has mangled non-ASCII characters in it, so the question was originally about some string with 4 characters in it.

The sequence � is obtained when you encode the replacement character U+FFFD (�) in UTF-8 and interpret the result in latin1. This character is used as a replacement for byte sequences that don't encode any character when reading text from a file, for example. What has happened is likely this:

The original question, stored in a latin1 text file, had: $1¢2 (you can replace ¢ with any non-ASCII character)

The file was read by a program that used UTF-8. Since the byte corresponding to ¢ could not be interpreted, the program substituted it and read the text $1�2. This text was then written out using UTF-8, resulting in $1\xEF\xBF\xBD2 in the file.

Then some third program comes that reads the file in latin1, and shows $1�2.

Answer 3

need to use Multibyte String Function mb_strlen() like:

mb_strlen($string, 'UTF-8');

Answer 4

The string you posted is six character long: $1�2 (dollar sign, digit one, lowercase i with diaeresis, upside-down question mark, one half fraction, digit two)

If strlen() was called with a UTF-8 representation of that string, you would get a result of nine (probably, though there are multiple representations with different lengths).

However, if we were to store that string as ISO 8859-1 or CP1252 we would have a six byte long sequence that would be legal as UTF-8. Reinterpreting those 6 bytes as UTF-8 would then result in 4 characters: $1�2 (dollar sign, digit one, Unicode Replacement Character, digit 2). That is, the UTF-8 encoding of the single character '�' is identical to the ISO-8859-1 encoding of the three characters "�".

The replacement character often gets inserted when a UTF-8 decoder reads data that's not valid UTF-8 data.

It appears that the original string was processed through multiple layers of misinterpretation; by the use of a UTF-8 decoder on non-UTF-8 data (producing $1�2), and then by whatever you used to analyze that data (producing $1�2).

Answer 5

how about using mb_strlen() ?

http://lt.php.net/manual/en/function.mb-strlen.php

But if you need to use strlen, its possible to configure your webserver by setting mbstring.func_overload directive to 2, so it will automatically replace using of strlen to mb_strlen in your scripts.

Answer 6

No.

I'll use a proof by contradiction.

strlen counts bytes, so with a strlen of 4, there would need to be exactly 4 bytes in that string.

UTF8 encoding needs at least 1 byte per character.

We have established that:

1. there are 4 bytes
2. a character is represented by no less than 1 byte

...yet, we have 6 characters....which is a contradiction. So, no.

However, what's not totally clear is which character set the displaying software(eg, the web browser) is using to intepret the string. It could use some uncommon encoding scheme where a character can be represented by less than 8 bits. If this were the case, then 4 bytes could display as 6 characters. So, the string could be utf8, but the browser could decide to interpret it as, say, some 5 bit character set.