Multibyte trim in PHP? | 易学教程

问题

Apparently there's no mb_trim in the mb_* family, so I'm trying to implement one for my own.

I recently found this regex in a comment in php.net:

/(^\s+)|(\s+$)/u

So, I'd implement it in the following way:

function multibyte_trim($str)
{
    if (!function_exists("mb_trim") || !extension_loaded("mbstring")) {
        return preg_replace("/(^\s+)|(\s+$)/u", "", $str);
    } else {
        return mb_trim($str);
    }
}

The regex seems correct to me, but I'm extremely noob with regular expressions. Will this effectively remove any Unicode space in the beginning/end of a string?

回答1:

The standard trim function trims a handful of space and space-like characters. These are defined as ASCII characters, which means certain specific bytes from 0 to 0100 0000.

Proper UTF-8 input will never contain multi-byte characters that is made up of bytes 0xxx xxxx. All the bytes in proper UTF-8 multibyte characters start with 1xxx xxxx.

This means that in a proper UTF-8 sequence, the bytes 0xxx xxxx can only refer to single-byte characters. PHP's trim function will therefore never trim away "half a character" assuming you have a proper UTF-8 sequence. (Be very very careful about improper UTF-8 sequences.)

The \s on ASCII regular expressions will mostly match the same characters as trim.

The preg functions with the /u modifier only works on UTF-8 encoded regular expressions, and /\s/u match also the UTF8's nbsp. This behaviour with non-breaking spaces is the only advantage to using it.

If you want to replace space characters in other, non ASCII-compatible encodings, neither method will work.

In other words, if you're trying to trim usual spaces an ASCII-compatible string, just use trim. When using /\s/u be careful with the meaning of nbsp for your text.

Take care:

  $s1 = html_entity_decode(" Hello &#160; "); // the NBSP
  $s2 = " 𩸽 exotic test ホ 𩸽 ";

  echo "\nCORRECT trim: [". trim($s1) ."], [".  trim($s2) ."]";
  echo "\nSAME: [". trim($s1) ."] == [". preg_replace('/^\s+|\s+$/','',$s1) ."]";
  echo "\nBUT: [". trim($s1) ."] != [". preg_replace('/^\s+|\s+$/u','',$s1) ."]";

  echo "\n!INCORRECT trim: [". trim($s2,'𩸽 ') ."]"; // DANGER! not UTF8 safe!
  echo "\nSAFE ONLY WITH preg: [". 
       preg_replace('/^[𩸽\s]+|[𩸽\s]+$/u', '', $s2) ."]";

回答2:

I don't know what you're trying to do with that endless recursive function you're defining, but if you just want a multibyte-safe trim, this will work.

function mb_trim($str) {
  return preg_replace("/(^\s+)|(\s+$)/us", "", $str); 
}

回答3:

This version supports the second optional parameter $charlist:

function mb_trim ($string, $charlist = null) 
{   
    if (is_null($charlist)) {
        return trim ($string);
    } 

    $charlist = str_replace ('/', '\/', preg_quote ($charlist));
    return preg_replace ("/(^[$charlist]+)|([$charlist]+$)/us", '', $string);
}

Does not support ".." for ranges though.

回答4:

You can also trim non-ascii compatible spaces (non-breaking space for example) on UTF-8 strings with preg_replace('/^\p{Z}+|\p{Z}+$/u','',$str);

\s will only match "ascii compatible" space character even with the u modifier.
but \p{Z} will match all known unicode space characters

回答5:

Ok, so I took @edson-medina's solution and fixed a bug and added some unit tests. Here's the 3 functions we use to give mb counterparts to trim, rtrim, and ltrim.

////////////////////////////////////////////////////////////////////////////////////
//Add some multibyte core functions not in PHP
////////////////////////////////////////////////////////////////////////////////////
function mb_trim($string, $charlist = null) {
    if (is_null($charlist)) {
        return trim($string);
    } else {
        $charlist = preg_quote($charlist, '/');
        return preg_replace("/(^[$charlist]+)|([$charlist]+$)/us", '', $string);
    }
}
function mb_rtrim($string, $charlist = null) {
    if (is_null($charlist)) {
        return rtrim($string);
    } else {
        $charlist = preg_quote($charlist, '/');
        return preg_replace("/([$charlist]+$)/us", '', $string);
    }
}
function mb_ltrim($string, $charlist = null) {
    if (is_null($charlist)) {
        return ltrim($string);
    } else {
        $charlist = preg_quote($charlist, '/');
        return preg_replace("/(^[$charlist]+)/us", '', $string);
    }
}
////////////////////////////////////////////////////////////////////////////////////

Here's the unit tests I wrote for anyone interested:

public function test_trim() {
    $this->assertEquals(trim(' foo '), mb_trim(' foo '));
    $this->assertEquals(trim(' foo ', ' o'), mb_trim(' foo ', ' o'));
    $this->assertEquals('foo', mb_trim(' Åfooホ ', ' Åホ'));
}

public function test_rtrim() {
    $this->assertEquals(rtrim(' foo '), mb_rtrim(' foo '));
    $this->assertEquals(rtrim(' foo ', ' o'), mb_rtrim(' foo ', ' o'));
    $this->assertEquals('foo', mb_rtrim('fooホ ', ' ホ'));
}

public function test_ltrim() {
    $this->assertEquals(ltrim(' foo '), mb_ltrim(' foo '));
    $this->assertEquals(ltrim(' foo ', ' o'), mb_ltrim(' foo ', ' o'));
    $this->assertEquals('foo', mb_ltrim(' Åfoo', ' Å'));
}

回答6:

mb_ereg_replace seems to get around that:

function mb_trim($str,$regex = "(^\s+)|(\s+$)/us") {
    return mb_ereg_replace($regex, "", $str);
}

..but I don't know enough about regular expressions to know how you'd then add on the "charlist" parameter people would expect to be able to feed to trim() - i.e. a list of characters to trim - so have just made the regex a parameter.

It might be that you could have an array of special characters, then step through it for each character in the charlist and escape them accordingly when building the regex string.

回答7:

My two cents

The actual solution to your question is that you should first do encoding checks before working to alter foreign input strings. Many are quick to learn about "sanitizing and validating" input data, but slow to learn the step of identifying the underlying nature (character encoding) of the strings they are working with early on.

How many bytes will be used to represent each character? With properly formatted UTF-8, it can be 1 (the characters trim deals with), 2, 3, or 4 bytes. The problem comes in when legacy, or malformed, representations of UTF-8 come into play--the byte character boundaries might not line up as expected (layman speak).

In PHP, some advocate that all strings should be forced to conform to proper UTF-8 encoding (1, 2, 3, or 4 bytes per character), where functions like trim() will still work because the byte/character boundary for the characters it deals with will be congruent for the Extended ASCII / 1-byte values that trim() seeks to eliminate from the start and end of a string (trim manual page).

However, because computer programming is a diverse field, one cannot possible have a blanket approach that works in all scenarios. With that said, write your application the way it needs to be to function properly. Just doing a basic database driven website with form inputs? Yes, for my money force everything to be UTF-8.

Note: You will still have internationalization issues, even if your UTF-8 issue is stable. Why? Many non-English character sets exist in the 2, 3, or 4 byte space (code points, etc.). Obviously, if you use a computer that must deal with Chinese, Japanese, Russian, Arabic, or Hebrew scripts, you want everything to work with 2, 3, and 4 bytes as well! Remember, the PHP trim function can trim default characters, or user specified ones. This matters, especially if you need your trim to account for some Chinese characters.

I would much rather deal with the problem of someone not being able to access my site, then the problem of access and responses that should not be occurring. When you think about it, this falls in line with the principles of least privilege (security) and universal design (accessibility).

Summary

If input data will not conform to proper UTF-8 encoding, you may want to throw an exception. You can attempt to use the PHP multi-byte functions to determine your encoding, or some other multi-byte library. If, and when, PHP is written to fully support unicode (Perl, Java ...), PHP will be all the better for it. The PHP unicode effort died a few years ago, hence you are forced to use extra libraries to deal with UTF-8 multi-byte strings sanely. Just adding the /u flag to preg_replace() is not looking at the big picture.

Update:

That being said, I believe the following multibyte trim would be useful for those trying to extract REST resources from the path component of a url (less the query string, naturally. Note: this would be useful after sanitizing and validating the path string.

function mb_path_trim($path)
{
    return preg_replace("/^(?:\/)|(?:\/)$/u", "", $path);
}

来源：https://stackoverflow.com/questions/10066647/multibyte-trim-in-php

标签

php

regex

unicode

multibyte

mbstring