What are some practical uses of PHP tokenizer?

Question

What are practical and day-to-day usage examples of PHP Tokenizer ?

Has anyone used this?

Answer 1

A pretty basic use is for syntax highlighting.

foreach(token_get_all($source) as $token) {
    if (is_array($token))
    {
        $map = "token_name";
        echo "$token[1]";
    }
    else {
        echo "$token";
    }
}

The token numbers are usually converted into nicer CSS class names of course, but you could just craft a stylesheet with only .T_COMMENT, .T_ARRAY, .T_ELSEIF, .T_FUNCTION ... classes.