file_get_contents() converts UTF-8 to ISO-8859-1

Question

I am trying to get search results from yahoo.com.

But file_get_contents() converts UTF-8 charset (charset, that yahoo uses) content to ISO-8859-1.

Try:

$filename = "http://search.yahoo.com/search;_ylt=A0oG7lpgGp9NTSYAiQBXNyoA?p=naj%C5%A1%C5%A5astnej%C5%A1%C3%AD&fr2=sb-top&fr=yfp-t-701&type_param=&rd=pref";

echo file_get_contents($filename);

Scripts as

header('Content-Type: text/html; charset=UTF-8');

or

<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

or

$er = mb_convert_encoding($filename , 'UTF-8');

or

$s2 = iconv("ISO-8859-1","UTF-8",$filename );

or

echo utf8_encode(file_get_contents($filename));

NOT help, because after getting web content speciall characters as š ť ž are replaced with question marks ???

I would appreciate any kind of help.

Answer 1

This seems to be a content negotiation problem as file_get_contents probably sends a request that only accepts ISO 8859-1 as character encoding.

You can create a custom stream context for file_get_contents using stream_context_create that explicitly states that you accept UTF-8:

$opts = array('http' => array('header' => 'Accept-Charset: UTF-8, *;q=0'));
$context = stream_context_create($opts);

$filename = "http://search.yahoo.com/search;_ylt=A0oG7lpgGp9NTSYAiQBXNyoA?p=naj%C5%A1%C5%A5astnej%C5%A1%C3%AD&fr2=sb-top&fr=yfp-t-701&type_param=&rd=pref";
echo file_get_contents($filename, false, $context);

Answer 2

file_get_contents should not change the charset. The data is pulled in as a binary string.

When checking out the url you provided, this is the header it provides:

Content-Type: text/html; charset=ISO-8859-1

Also, in the body:

<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">

Also, you can't convert UTF-8 losslessly convert to ISO-8859-1 and get the characters back when going back to UTF-8. UTF-8 / unicode supports many many more characters, so the characters are lost in the first step.

In the browser this is not the case, so perhaps you just need to provide a correct Accept-Encoding header to instruct yahoo's system you can accept UTF-8.

Answer 3

$s2 = iconv("ISO-8859-1","UTF-8//TRANSLIT//IGNORE",$filename );

Better solution...

function curl($url){
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($ch, CURLOPT_ENCODING, 1);
    return curl_exec($ch);
    curl_close($ch);
}

echo curl($filename);

Answer 4

For anyone investigating on this:

The time I spent on encoding issues taught me that rarely php functions "magically" change the encoding of strings. (One of these rare examples is :

exec( $command, $output, $returnVal )

Please note also that the working header set is as follows:

header('Content-Type: text/html; charset=utf-8');

and not:

header('Content-Type: text/html; charset=UTF-8');

As I had a similar issue as the one you describe, it was enough to set the headers properly.

Hope this helps!