http.get and ISO-8859-1 encoded responses

Question

I\'m about to write a RSS-feed fetcher and stuck with some charset problems.

Loading and parsing the feed was quite easy compared to the encoding. I\'m loading the f

Answer 1

You are probably hitting the same problem described on https://groups.google.com/group/nodejs/browse_thread/thread/b2603afa31aada9c.

The solution seems to be to set the response encoding to binary before processing the Buffer with Iconv.

The relevant bit is

set response.setEncoding('binary') and aggregate the chunks into a buffer before calling Iconv.convert(). Note that encoding=binary means your data callback will receive Buffer objects, not strings.

---

Updated: this was my initial response

Are you sure that the feed you are receiving has been encoded correctly?

I can see two possible errors:

1. the feed is being sent with Latin-1-encoded data, but with a Content-Type that states charset=UTF-8.
2. the feed is being sent with UTF-8-encoded data but the Content-Type header does not state anything, defaulting to ASCII.

You should check the content of your feed and the sent headers with some utility like Wireshark or cURL.

Answer 2

I think the issue is probably with the way that you are storing the data before you are passing it to feedparser. It is hard to say without seeing your data event handler, but I'm going to guess that you are doing something like this:

values = '';
stream.on('data', function(chunk){
  values += chunk;
});

Is that right?

The issue is that in this case, chunk is a buffer, and by using '+' to append them all together, you implicitly convert the buffer to a string.

Looking into it further, you should really be doing the iconv conversion on the whole feed, before running it through feedparser, because feedparser is likely not aware of other encodings.

Try something like this:

var iconv = new Iconv('ISO-8859-1', 'UTF8');
var chunks = [];
var totallength = 0;
stream.on('data', function(chunk) {
  chunks.push(chunk);
  totallength += chunk.length;
});
stream.on('end', function() {
  var results = new Buffer(totallength);
  var pos = 0;
  for (var i = 0; i < chunks.length; i++) {
    chunks[i].copy(results, pos);
    pos += chunks[i].length;
  }
  var converted = iconv.convert(results);
  parser.parseString(converted.toString('utf8'));
});