Ruby `split': invalid byte sequence in UTF-8 (ArgumentError)

Question

I am trying to populate the movie object, but when parsing through the u.item file I get this error:

`split': invalid byte sequence in UTF-8 (ArgumentError)

File.open("Data/u.item", "r") do |infile|
            while line = infile.gets
                line = line.split("|")
            end
end

The error occurs only when trying to split the lines with fancy international punctuation.

Here's a sample

543|Misérables, Les (1995)|01-Jan-1995||http://us.imdb.com/M/title-exact?Mis%E9rables%2C%20Les%20%281995%29|0|0|0|0|0|0|0|0|1|0|0|0|1|0|0|0|0|0|0

Is there a work around??

Answer 1

I had to force the encoding of each line to iso-8859-1 (which is the European character set)... http://en.wikipedia.org/wiki/ISO/IEC_8859-1

a=[]
IO.foreach("u.item") {|x| a << x}
m=[]
a.each_with_index {|line,i| x=line.force_encoding("iso-8859-1").split("|"); m[i]=x}

Answer 2

Ruby is somewhat sensitive to character encoding issues. You can do a number of things that might solve your problem. For example:

1. Put an encoding comment at the top of your source file.

   # encoding: utf-8
   

2. Explicitly encode your line before splitting.

   line = line.encode('UTF-8').split("|")
   

3. Replace invalid characters, instead of raising an Encoding::InvalidByteSequenceError exception.

   line.encode('UTF-8', :invalid => :replace).split("|")
   

Give these suggestions a shot, and update your question if none of them work for you. Hope it helps!