Ruby 2.0.0 String#Match ArgumentError: invalid byte sequence in UTF-8

Question

I see this a lot and haven\'t figured out a graceful solution. If user input contains invalid byte sequences, I need to be able to have it not raise an exception. For example:<

Answer 1

In Ruby 2.0 the encode method is a no-op when encoding a string to its current encoding:

Please note that conversion from an encoding enc to the same encoding enc is a no-op, i.e. the receiver is returned without any changes, and no exceptions are raised, even if there are invalid bytes.

This changed in 2.1, which also added the scrub method as an easier way to do this.

If you are unable to upgrade to 2.1, you’ll have to encode into a different encoding and back in order to remove invalid bytes, something like:

if ! s.valid_encoding?
  s = s.encode("UTF-16be", :invalid=>:replace, :replace=>"?").encode('UTF-8')
end