Ruby 1.9 doesn't support Unicode normalization yet

Question

I\'m trying to port over some of my old rails apps to Ruby 1.9 and I keep getting warnings about how \"Ruby 1.9 doesn\'t support Unicode normalization yet.\" I\'ve tracked

Answer 1

The StringEx Gem seems to work pretty well. It has no dependency on Iconv either.

It adds some methods to the String class, like "to_ascii" which does beautiful transliteration out of the box:

require 'stringex'
"äöüÄÖÜßë".to_ascii #=> "aouAOUsse"

Also, the Babosa Gem does a great job transliterating UTF-8 strings, even with language support:

"Jürgen Müller".to_slug.transliterate.to_s           #=> "Jurgen Muller"
"Jürgen Müller".to_slug.transliterate(:german).to_s  #=> "Juergen Mueller"

Enjoy.

Answer 2

If you'd rather not monkey patch the Inflector module, you can also do this...

Both of the following worked for me to silence this annoying "Ruby 1.9 doesn't support Unicode normalization yet" warning:

silence_stream(STDERR) {
  whatever_code_caused_transliterate_to_be_called
}

or

silence_warnings {
  whatever_code_caused_transliterate_to_be_called
}

This does have the disadvantage that it requires cluttering up your calling code, but it is a technique you can use generally whenever you don't want to see warnings or other output.

activesupport provides silence_stream and silence_warnings in activesupport-2.3.11/lib/active_support/core_ext/kernel/reporting.rb

Answer 3

Replace the body of the method with

raise "transliterate called"

and observe a backtrace which will show you where the stuff is coming from at the first call. Your app will of course collapse as well but that will likely give you the culprit from the first try.

Answer 4

I appreciate that this is a dirty way to solve the problem, but having read the error message I'm aware of the issue. So I want to get rid of the warnings. I dropped this code in environment.rb:

module ActiveSupport
  module Inflector
    # Calling String#parameterize prints a warning under Ruby 1.9,
    # even if the data in the string doesn't need transliterating.
    # Maybe Rails 3 will have fixed it...?
    if RAILS_GEM_VERSION =~ /^2\.3/
      undef_method :transliterate
      def transliterate(string)
        string.dup
      end
    end
  end
end

Answer 5

That method definition is wrapped in an if-statement for Ruby 1.9. Right above it, you will find the regular definition, which shows a bit more of what this is doing. It's a method used to convert accented characters into their regular variants. E.g.: á => a, or ë => e

But this method is only used in parameterize, which is in turn defined right above transliterate. This is all still in ActiveSupport. I can't find anything that is directly calling parameterize.

So perhaps you're using parameterize or transliterate yourself, somewhere in your Rails application?

Common usage (according to the parameterize documentation) is for creating friendly permalinks from arbitrary strings, much like SO does, for example:

http://stackoverflow.com/questions/2135247/ruby-1-9-doesnt-support-unicode-normalization-yet
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Answer 6

If you are aware of the consequences, i.e. accented characters will not be transliterated in Ruby 1.9.1 + Rails 2.3.x, place this in config/initializers to silence the warning:

# http://stackoverflow.com/questions/2135247/ruby-1-9-doesnt-support-unicode-normalization-yet
module ActiveSupport
  module Inflector
    # Calling String#parameterize prints a warning under Ruby 1.9,
    # even if the data in the string doesn't need transliterating.
    if Rails.version =~ /^2\.3/
      undef_method :transliterate
      def transliterate(string)
        string.dup
      end
    end
  end
end

Rails 3 does indeed solve this issue, so a more future-proof solution would be to migrate towards that.