Thor & YAML outputting as binary?

Question

I'm using Thor and trying to output YAML to a file. In irb I get what I expect. Plain text in YAML format. But when part of a method in Thor, its output is different...

class Foo < Thor
  include Thor::Actions

  desc "bar", "test"
  def set
    test = {"name" => "Xavier", "age" => 30}
    puts test
    # {"name"=>"Xavier", "age"=>30}
    puts test.to_yaml
    # !binary "bmFtZQ==": !binary |-
    #   WGF2aWVy
    # !binary "YWdl": 30
    File.open("data/config.yml", "w") {|f| f.write(test.to_yaml) }
  end
end

Any ideas?

Answer 1

All Ruby 1.9 strings have an encoding attached to them.

YAML encodes some non-UTF8 strings as binary, even when they look innocent, without any high-bit characters. You might think that your code is always using UTF8, but builtins can return non-UTF8 strings (ex File path routines).

To avoid binary encoding, make sure all your strings encodings are UTF-8 before calling to_yaml. Change the encoding with force_encoding("UTF-8") method.

For example, this is how I encode my options hash into yaml:

options = {
    :port => 26000,
    :rackup => File.expand_path(File.join(File.dirname(__FILE__), "../sveg.rb"))
}
utf8_options = {}
options.each_pair { |k,v| utf8_options[k] = ((v.is_a? String) ? v.force_encoding("UTF-8") : v)}
puts utf8_options.to_yaml

Here is an example of yaml encoding simple strings as binary

>> x = "test"
=> "test"
>> x.encoding
=> #<Encoding:UTF-8>
>> x.to_yaml
=> "--- test\n...\n"
>> x.force_encoding "ASCII-8BIT"
=> "test"
>> x.to_yaml
=> "--- !binary |-\n  dGVzdA==\n"

Answer 2

After version 1.9.3p125, ruby build-in YAML engine will treat all BINARY encoding differently than before. All you need to do is to set correct non-BINARY encoding before your String.to_yaml.

in Ruby 1.9, All String object have attached a Encoding object and as following blog ( by James Edward Gray II ) mentioned, ruby have build in three type of encoding when String is generated: http://blog.grayproductions.net/articles/ruby_19s_three_default_encodings.

One of encoding may solve your problem => Source code Encoding

This is the encoding of your source code, and can be specify by adding magic encoding string at the first line or second line ( if you have a sha-bang string at the first line of your source code ) the magic encoding code could be one of following:

• # encoding: utf-8
• # coding: utf-8
• # -- encoding : utf-8 --

so in your case, if you use ruby 1.9.3p125 or later, this should be solved by adding one of magic encoding in the beginning of your code.

# encoding: utf-8
require 'thor'
class Foo < Thor
  include Thor::Actions

  desc "bar", "test"
  def bar
    test = {"name" => "Xavier", "age" => 30}
    puts test
    #{"name"=>"Xavier", "age"=>30}
    puts test["name"].encoding.name
    #UTF-8
    puts test.to_yaml
    #---
    #name: Xavier
    #age: 30
    puts test.to_yaml.encoding.name
    #UTF-8
  end
end

Answer 3

I have been struggling with this using 1.9.3p545 on Windows - just with a simple hash containing strings - and no Thor.

The gem ZAML solves the problem quite simply:

require 'ZAML'
yaml = ZAML.dump(some_hash)
File.write(path_to_yaml_file, yaml)