How do Enumerators work in Ruby 1.9.1?

Question

This question is not about how to use Enumerators in Ruby 1.9.1 but rather I am curious how they work. Here is some code:

class Bunk
  def initialize
    @h = [*1..100]
  end

  def each
    if !block_given?
      enum_for(:each)
    else
      0.upto(@h.length) { |i|
        yield @h[i]
      }
    end
  end
end

In the above code I can use e = Bunk.new.each, and then e.next, e.next to get each successive element, but how exactly is it suspending execution and then resuming at the right spot?

I am aware that if the yield in the 0.upto is replaced with Fiber.yield then it's easy to understand, but that is not the case here. It is a plain old yield, so how does it work?

I looked at enumerator.c but it's neigh on incomprehensible for me. Maybe someone could provide an implementation in Ruby, using fibers, not 1.8.6 style continuation-based enumerators, that makes it all clear?

Answer 1

Here's a plain ruby enumerator that uses Fibers and should pretty much behave like the original:

class MyEnumerator
  include Enumerable

  def initialize(obj, iterator_method)
    @f = Fiber.new do
      obj.send(iterator_method) do |*args|
        Fiber.yield(*args)
      end
      raise StopIteration
    end
  end

  def next
    @f.resume
  end

  def each
    loop do
      yield self.next
    end
  rescue StopIteration
    self
  end
end

And before someone complains about exceptions as flow control: The real Enumerator raises StopIteration at the end, too, so I just emulated the original behaviour.

Usage:

>> enum = MyEnumerator.new([1,2,3,4], :each_with_index)
=> #<MyEnumerator:0x9d184f0 @f=#<Fiber:0x9d184dc>
>> enum.next
=> [1, 0]
>> enum.next
=> [2, 1]
>> enum.to_a
=> [[3, 2], [4, 3]]

Answer 2

Actually in your e = Bunk.new.each the else clause is not executed initially. Instead the 'if !block_given' clause executes and returns an enumerator object. The enumerator object does keep a fiber object internally. (At least that is what it looks like in enumerator.c)

When you call e.each it is calling a method on an enumerator which uses a fiber internally to keep track of its execution context. This method calls the Bunk.each method using the fibers execution context. The Bunk.each call here does execut the else clause and yields up the value.

I do not know how yield is implemented or how a fiber tracks the execution context. I haven't looked at that code. Almost all of the enumerator and fiber magic is implemented in C.

Are you really asking how fibers and yield are implemented? What level of detail are you looking for?

If I am off base please correct me.

Answer 3

As the other posters noted, I believe it creates its own private "fiber" [in 1.9]. In 1.8.7 (or 1.8.6 if you use the backports gem) somehow or other it does the same thing (perhaps because all threads in 1.8 are the equivalent of fibers, it just uses them?)

Thus in 1.9 and 1.8.x, if you chain several of them together a.each_line.map.each_with_index { }

It actually flows through that whole chain with each line, kind of like a pipe on the command line

http://pragdave.blogs.pragprog.com/pragdave/2007/12/pipelines-using.html

HTH.

Answer 4

I think this would be more accurate. Calling each on the enumerator should be the same as calling the original iterator method. So I would slightly change the original solution to this:

class MyEnumerator
  include Enumerable

   def initialize(obj, iterator_method)
    @f = Fiber.new do
      @result = obj.send(iterator_method) do |*args|
       Fiber.yield(*args)
      end
      raise StopIteration
    end
   end

   def next(result)
     @f.resume result
   end

   def each
     result = nil
     loop do
      result = yield(self.next(result))
     end
     @result
   end
end