Has anyone figured out a way to run the same cucumber scenario on multiple browsers/web drivers?

Question

I\'m using cucumber + capybara for some web automation testing. I\'d love to be able to wire up my own tag (something like @all_browsers before the scenario) and have it run aga

Answer 1

I do it the way that 'watir' project recommends. I use require parallel_cucumber in my Rakefile and then each Cucumber scenario gets its own parallel thread (up to 20 in this case):

task :run_cucumber do
  FileUtils.mkpath(ENV['OUT_DIR'])
  begin
    # cannot format report as HTML because of parallel forking
    threads = 20
    @result = system "parallel_cucumber features -o \"--format junit --out #{ENV['OUT_DIR']} --format pretty --tag @sauce\" -n #{threads}"
  ensure
    @success &= @result
  end
end

Then, the remaining part of your Cucumber project can be written as usual!! So simple! My full example here: https://github.com/djangofan/cuke-parallel-starter

Answer 2

So, I wound up rolling my own solution to this. Not sure if it was the best or most elegant approach, but I actually just wound up:

1. Abstracting all common environment stuff into env.rb
2. Using Cucumber profiles which would require a specific environment file (such as firefox.rb) that required env.rb and then set the default driver for Capybara to the appropriate driver.
3. Wrote a big ol' thor class with tasks that bundle up a bunch of cucumber commands and call out to run the bad boy with the proper profile.
4. Wrote an 'all_browsers' task which bundles up the commands, then calls out to each specific driver task, so I can now have one task that runs any set of scenarios I supply on all the supported drivers.

Working like a charm and I think might have actually wound up better in the end than anything I was trying above, as within the Thor file I was able to add things like a benchmarking option, as well as whether or not to split the feature run up into multiple threads. Still curious if anyone else came up with a solution for this though.

cucumber.yaml:
Here, the all_features file just does a glob of everything ending in .feature, because if I pulled in the entire features directory it would pull in everything beneath it, including all the profile files, etc, which isn't what I wanted since each profile file sets the default capybara driver to a different value. Once you specify -r as an option to cucumber, all autoloading of any file is halted.

default: --format pretty

chrome: --format pretty -r features/support/profiles/chrome.rb -r features/all_features -r features/step_definitions

firefox: --format pretty -r features/support/profiles/firefox.rb -r features/all_features -r features/step_definitions

celerity: --format pretty -r features/support/profiles/celerity.rb -r features/all_features -r features/step_definitions

firefox.rb (the 'profile' file):

require File.dirname(__FILE__) + "/../env.rb"

Capybara.configure do |config|
  config.default_driver = :selenium_firefox
end

selenium_firefox.rb (where I register the driver, and set some tag capability which I've wound up not needing now, as the @selenium_firefox tag was part of my original attempt at this posted in the question):

# Register a specific selenium driver for firefox
Capybara.register_driver :selenium_firefox do |app|
  Capybara::Driver::Selenium.new(app, :browser => :firefox)
end

# Allows the use of a tag @selenium_firefox before a scenario to run it in selenium with firefox
Before('@selenium_firefox') do
  Capybara.current_driver = :selenium_firefox
end

feature_runner.thor:

require 'benchmark'

class FeatureRunner < Thor
  APP_ROOT = File.expand_path(File.dirname(__FILE__) + "/../")

  # One place to keep all the common feature runner options, since every runner in here uses them.
  # Modify here, and all runners below will reflect the changes, as they all call this proc.
  feature_runner_options = lambda { 
    method_option :verbose, :type => :boolean, :default => true, :aliases => "-v"
    method_option :tags, :type => :string
    method_option :formatter, :type => :string
    method_option :other_cucumber_args, :type => :string
  }


  desc "all_drivers_runner", "Run features in all available browsers"
  method_option :benchmark, :type => :boolean, :default => false
  method_option :threaded, :type => :boolean, :default => true
  feature_runner_options.call # Set up common feature runner options defined above
  def all_drivers_runner
    if options[:threaded]
      feature_run = lambda { 
        thread_pool = []

        t = Thread.new do |n|
          invoke :firefox_runner
        end
        thread_pool << t

        t = Thread.new do |n|
          invoke :chrome_runner
        end
        thread_pool << t

        t = Thread.new do |n|
          invoke :celerity_runner
        end
        thread_pool << t

        thread_pool.each {|th| th.join}
      }
    else
      feature_run = lambda { 
        invoke "feature_runner:firefox_runner", options
        invoke "feature_runner:chrome_runner", options
        invoke "feature_runner:celerity_runner", options
      }
    end

    if options[:benchmark]
      puts "Benchmarking feature run"
      measure = Benchmark.measure { feature_run.call }
      puts "Benchmark Results (in seconds):"
      puts "CPU Time: #{measure.utime}"
      puts "System CPU TIME: #{measure.stime}"
      puts "Elasped Real Time: #{measure.real}"
    else
      feature_run.call
    end
  end

  desc "firefox_runner", "Run features on firefox"
  feature_runner_options.call # Set up common feature runner options defined above
  def firefox_runner
    command = build_cucumber_command("firefox", options)
    run_command(command, options[:verbose])
  end

  desc "chrome_runner", "Run features on chrome"
  feature_runner_options.call # Set up common feature runner options defined above
  def chrome_runner
    command = build_cucumber_command("chrome", options)
    run_command(command, options[:verbose])
  end

  desc "celerity_runner", "Run features on celerity"
  feature_runner_options.call # Set up common feature runner options defined above
  def celerity_runner
    command = build_cucumber_command("celerity", options)
    run_command(command, options[:verbose])
  end

  private
  def build_cucumber_command(profile, options)
    command = "cd #{APP_ROOT} && ./bin/cucumber -p #{profile}"
    command += " --tags=#{options[:tags]}" if options[:tags]
    command += " --formatter=#{options[:formatter]}" if options[:formatter]
    command += " #{options[:other_cucumber_args]}" if options[:other_cucumber_args]
    command
  end

  def run_command(command, verbose)
    puts "Running: #{command}" if verbose
    output = `#{command}`
    puts output if verbose
  end

end

Where everything wound up, in relation to the root directory:

.
|____cucumber.yml
|____features
| |____all_features.rb
| |____google_search.feature
| |____step_definitions
| | |____google_steps.rb
| | |____web_steps.rb
| |____support
| | |____custom_formatters
| | | |____blah.rb
| | |____env.rb
| | |____paths.rb
| | |____profiles
| | | |____celerity.rb
| | | |____chrome.rb
| | | |____firefox.rb
| | |____selenium_drivers
| | | |____selenium_chrome.rb
| | | |____selenium_firefox.rb
| | | |____selenium_ie.rb
| | | |____selenium_remote.rb
| | |____selenium_drivers.rb
|____tasks
| |____feature_runner.thor
| |____server_task.rb  

Output of thor -T

feature_runner
--------------
thor feature_runner:all_drivers_runner  # Run features in all available browsers
thor feature_runner:celerity_runner     # Run features on celerity
thor feature_runner:chrome_runner       # Run features on chrome
thor feature_runner:firefox_runner      # Run features on firefox  

Now I can run something like:
thor feature_runner:all_drivers_runner --benchmark
This would run all features on all capybara drivers in a thread for each driver, benchmnarking the results.

Or
thor feature_runner:celerity_runner
This would run all features only on celerity.

But I can now also supply some other options to the thor command which get passed onto cucumber such as:
--tags=@all_browsers
--formatter=hotpants
--other_cucumber_args="--dry-run --guess --etc"

What a feature file can now look like:

Feature: Start up browser
  @all_browsers
  Scenario: Search Google
   Given I am on the home page
   When I fill in the search bar with "Capybara"
   And I press "Search"
   Then I should see "Capybara"

Seems like a lot of setup, but now if I tag a feature with @all_browsers, I can build out a suite to test against all capybara drivers, in a multi-threaded environment, with one thor command:
thor feature_runner:all_drivers_runner --threaded --tags=@all_browsers

Or build out a smoke test suite that runs in celerity:
thor feature_runner:celerity_runner --tags=@smoke_test

Answer 3

This is possible through the hosted service by SauceLabs. The Cucumber Sauce gem gives you parallel, multi-browser tests.

Alternatively, you may be able to borrow from the source of that gem if you wanted to implement it yourself.

Answer 4

Here is my hack: (my situation is proving a feature works with javascript on and javascript off)

1. Put each scenario into its own feature file.
2. Move every line but the last into the 'Background:' section.
3. Put the last line into a scenario per browser

4. Tag each scenario appropriately

   Feature: a feature
   
   Background:
   Given a user "Jim" exists
   Given a user "Mike" exists
   
   When I login as "mike"
   And I follow "Lesson 1"
   
   And I follow "Upload a video"
   Then "#upload_a_video" should be active
   
   And I fill in "video_title" with "my film"
   And I attach the file "video.mov" to "video_upload"
   And I press "Post"
   
   Scenario: normal
   And I should see "my film"
   
   @javascript
   
   Scenario: javascript
   And I should see "my film"