RSpec - Rerun failed tests that are also tagged?

Question

We have written a large number of selenium UI tests that very occasionally fail due to dumb reasons like a web server being too slow. Additionally, we make heavy use of rspec tags to only run the tests that we want to run. We want to retry ONLY the tests that are tagged with something AND that have previously failed. In the following example, running rspec --tag all --only_failures runs the last two tests, when I only want it to run the 2nd test.

spec_helper.rb

RSpec.configure do |config|
  config.example_status_persistence_file_path = "RunResults.txt"
end

test_spec.rb

require 'rspec'
require 'spec_helper'
require './test'

RSpec.describe Test do
    it "should pass", :all, :one do
        expect(1).to eql(1)
    end

    it "should fail", :all, :two do
        expect(1).to eql(2)
    end

    it "should fail", :three do
        expect(1).to eql(2)
    end
end

RunResults.txt

example_id               | status | run_time        |
------------------------ | ------ | --------------- |
./spec/test_spec.rb[1:1] | passed | 0.00055 seconds |
./spec/test_spec.rb[1:2] | failed | 0.01796 seconds |
./spec/test_spec.rb[1:3] | failed | 0.00017 seconds |

RSpec

$ rspec --tag all --only_failures
Run options: include {:all=>true, :last_run_status=>"failed"}
.FF

Failures:

  1) Test should fail
     Failure/Error: expect(1).to eql(2)
     
       expected: 2
            got: 1
     
       (compared using eql?)
     # ./spec/test_spec.rb:11:in `block (2 levels) in <top (required)>'

  2) Test should fail
     Failure/Error: expect(1).to eql(2)
     
       expected: 2
            got: 1
     
       (compared using eql?)
     # ./spec/test_spec.rb:15:in `block (2 levels) in <top (required)>'

Finished in 0.01949 seconds (files took 0.24733 seconds to load)
3 examples, 2 failures

Failed examples:

rspec ./spec/test_spec.rb:10 # Test should fail
rspec ./spec/test_spec.rb:14 # Test should fail

Answer 1

When you run:

rspec --tag all --only_failures

all three tests get selected: 3 examples, 2 failures . So it's running failed tests + tests tagged all .

I hope I didn't miss an obvious command for doing this. Best I could come up is a little monkey patch to save tags into rspec persistence file:

config.example_status_persistence_file_path = "tmp/spec_persistence.txt"

require "rspec"
require "spec_helper"

# NOTE: override the way persistence file is saved;
#       just put this somewhere appropriate
module RSpec
  module Core
    class ExampleStatusPersister
      private
      def statuses_from_this_run
        @examples.map do |ex|
          result = ex.execution_result

          # NOTE: collect all the tags;
          #       maybe there is a better way, i haven't searched too hard
          tags = ex.metadata.except(*[:block, :description_args, :description, :full_description, :described_class, :file_path, :line_number, :location, :absolute_file_path, :rerun_file_path, :scoped_id, :execution_result, :example_group, :shared_group_inclusion_backtrace, :last_run_status])

          {
            :example_id => ex.id,
            :status     => result.status ? result.status.to_s : Configuration::UNKNOWN_STATUS,
            :run_time   => result.run_time ? Formatters::Helpers.format_duration(result.run_time) : "",

            # NOTE: save tags for reference later
            :tags       => tags.keys.join(","),
          }
        end
      end
    end
  end
end

RSpec.describe "Test" do
  it("1 should pass", :all, :one) { expect(1).to eql(1) }
  it("2 should fail", :all, :two) { expect(1).to eql(2) }
  it("3 should fail", :three)     { expect(1).to eql(2) }
end

https://github.com/rspec/rspec-core/blob/v3.12.0/lib/rspec/core/example_status_persister.rb#L46

When you run tests now, all tags are saved into persistence file:

$ cat tmp/spec_persistence.txt 
example_id         | status | run_time        | tags    |
------------------ | ------ | --------------- | ------- |
./spec/one.rb[1:1] | passed | 0.00075 seconds | one,all |
./spec/one.rb[1:2] | failed | 0.01538 seconds | two,all |
./spec/one.rb[1:3] | failed | 0.00021 seconds | three   |

This is a quick script to select the appropriate tests:

# ./rspec_runner

#!/usr/bin/env ruby

require "rspec"

tag = ARGV[0]

# https://github.com/rspec/rspec-core/blob/v3.12.0/lib/rspec/core/example_status_persister.rb#L8
results = RSpec::Core::ExampleStatusPersister.load_from("tmp/spec_persistence.txt")

filtered = results.filter_map do |result|
  result[:example_id] if result[:status] == "failed" && result[:tags].split(",").include?(tag)
end

if filtered.empty?
  puts "No failed tests found for tag: #{tag}"
else
  puts "Running failed tests with tag: #{tag}"
  puts "rspec #{filtered.join(' ')}"
  puts

  # NOTE: run tests by example_id
  system "rspec #{filtered.join(' ')}"
end

Drumroll:

$ ./rspec_runner all
Running failed tests with tag: all
rspec ./spec/one.rb[1:2]

Run options: include {:ids=>{"./spec/one.rb"=>["1:2"]}}
F
# ...
1 example, 1 failure
Failed examples:
rspec ./spec/one.rb:31 # Test 2 should fail

$ ./rspec_runner one
No failed tests found for tag: one