why is rspec running different numbers of tests?

Question

I realize this is a very broad question but I've seen our tests run 800 or 1,500 individual tests and pass both times. What could be causing rspec to run a different number of tests on different builds?

Answer 1

I believe it is do with Rspec running test in random order. See here for more information.

Rspec order option

for example if you want to see the same test results and still want to run them randomly you can do following

When you use --order random, RSpec prints out the random number it used to seed the randomizer.

--order rand:1234 --order default tells RSpec to load groups and examples as they are declared in each file

bundle exec rspec --order rand ## for example this will results with a seed 1234

if you want to run same number of test in same random order you can run

bundle exec spec --order rand:1234 # This will always run the test in same random order with the given seed (1234)