ExUnit's --seed surprising behavior! đ˛
Notes
If I asked you what ExUnitâs --seed option does, youâd probably say ârandomize
the order of testsâ, right? And thatâs right.
But it turns out it does more!
I recently discovered this through a weird intermittent failure. đ
The test my team and I were working on had a more complicated version of this:
test "randomness" do
integer = Enum.random([1, 2, 3])
refute integer == 2
end
Just by looking at that, you can see the test would fail intermittently
(whenever Enum.random/1 returned 2).
But thatâs not surprising.
What was surprising was that once we found a seed that failed, we could consistently fail the test with the seed â even if we only ran that test!
So, clearly --seed had to be doing more. đ¤
After a bit of digging, we discovered that ExUnit uses the seed to populate
Erlangs :rand.seed/2 function:
defp generate_test_seed(seed, %ExUnit.Test{module: module, name: name}, rand_algorithm) do
:rand.seed(rand_algorithm, {:erlang.phash2(module), :erlang.phash2(name), seed})
end
And, as it turns out, Enum.random/2 uses Erlangâs :rand module
to calculate its random values!:
This function uses Erlangâs :rand module to calculate the random value.
So, first ExUnit uses the seed to seed the random number generator. Then,
Enum.random/1 uses that number generator to calculate the random values! đ˛
And thatâs a good thing!
At first, I was confused and wondered why ExUnit would do that. But then I realized â by seeding the random number generator, ExUnit is able to create more consistent, reproducible tests even when dealing with randomness.
Thatâs why reproducing the test failure was easy in the first place!