ExUnit's --seed surprising behavior! 😲


If I asked you what ExUnit’s --seed option does, you’d probably say “randomize the order of tests”, right? And that’s right.

But it turns out it does more!

I recently discovered this through a weird intermittent failure. 👇

The test my team and I were working on had a more complicated version of this:

test "randomness" do
  integer = Enum.random([1, 2, 3])

  refute integer == 2

Just by looking at that, you can see the test would fail intermittently (whenever Enum.random/1 returned 2).

But that’s not surprising.

What was surprising was that once we found a seed that failed, we could consistently fail the test with the seed – even if we only ran that test!

So, clearly --seed had to be doing more. 🤔

After a bit of digging, we discovered that ExUnit uses the seed to populate Erlangs :rand.seed/2 function:

defp generate_test_seed(seed, %ExUnit.Test{module: module, name: name}, rand_algorithm) do
  :rand.seed(rand_algorithm, {:erlang.phash2(module), :erlang.phash2(name), seed})

And, as it turns out, Enum.random/2 uses Erlang’s :rand module to calculate its random values!:

This function uses Erlang’s :rand module to calculate the random value.

So, first ExUnit uses the seed to seed the random number generator. Then, Enum.random/1 uses that number generator to calculate the random values! 😲

And that’s a good thing!

At first, I was confused and wondered why ExUnit would do that. But then I realized – by seeding the random number generator, ExUnit is able to create more consistent, reproducible tests even when dealing with randomness.

That’s why reproducing the test failure was easy in the first place!

Want the latest Elixir Streams in your inbox?

    No spam. Unsubscribe any time.