Parameterized tests in Elixir

It’s a note about my idiomatic way to write parameterized tests in Elixir.

These days, most developers know the DRY principle, they heard it over and over again in their professional life. So they carefully craft and refactor their production code to make it follows the principle.

Furthermore, if you are one of 21-century software developers, you may realize the value of Unit testing and it should be your daily life friend. In many cases, unit testing is lightweight and running tests don’t take much time, but it can catch (surprisingly many) mistakes by developers. Regardless of whether you are a TDD follower or not, many developers including I love it.

For instance, my typical development workflow when implementing a new feature or fixing a bug is:

1. Write tests satisfying feature specification or reproducing a bug.
2. Write code until the tests pass.
3. commit, push and submit a pull request.

It’s a very clear and safe way I think. +1 bonus, as a result, most of the code paths have a corresponding unit test, so we can safely refactor existing code. But in reality, we often forget to refactor tests itself. We’re tempted to copy-and-paste a similar test that differs only in some parameters. We’re satisfied too quickly just by refactoring production code other than tests.

My experience says that duplication in tests is also a cause of trouble. It happens a lot that corresponding tests must be modified when you add a new feature to production code. If you have similar tests, you have to modify all of them. It’s annoying. We should remove duplications from our tests by refactor out parameters. It’s so-called Parameterized tests in many testing frameworks.

Existing Solutions

Elixir’s built-in ExUnit is a very good standard unit testing framework, but unfortunately it lacks parameterized testing.

I’ve tried KazuCocoa/ex_parameterized, but I don’t chose it because the syntax is a bit weird (I understand why) and some cases in parameters can’t be evaluated.

My preferred approach

Currently, I settled on the code like below after trying a few times:

# 1. The helper functions for the test module. To make it possible to import
# this helper module in the test module, define this module outside the context that uses it.
defmodule MyTest.Helpers do
  @spec fake_params(Enumerable.t()) :: map
  def fake_params(override \\ %{}) do
    %{
      country: "jp",
      phone_number: Faker.phone_number(),
      locale: "ja",
      company: "My Company",
      department: "My Department",
      email: Faker.Internet.email(),
      first_name: Faker.Name.first_name(),
      last_name: Faker.Name.last_name()
    }
    |> Map.merge(Map.new(override))
  end
end

defmodule MyTest do
  use MyApp.ConnCase

  # Because I'd like to use functions in the helper module both in parameterized cases and
  # test cases, alias and import it.
  alias MyTest.Helpers
  import Helpers

  describe "signup" do
    for {description, signup_params} <- [
          # 2. You cannot invoke functions in the testing module which is not defined yet.
          # So we need the helper module.
          "all filled": Helpers.fake_params(),
          "department can be omitted": Helpers.fake_params(department: nil),
          "department can be null": Helpers.fake_params() |> Map.delete("department")
        ] do
      # 3. You cannot use variables in this context in the context inside a test case.
      # So you have to use module attributes or `@tag` feature in ExUnit. Personally,
      # I prefer the latter.
      @tag signup_params: signup_params
      test "no errors: #{description}", %{conn: conn, signup_params: signup_params} do
        # ...
      end
    end
  end
end

As for why it’s like the code above, there are twists and turns (as I wrote in the comment):

1. The module that defines helper functions is defined outside the test module. Because otherwise, you can’t import it. Of course, you can write the absolute path to reference it, but you may want to reduce the noise from the test code as much as possible.
2. In the first place, the reason why we define helper functions in a separate module instead of private functions is because we want to use them to generate parameters. When you write parameters inside the test module, private functions are not defined yet.
3. The part that is a little tricky is the part that describes test. Inside test do ... end block, you can’t see any outer variables.

It’s a bit redundant, but there are only well known constructs. I think it’s important to anyone who learned Elixir but not be a familiar with the existing tests can easily understand what are happening in the tests.