Useful pytest fixtures to help you improve legacy code

Recently, I had the opportunity to work on improving legacy code – in short, code that has no testing at all¹. The following improvements were made by me as rough steps.

1. Adding Poetry as package manager.
2. Adding tests with pytest.
3. Running tests automatically on CI.
4. Necessary code modifications and refactoring.

In particular, in step 2, the fixture feature of pytest was very useful, and I used it extensively, from generating dummy data to mocking external resource access. For example, I often use requests-mock to mock the Web API. To use it as a pytest fixture is easy:

import pytest
import requests_mock as requests_mock_module

@pytest.fixture
def requests_mock():
    with requests_mock_module.Mocker() as m:
        yield m

@pytest.fixture
def user_api_response():
    def response(user_id):
      return {
          "id": user_id,
          "name": "dummy",
          "updated_at": fake_iso8601(),
          "created_at": fake_iso8601()
      }
    return response

if you save the above Python code as conftest.py, you can mock Web API accesses in any unit test as shown below:

def test_web_api(requests_mock, user_api_response):
    # Mocking access to Web API
    requests_mock.get(
        f'https://api.example.com/users/{user_id}',
        json=user_api_response(user_id))
    
    # Invoke test target function
    get_user()

I also learned for the first time that pytest not only allows users to define their own fixtures, but also provides built-in fixtures. You know that I hadn’t read the document very seriously until now.

In this article, I’d like to show you two of the most frequently used built-in fixtures.

monkeypatch

First up is monkeypatch. As its name implies, it provides a fixture where methods and attributes can be replaced.

In this project, this fixture was very useful because the target module depends on many environment variables and executes external commands with subprocess.run() and os.system().

For example, if you want to replace environment variables with unittest.mock, you have to do the following:

from unittest.mock import patch

@patch.dict('os.environ', {'FOO': 'value'})
def test_sample():
  ...

Maybe it’s partly because I personally don’t like unittest.mock very much because of its complexity, I feel that monkeypatch.setenv() is easier to understand.

def test_sample(monkeypatch):
  monkeypatch.setenv('FOO', 'value')
  ...

Furthermore, it was easy to mock executing commands with subprocess.run() except for specific commands.

import subprocess

def test_sample(monkeypatch, subprocess_completion_process_factory):
    # Mock: subprocess.run
    subprocess_run = subprocess.run

    def mock_subprocess_run(command):
        # pass through invocation of some commands to the original function.
        if command[0] in ['gunzip', 'unzip']:
            return subprocess_run(command)

        return subprocess_completion_process_factory(returncode=0)

    monkeypatch.setattr(subprocess, 'run', mock_subprocess_run)

The parameter subprocess_completion_process_factory is a user-defined fixture that returns a namedtuple that implements only the necessary properties from the subprocess.CompletedProcess.

from collections import namedtuple

FakeCompletionProcess = namedtuple('FakeCompletionProcess', ['returncode'])

@pytest.fixture
def subprocess_completion_process_factory():
    def factory(returncode):
        return FakeCompletionProcess(returncode=returncode)

    return factory

Of course, you can also define your own fixture to mock other commands. Pytest fixture can take other fixtures as arguments, so you can write a fixture that returns a function to mock other commands than the one that matches the array of commands given as arguments, like the following:

import subprocess
import pytest
import re

@pytest.fixture
def monkeypatch_subprocess_run(monkeypatch, subprocess_completion_process_factory):
    subprocess_run = subprocess.run

    def patch(allowed_commands):
        if command[0] in allowed_commands:
            return subprocess_run(command)

        return subprocess_completion_process_factory(returncode=0)
    return patch

...

def test_sample(monkeypatch_subprocess_run):
    monkeypatch_subprocess_run(['gunzip', 'unzip'])
    ...

tmpdir

The module in question also depended on the files that other scripts put out on the filesystem, so we had to put a dummy file in the appropriate path to test it. Of course, you don’t want to leave these files after the test, so you want to work in a temporary directory.

Of course, you can do this with the built-in Python tempfile module, but even here, tmpdir fixture is easy to use.

def test_sample(tmpdir):
    with tmpdir.mkdir('work').as_cwd():
        ...

Since tmpdir returns a py.path.local object, we can write the following in an intuitive way as described above.

1. Create a temporary directory
2. Create a directory called work in it
3. Change the working directory to work

Conclusion

In addition to this, pytest also provides parameterize tests, and since it is on the fixture mechanism, it is easier to use than adopting a separate library for Parameterized testing. It may take some getting used to the style where fixtures are parameter injected into test function arguments, but once you get used to it, it’s a reassuring way to improve your code.

---

1. Reference: Working Effectively With Legacy Code ↩︎