Python mocks should always have a spec

31 Jul 2022

If you’ve worked with Python tests for a while you’ve probably had a moment where you encountered an assertion that looks like one of these:

assert mock.called_once()
...
mock.assret_called_once()

Because Python is often so very readable, and because these two lines look plausible at a glance, these assertions will lie in wait right until you need to make changes to the code that they test. At that point one of two things are likely to happen: if you’re doing test-driven development you might be confused as to why these assertions are still passing; or if you’re not doing test-driven development you’ll see a green test suite report, go on to deploy your changes and then see a bunch of errors.

The problem, of course, is mocks. When you use unittest.mock.patch you’ll end up with a mock object that is dangerously permissive. It doesn’t care that called_once may not be an existing method on itself or the thing it’s mocking, or that assret_called_once is a typo, it will always return a new mock object each time and that object will always evaluate as True in an assert statement. This eventually became such a problem that common typos were actually hard-coded into the standard library in Python 3.5 and now raise an AttributeError! This is a helpful improvement, but it only helps that narrow case.

Suppose you’re testing a class you’ve written:

# test.py
from unittest import TestCase, mock

class MyClass:
    pass


def some_code_under_test():
    MyClass().non_existent_method()


class TestMyClass(TestCase):

    @mock.patch('test.MyClass')
    def test_my_class(self, klass):
        some_code_under_test()  # would blow up outside this test

        # This should also fail, `called_once` is incorrect
        assert klass.return_value.non_existent_method.called_once()

This test will pass just fine even though MyClass has no methods defined on it at all, which likely isn’t the behaviour you want. To take a slightly more realistic example:

from typing import Dict, Tuple

import requests


class ThirdPartyClient:

    def patch(self, identifier: str, data: Dict[str, str]) -> requests.Response:
        url = f'https://www.third-party.com/api/v1/product/{identifier}/'

        response = requests.patch(url, json=data)
        response.raise_for_status()

        return response


def update_third_party_api(identifier: str, data: Dict[str, str]) -> bool:
    client = ThirdPartyClient()

    try:
        client.patch(identifier, data)
    except requests.RequestException:
        return False
    else:
        return True

You want to test that update_third_party_api returns the correct value in each scenario, but you don’t want your test to actually call out to the third party, so you reach for a mock.

from unittest import TestCase, mock


class TestUpdateThirdPartyApi(TestCase):

    @mock.patch.object(ThirdPartyClient, 'patch')
    def test_returns_true_if_update_succeeds(self, third_party_patch):
        response = requests.Response()
        response.status_code = 200
        response._content = b'Success!'
        third_party_patch.return_value = response

        assert update_third_party_api('an-identifier', {'new': 'data'}) is True
        third_party_patch.assert_called_once()

This test passes, and is a reasonable enough test to have written even if you were doing test-driven development. Let’s say that further down the line, though, that the third party updates their client and the function signature for ThirdPartyClient.patch changes – it now expects you to pass in a timeout:

class ThirdPartyClient:

    def patch(
        self,
        identifier: str,
        data: Dict[str, str],
        timeout: Tuple[float, float],  # new required argument
    ) -> requests.Response:
        url = f'https://www.third-party.com/api/v1/product/{identifier}/'

        response = requests.patch(url, json=data, timeout=timeout)
        response.raise_for_status()

        return response

Your test will still pass even though the code in update_third_party_api is now calling ThirdPartyClient.patch incorrectly! You’ll only end up finding out once you’ve deployed the code and you start getting errors, if not complaints from users.

autospec to the rescue

If you pass autospec=True to your mock.patch.object decorator then you’ll end up with a much less permissive mock, one that expects to be called correctly:

TypeError: missing a required argument: 'timeout'

Now you can truly be confident that your tests have got you covered when your code is broken. A specced mock will also catch non-existent mock methods like assert third_party_patch.called_once(), which I’ve come across a surprising number of times in some of our test suites:

AttributeError: 'function' object has no attribute 'called_once'

If you’re creating mocks outside of a decorator then you can use create_autospec to achieve the same ends.

Limitations of autospec

You might reasonably ask why autospec=True isn’t the default for all new mocks, or at least why you wouldn’t use it on all of yours. As with most things in this job there are always gotchas and caveats to consider.

Unintended side effects

If you’re mocking a class then create_autospec and autospec=True work by calling getattr(original, attribute) for every attribute on original. If you’ve overridden the __getattribute__ method on your class for whatever reason then creating a mock may end up having some surprising consequences:

def __getattribute__(self, name: str) -> Any:
    print('side effect!')
    return super().__getattribute__(name)
$ pytest test.py
...

----------------------- Captured stdout call -----------------------
side effect!

Performance implications

Because creating a spec involves looping over every attribute on an object this can have an impact on the performance of your test suite if the object in question is very large and/or it is mocked in a large number of tests. On my machine an object with 50 attributes that is mocked in 5,000 tests will add a little over two seconds to your test suite:

An autospecced class with 10 attributes will add 1.66s over 5000 tests via create_autospec
An autospecced class with 10 attributes will add 1.67s over 5000 tests via the patch decorator
An autospecced class with 25 attributes will add 1.78s over 5000 tests via create_autospec
An autospecced class with 25 attributes will add 1.86s over 5000 tests via the patch decorator
An autospecced class with 50 attributes will add 2.11s over 5000 tests via create_autospec
An autospecced class with 50 attributes will add 2.24s over 5000 tests via the patch decorator

The patch decorator seems a little slower than running create_autospec, but not significantly so. If you need/want your test suite to be blindingly fast then you might want to consider where it’s most valuable to use autospeccing.

Dynamic attributes

Looping over all the attributes returned by an object’s dir method works perfectly for methods and attributes that are declared at runtime, but will end up missing anything that’s created dynamically, in the __init__ method for example:

class ThirdPartyClient:

    def __init__(self):
        self.version = '1.0.0'

    ...


    @mock.patch.object(ThirdPartyClient, 'version', autospec=True)
    def test_version_specific_logic(self, third_party_version):
        ...

Try to mock version you’ll end up with the following error:

AttributeError: <class 'test.ThirdPartyClient'> does not have the attribute 'version'

I ran into this recently with the Session class from the requests package because the auth and headers attributes are declared dynamically, so your specced mock will throw an error when you try to access them in your test. In some cases this might indicate that either your code or your test isn’t implemented quite right, if not both, or you might decide that the security that speccing provides isn’t necessary in this use case.

Return values

This one feels slightly uncharitable to autospec, but it’s worth remembering that the return value of mocked methods will always end up being a permissive mock by default unless you specify otherwise, even if you’ve applied autospec=True in what might look like the right place:

class MyClass:
    pass


class OtherClass:
    def get_my_class_instance(self):
        return MyClass()


def some_code_under_test():
    instance = OtherClass().get_my_class_instance()
    instance.non_existent_method()


class TestMyClass(TestCase):

    @mock.patch('test.OtherClass', autospec=True)
    def test_my_class(self, klass):
        some_code_under_test()  # would blow up outside this test

In the not-very-realistic example above MyClass has no method called non_existent_method, so when some_code_under_test is running outside of your test then it will blow up. You might think that mocking OtherClass with autospec=True would help you here but it won’t, the return value of get_my_class_instance will end up being a permissive mock that won’t check the attributes on MyClass. If you want to fix that then you need to get terribly specific about what get_my_class_instance should return in your test:

class TestMyClass(TestCase):

    @mock.patch('test.OtherClass', autospec=True)
    def test_my_class(self, klass):
        my_klass = mock.create_autospec(MyClass)
        klass.return_value.get_my_class_instance.return_value = my_klass

        some_code_under_test()  # now this will fail

It’s not really reasonable to expect the built-in speccing to catch this for you, but I wanted to include it just in case you thought that autospec makes you invincible!

Conclusion

Without getting into the extremely murky waters of when you should and shouldn’t use mocks in Python, about which many words have been and will be written, the chances are that you’re going to encounter a number of them in whatever codebase you’re working on. Using autospec and create_autospec should, in my opinion, be your default way of constructing these mocks because it provides a much-needed safety net in your tests that persists over time as interfaces naturally develop. Depending on what you’re mocking it can have some significant downsides, but for most use cases it’s as close to a testing no-brainer as you’re going to find.