A month ago, I wrote about my Playwright fixture for testing static websites in a browser. I’ve been copying that fixture from project-to-project, but recently I decided to add it to chives, the utility library I use for all my static websites (or tiny archives).

One of my rules for chives is that everything in it has to be tested – but how do you test a pytest fixture? Test code is just code, and it isn’t immune to bugs. Who tests the tests?

Enter Pytester, a tool designed for testing pytest plugins. Pytester allows you to run isolated test suites, make assertions about the outcomes, and verify the behaviour of custom fixtures. In your top-level test suite, you always want everything to be passing, but with Pytester you can write a mixture of passing and failing tests, and check the results are what you expect.

Pytester is disabled by default, so you first enable it in your top-level conftest.py file (the pytest configuration file where you configure plugins and fixtures):

# conftest.py
pytest_plugins = ["pytester"]

Here’s an example of using Pytester where we create a test suite with two tests and check that one passes, one fails:

from pytest import Pytester


def test_with_pytester(pytester: Pytester):
    """
    Run an isolated test suite with pytester.
    """
    # Make a temporary pytest test file
    pytester.makepyfile(
        """
        def test_arithmetic():
            assert 2 + 2 == 4
    
        def test_list_inclusion():
            assert "yellow" in ["red", "green", "blue"]
        """
    )

    # Run the isolated test suite with pytest
    result = pytester.runpytest()

    # Check that one test passed, one failed
    result.assert_outcomes(passed=1, failed=1)

I can imagine creating something similar with some complicated collection of nested functions, exec() and pytest.raises, but using Pytester is a cleaner interface than what I’d build.

Under the hood, Pytester creates a temporary directory, writes specified files into it, then runs a fresh pytest subprocess against it. It has helper functions for writing files, including Python files (makepyfile), a conftest.py file (makeconftest), and plain text files (maketxtfile).

When we’re testing a fixture, we can create a conftest.py file that imports that fixture, then reference it in the tests. Here’s a more complicated example, where we import one of my Playwright fixtures in my conftest.py, write an HTML file into the temporary directory, then use them both in the test:

from pytest import Pytester


def test_browser_fixture(pytester: Pytester):
    """
    Try testing the browser fixture with pytester.
    """
    # Make a conftest.py file
    pytester.makeconftest("""
        from chives.browser_fixtures import browser
    """)
    
    # Make an HTML file
    (pytester.path / "greeting.html").write_text("""
        <p>Hello world!</p>
    """)
    
    # Make a temporary pytest test file
    pytester.makepyfile(
        """
        from chives.browser_fixtures import file_uri
        from playwright.sync_api import Browser, expect
    
        def test_browser_fixture(browser: Browser) -> None:
            uri = file_uri("greeting.html")

            p = browser.new_page()
            p.goto(uri)
            expect(p.get_by_text("Hello world!")).to_be_visible()
        """
    )

    # Run the isolated test suite with pytest
    result = pytester.runpytest()

    # Check that one test passed
    result.assert_outcomes(passed=1)

This pattern is sufficient for many fixtures, but it doesn’t work for Playwright – if you run this test, the isolated test suite gives an error rather than a passing test. Playwright needs you to install a web browser to work (for example, playwright install webkit), and Pytester runs in a sufficiently isolated environment that Playwright can’t find the browsers you already have installed.

We could run the install command inside the temporary directory, but that would be slow and inefficient – it would be better if we could tell Playwright to look for the already-installed browsers elsewhere. If we set the PLAYWRIGHT_BROWSERS_PATH environment variable inside our isolated test suite, Playwright will look there for browsers.

First, we need to work out where browsers are installed – we could hard-code the location, or we could inspect the executable_path property property on a browser:

from pathlib import Path
from playwright.sync_api import sync_playwright
import pytest


@pytest.fixture(scope="session")
def playwright_browsers_path() -> str:
    """
    Return the cache directory where Playwright browsers are installed.
    """
    with sync_playwright() as p:
        # In my local builds, this returns a path like:
        #
        #    ~/Library/Caches/ms-playwright/webkit-2272/pw_run.sh
        #
        # Unwrap two levels to get to the `ms-playwright` folder.
        return str(Path(p.webkit.executable_path).parent.parent)

Then we need to set this as an environment variable inside the Pytester test suite. I couldn’t find an easy way to set an environment variable; the best approach I came up with was to modify os.environ inside the conftest.py file. (Perhaps we could access the MonkeyPatch object and set more environment variables, but using private attributes is icky.)

Here’s how the new test starts:

def test_browser_fixture(pytester: Pytester, playwright_browsers_path: str):
    """
    Test the browser fixture with pytester.
    """
    # Make a conftest.py file
    pytester.makeconftest(f"""
        from chives.browser_fixtures import browser
        import os
        
        os.environ["PLAYWRIGHT_BROWSERS_PATH"] = {playwright_browsers_path!r}
    """)

    ...

and now the overall test passes. Here’s the complete code for the new test:

from pathlib import Path
from playwright.sync_api import sync_playwright
import pytest
from pytest import Pytester


@pytest.fixture(scope="session")
def playwright_browsers_path() -> str:
    """
    Return the cache directory where Playwright browsers are installed.
    """
    with sync_playwright() as p:
        # In my local builds, this returns a path like:
        #
        #    ~/Library/Caches/ms-playwright/webkit-2272/pw_run.sh
        #
        # Unwrap two levels to get to the `ms-playwright` folder.
        return str(Path(p.webkit.executable_path).parent.parent)


def test_browser_fixture(pytester: Pytester, playwright_browsers_path: str):
    """
    Test the browser fixture with pytester.
    """
    # Make a conftest.py file
    pytester.makeconftest(f"""
        from chives.browser_fixtures import browser
        import os
        
        os.environ["PLAYWRIGHT_BROWSERS_PATH"] = {playwright_browsers_path!r}
    """)

    # Make an HTML file
    (pytester.path / "greeting.html").write_text("""
        <p>Hello world!</p>
    """)

    # Make a temporary pytest test file
    pytester.makepyfile(
        """
        from chives.browser_fixtures import file_uri
        from playwright.sync_api import Browser, expect
    
        def test_browser_fixture(browser: Browser) -> None:
            uri = file_uri("greeting.html")

            p = browser.new_page()
            p.goto(uri)
            expect(p.get_by_text("Hello world!")).to_be_visible()
        """
    )

    # Run the isolated test suite with pytest
    result = pytester.runpytest()

    # Check that one test passed
    result.assert_outcomes(passed=1)

The full test suite is more extensive, and checks that certain scenarios fail or error – will the fixtures spot the mistakes I expect them to? For example, my Page fixture is meant to load a page and fail the test if there are any console warnings or errors; does it actually fail the test correctly?

I don’t expect to use Pytester very often, because it’s rare for me to write fixtures complex enough to need their own test suite – but sometimes I do, and it’s good to know how to create another layer of safety net.