Testing multiple versions of Python in parallel

I was reading my RSS feed when I came across Daniel Roy Greenfeld’s article about using uv to run unit tests on multiple versions of Python. Daniel writes:

But what if I want to test a particular version of Python? Then I simple specify the version of Python to run the test:

uv run --python=3.13 --with pytest --with httpx pytest

Here’s where it gets fun. I can use a Makefile (or a justfile) to test on multiple Python versions.

testall:  ## Run all the tests for all the supported Python versions
    uv run --python=3.10 --with pytest --with httpx pytest
    uv run --python=3.11 --with pytest --with httpx pytest
    uv run --python=3.12 --with pytest --with httpx pytest
    uv run --python=3.13 --with pytest --with httpx pytest

This is fantastic! I’ve used uv for only a couple of projects, and already I cannot imagine ever giving it up again.

However, there’s one disadvantage to the technique Daniel describes above: every time uv switches between Python versions, it will have to remove and replace the whole .venv directory and reinstall all dependencies.

For example, this is what happens when I consecutively run the tests on a project with two different python versions:

$ uv run --python 3.12 --extra test pytest
Using CPython 3.12.11 interpreter at: ...
Creating virtual environment at: .venv
Installed 5 packages in 10ms
...

$ uv run --python 3.13 --extra test pytest
Using CPython 3.13.5 interpreter at: ...
Removed virtual environment at: .venv
Creating virtual environment at: .venv
Installed 5 packages in 8ms

This most likely isn’t much of a problem, because of how fast uv is, but it’s still not ideal. In addition, using Makefiles to run multiple commands sequentially is sooOOoOo boring, especially now that we have computers with insane processor counts. Let’s run these commands in parallel!

Caveat emptor: This presumes that your tests don’t have any real side-effects, and that they can be run fully simultaneously without clashing with one another (shared databases like sqlite, or creating test files could wreak havoc here).

We need to deploy a few tricks so that uv won’t stumble over itself and try to re-use the .venv directory. In the end, this ends up being slightly less elegant than I wanted, but I guess that’s a lesson for me in believing slightly too much in Makefiles.

.PHONY: testall
.PRECIOUS: .venv-%/pyvenv.cfg

PYTHON_VERSIONS := 3.10 3.11 3.12 3.13
TEST_TARGETS := $(addprefix .tests-,$(PYTHON_VERSIONS))

testall: $(TEST_TARGETS)

.venv-%/pyvenv.cfg:
    uv venv \
      --python $(patsubst .venv-%,%,$(@D)) \
      $(@D)

.tests-%: .venv-%/pyvenv.cfg
    venv=$(patsubst .tests-%,.venv-%,$@) && \
    source $$venv/bin/activate && \
    uv run \
      --active \
      --python $$venv \
      --extra test \
      pytest
    touch $@

Okay, there’s a lot to unpack here. Let’s dive in.

The .PHONY and .PRECIOUS lines are easily explained: there will never be a file called testall, so no need to check if it exists or not, and don’t delete the intermediate files created when we initiate a venv.

Similarly easy to understand, the PYTHON_VERSIONS just contains all the versions of Python we want to test against. And the TEST_TARGETS represents the actual targets in the file we want to execute for (tests-3.13, tests-3.12, etc).

The pyvenv.cfg target just says “if anyone needs .venv-whatever/pyvenv.cfg, here’s how to create it”. It very basically calls uv venv and initialises a virtualenv with the correct Python version. The pyvenv.cfg file is created by uv when it sets up the virtualenv, satisfying the Make rule.

The final target, .tests-%, we create manually (touch $@) after running uv run inside the proper virtualenv. This is most likely the part I’m the least happy with; I wish uv had a way of either prefixing venv dirs, or the ability to configure it to say “Python version 3.12 uses .venv-3.12 as the venv dir” or something.

But what does it do? Well, for starters, if you use it exactly like Daniel’s version, it behaves exactly the same way, but it doesn’t replace .venvs all the time:

$ make testall
venv=.venv-3.10 && source $venv/bin/activate && \
  uv run --active --python $venv --extra test pytest
============= 1 passed in 1.01s =============
touch .tests-3.10

venv=.venv-3.11 && source $venv/bin/activate && \
  uv run --active --python $venv --extra test pytest
============= 1 passed in 1.01s =============
touch .tests-3.11

venv=.venv-3.12 && source $venv/bin/activate && \
  uv run --active --python $venv --extra test pytest
============= 1 passed in 1.01s =============
touch .tests-3.12

venv=.venv-3.13 && \ source $venv/bin/activate && \
  uv run --active --python $venv --extra test pytest
============= 1 passed in 1.01s =============
touch .tests-3.13

But more interestingly, we can now run run all tests in parallel across versions of Python!

$ time make testall
make testall  0.55s user 0.13s system 14% cpu 4.725 total

$ time make -j testall
make -j testall  0.63s user 0.17s system 64% cpu 1.234 total

For real-world use, I would most likely also add a clean target, and I would make the test targets depend on the actual pyproject.toml and .py files, but that’s an exercise left for the reader.