Upgrading to Python 3.12 and my battle with virtualenv cache

edited Oct 3, 2023 with a new fix and a new understanding of the problem

Initial title was …

Python 3.12 + "AttributeError: module ‘pkgutil’ has no attribute ‘ImpImporter’. Did you mean: ‘zipimporter’?"

This is a tale of upgrading Python 3.12

I ran into this error while I was trying to update a project to test on Python 3.12:

  ...
  File "/Users/okken/Library/Application Support/virtualenv/wheel/house/pip-23.0.1-py3-none-any.whl/pip/_internal/metadata/importlib/_envs.py", line 123, in _find_eggs_in_zip
    from pip._vendor.pkg_resources import find_eggs_in_zip
  File "/Users/okken/Library/Application Support/virtualenv/wheel/house/pip-23.0.1-py3-none-any.whl/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?

py312: internal error
Copy

Specifically:

• When I install Python 3.12, and latest versions of pip, tox, and pytest, my tests ran fine.
• However, when running tox from Python 3.12, something blew up, and I got that message at the end of a stack trace.

My original fix for the problem

1. Keep Python 3.11 as my development environment for now. That means my working virtual environment is based on Python 3.11, and that’s what I have tox installed into.

2. Change tox.ini to specify downloading the latest pip for either all environments, or at least for 3.12. Specifically, add download = true.

Here’s a snippet from a work in progress tox.ini change for pytest-check:

[testenv:py312]
download = true
commands = pytest {posargs}
description = Run pytest
Copy

New fix

After trying to isolate the problem to a toy example to submit to tox, and with some help, I realized that:

1. tox uses virtualenv when creating environments.
2. virtualenv likes to cache packages to save on loading up environments, including caching old versions of pip.
3. I had an old cached version of pip on one of my computers.

New fix: run virtualenv --upgrade-embed-wheels. I actually also ran virtualenv --reset-app-data first, but I don’t think it was necessary.

Also, I took out the download = true. It isn’t needed anymore. And it runs fine with Python 3.12.

Problem solved. Yay.

Reference: https://virtualenv.pypa.io/en/latest/cli_interface.html