Python lazy imports you can use today

There’s a proposal, PEP 810 – Explicit lazy imports for Python to natively support lazy importing starting in Python 3.15.

However, it has not been accepted yet, and even if it is accepted, 3.15 is a year away. What do we do now?

The techniques covered in this post that allow you to use lazy importing NOW with Python 3.13, Python 3.12, …, really every version of Python.

Let’s look at a small code example that imports something.

mycode.py:

import somelib

def main():
    print("main() called")
    somelib.a_function()

if __name__ == '__main__':
    main()
Copy

The module mycode.py needs to call a_function from somelib. In order to do so, mycode needs to import somelib.

We traditionally put imports at the top of a file at the top level (not in a function). The import somelib will do whatever Python does to import something, and end up with somelib as a name in the global scope of mycode.

Since the import is outside of a function, that importing happens at the time that mycode is imported.

Let’s throw some print statements into somelib:

somelib.py:

import time
time.sleep(1)  # Simulate a heavy import

def a_function():
    print("a_function called")

# global scope print
print("somelib imported")
Copy

Python imports code before running it

When we run the code with python mycode.py, Python will import the module, then run it.

$ python mycode.py
somelib imported
main() called
a_function() called
Copy

All global level imports run upon import

If we don’t “run” mycode, but simply import it, somelib also gets imported, since the import is not in a function.
We can try this from the REPL or from the command line.

REPL:

>>> import mycode
somelib imported
Copy

command line:

$ python -c 'import mycode'
somelib imported
Copy

Use lazy import by moving import into a function

We can move the import somelib out of the module global scope and into a function.

# import somelib

def main():
    print("main() called")
    import somelib
    somelib.a_function()
Copy

This causes Python to hold off on importing somelib until this function is called.

$ python mycode.py
main() called
somelib imported
a_function() called
Copy

This has at some benefits:

• somelib isn’t imported at mycode import time.
• Anything somelib might import is also not imported right away.
• We can speed up Python startup time by not importing stuff until needed.
• somelib will end up never being imported if main() isn’t called.

Just import:

$ python -c 'import mycode'
Copy

Ok. That’s not very exciting. But note that “somelib imported” isn’t being printed.

REPL:

>>> import mycode
>>> mycode.main()
main() called
somelib imported
a_function() called
Copy

In the REPL we can really see that somelib doesn’t get imported when import mycode happens.

You can import something multiple times with no performance hit

If we need somelib in multiple functions, and we don’t know which will be called first, we can just import from every use. This effectively still imports once though, so you don’t get a performance hit.

mycode.py

def one():
    print("one() called")
    import somelib
    somelib.a_function()


def two():
    print("two() called")
    import somelib
    somelib.a_function()

def main():
    one()
    two()

if __name__ == '__main__':
    main()
Copy

Even though it looks like import somelib is done twice, it ends up only being one import. But it puts the name somelib in the local namespace of both functions one() and two().

$ python mycode.py
one() called
somelib imported
a_function() called
two() called
a_function() called
Copy

You’ll notice that “somelib imported” only shows up once.

The reason we put it in both places is because the name somelib needs to be available in the local scope of both functions.

Keeping global scope with local import

If we know which function is always going to be called first, we can just import it at that time and set it to a global name

def one():
    global somelib
    print("one() called")
    import somelib
    somelib.a_function()


def two():
    print("two() called")
    somelib.a_function()
Copy

This works just fine.

Using importlib.import_module() for lazy importing

You can also use importlib.import_module() to the same effect.

from importlib import import_module

def one():
    print("one() called")
    global somelib
    somelib = import_module("somelib")
    somelib.a_function()


def two():
    print("two() called")
    somelib.a_function()

def main():
    one()
    two()

if __name__ == '__main__':
    main()
Copy

With import_module(), you can put it in multiple functions if you don’t want it in global scope. It will still only do the actual import once.

Using python -X importtime to find slow imports

I don’t usually start with lazy imports.
I usually start with any imports at the top of the file.

It’s when I’m optimizing startup time or test collection time that I look for which imports to switch to lazy loading.

To find which libraries are loading slowly, you can use Python’s builtin -X importtime.

$ python -X importtime mycode.py 2>&1 | sort -r | head -10          
two() called
somelib imported
one() called
import time: self [us] | cumulative | imported package
import time:   1002021 |    1002021 | somelib
import time:      1206 |       1206 |     _collections_abc
import time:      1095 |       4836 | site
import time:       974 |       1882 | _frozen_importlib_external
import time:       897 |       2327 | encodings
import time:       718 |       2555 |   os
Copy

This runs the code, but pays attention to how long it takes to import something. It reports cumulative import time, and it shows up in a kind of tree.

The report goes to stderr. So if you want to pipe it to something, keep that in mind.

In bash like terminals

• 2>&1 sends stderr to the stdout stream.
• sort -r does a reverse sort. I want to see big numbers first.
• head -10 shows the top 10 lines.

Speeding up test collection with lazy imports during testing

When writing test code, you have even more reasons to avoid slow imports, but also have a couple extra techniques available.

When pytest collects all of the tests it can see, it will import all test files during the collection phase, before running any fixtures or tests.

So if you have imports in global scope of test files, those imports will happen during test collection.

Let’s take a look at an example test.

test_somelib.py

import somelib

def test_a_function(capsys):
    somelib.a_function()
    captured = capsys.readouterr()
    assert "a_function() called" in captured.out

def test_something_else():
    ...
Copy

With import somelib at the normal global namespace, somelib will be imported during test collection.

$ pytest -s test_somelib.py
============ test session starts ============
collecting ... somelib imported
collected 2 items                           

test_somelib.py ..

============= 2 passed in 1.05s =============
Copy

That “1.05s” is way too long, and we can see with -s allowing print statements that somelib is imported.

That’s not surprising. Just pointing it out.

This import happens during collection and even happens when we call the test that doesn’t need somelib.

$ pytest -s test_somelib.py::test_something_else
============ test session starts ============
collecting ... somelib imported
collected 1 item                            

test_somelib.py .

============= 1 passed in 1.05s =============
Copy

Still getting imported.

Moving import to test functions

Let’s use a technique we already have learned with straight Python and move the import into the test code function.

def test_a_function(capsys):
    import somelib
    somelib.a_function()
    captured = capsys.readouterr()
    assert "a_function() called" in captured.out

def test_something_else():
    ...
Copy

Now, when we call only test_something_else(), the import doesn’t happen.

$ pytest -s test_somelib.py::test_something_else
============ test session starts ============
collected 1 item                            

test_somelib.py .

============= 1 passed in 0.04s =============
Copy

Using importlib with test functions

With imports in test code, we can use importlib.import_module() also.

from importlib import import_module

def test_a_function(capsys):
    somelib = import_module('somelib')
    somelib.a_function()
    captured = capsys.readouterr()
    assert "a_function() called" in captured.out
Copy

Don’t use the global trick in test functions

We can’t just put global somelib in the first test functions and expect it to be there for other test functions. It might work when you try it, as we know pytest by default runs tests from top to bottom. However, with the ability for users to just call one test function, and randomize order and change order, and deselect tests, etc. you really can’t rely on the order of test functions being run.

However, we can stick imports in fixtures.

Moving import to a fixture

You can create fixtures for libraries you want to use, and just include them in the parameter list.

import pytest

@pytest.fixture(name='somelib')
def somelib_import():
    global somelib
    import somelib
    return somelib

def test_a_function(capsys, somelib):
    somelib.a_function()
    captured = capsys.readouterr()
    assert "a_function() called" in captured.out

def test_something_else():
    ...
Copy

This works fine, but looks weird.
It looks a little less weird with importlib.

import pytest
from importlib import import_module

@pytest.fixture()
def somelib():
    _somelib = import_module('somelib')
    return _somelib

def test_a_function(capsys, somelib):
    somelib.a_function()
    ...
Copy

Either of these methods works to speed up test collection and avoid imports for test methods that don’t need them.

Moving import to an autouse fixture

If everything in the test file (or nearly so) needs the import, we can make the fixture autouse and set the module name to global. Then we don’t have to name it in each test.

import pytest

@pytest.fixture(autouse=True)
def somelib_import():
    global somelib
    import somelib

def test_a_function(capsys):
    somelib.a_function()
    captured = capsys.readouterr()
    assert "a_function() called" in captured.out

def test_something_else():
    ...
Copy

Setting up an “autouse” import doesn’t save us the import time for test_something_else anymore, so I reserve this technique for test modules that all use the import.

$ pytest -s test_somelib.py::test_something_else
============ test session starts ============
collected 1 item                            

test_somelib.py somelib imported
.

============= 1 passed in 1.05s =============
Copy

However, when we run other test files, we still get the test collection speedup.

$ pytest -s -k foo            
============ test session starts ============
collected 3 items / 2 deselected / 1 selected

test_foo.py .

====== 1 passed, 2 deselected in 0.04s ======
Copy

Feedback welcome

These are techniques I use. If you have other, I’d love to hear them. Feel free to contact me