Using Python and Tox without internet access

I'm working on an implementation of an AVL tree in Python, and I've pulled out all the stops. Being a good developer, I have a pyproject.toml, I'm using flit, I'm using tox, it's almost like I know what I'm doing! (I don't.)

Imagine my surprise when I try to run my tests without internet access, and I'm treated to this message on my console:

  Installing build dependencies ... error

  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [7 lines of output]
      WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f7381ed9e80>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/flit-core/
      WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f7381eb2210>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/flit-core/
      WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f7381eb2490>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/flit-core/
      WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f7381eb2710>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/flit-core/
      WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f7381eb2990>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/flit-core/
      ERROR: Could not find a version that satisfies the requirement flit_core<4,>=3.11 (from versions: none)
      ERROR: No matching distribution found for flit_core<4,>=3.11
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

Okay, I hadn't even realized that it was making network requests every time I ran my tests. Why wouldn't it? I only installed it on my system through Debian. Must be a lot of load on PyPI's servers!

I tried looking online, and found answers like this one on StackOverflow, which involve modifying the pyproject.toml. My concern is that I don't want to require other people to download things in advance—I just want it to work offline, dangit!

Eventually, looking on Tox' website, I found a section where they discuss how to use a custom PyPI server. They mention a PIP_INDEX_URL environment variable, which is the ticket to my way out.

Creating a local PyPI server

The idea of this solution is to make a PyPI server that runs locally. By 'server', I mean 'local filesystem directory'—pip appears to support file URLs, which we can make use of. You could also host this on a local server (e.g. a Raspberry Pi), if you'd prefer.

You will need internet access to run these steps! The offline access should work afterwards.

1. Create a folder to hold your files and future repository. I picked ~/pippkgs, you can pick whatever.
2. Determine the packages you need. These should be in your pyproject.toml file. In my case, the only dependency is flit_code<4,>=3.11, but I could also add others (like pytest, tox, and so on.)
3. Download them with pip download. This will download a bunch of .whl files that store the package info.
4. Create a packages.txt file with the wheels. This will be useful in the next step. You might try 'ls *.whl >packages.txt' to generate this.
5. Run dumb-pypi to create a local repository. dumb-pypi (found via packaging.python.org) creates a static-file simple repository that contains Python packages that pip can download. It doesn't appear to be in Debian, so I ran it with pipx (which is in Debian).

   The command I used was:

   pipx run dumb-pypi --output-dir repo --packages-url `pwd` --package-list packages.txt
   

   I originally used '.' instead of `pwd`, but that caused pip to not find the package—that URL is relative to the package location! As such, you want the absolute path to the directory you put the wheels in. If it's on a server, provide the full URL to that directory.
6. Set the PIP_INDEX_URL environment variable when working offline. For example, I run tests with

   PIP_INDEX_URL=file:///home/duncan/pippkgs/repo/simple tox run
   

   Of course, you'll need to adjust the filesystem path. 'simple' seems to need to be there, presumably since it says what type of repo is in use. You can also use an HTTP URL, but it'll ignore it unless it's HTTPS or localhost.

   Now, when e.g. tox runs pip, it'll look in your offline repository, and work without network access!

You only need to use that environment variable when you're offline, but you can also use it when you're online, for example to have your own local packages, to ensure that you're ready if the internet goes out, or to avoid network traffic.

Hopefully this helps you work without needing the internet as much!