Managing Python Environments with direnv and pyenv

Introduction

As Python developers, most of us are familiar with Virtual Environments. One of the first things we do when working on a new project is to create an environment. We commonly use virtualenv or venv exactly for that purpose.

Each project we work on uses different packages and may even be compatible with only one Python version.

Doing something repeatedly warrants automation. In this article, we'll see how direnv and pyenv can help us do that.

As a side note, some modern IDEs already automated these steps. For example, PyCharm will create the Virtual environment when initializing a project:

Although automating all these steps is a great win if we use IDEs that support such functionalities, a more generic solution should be IDE-agnostic.

The Problems of virtualenv

Imagine we found a project on GitHub and we would like to play around with it. Pyweather is a simple script that requests the extended weather forecast for our location and prints it on the terminal.

These are the steps we take in order to try the script on our machine:

$ git clone https://github.com/lcofre/pyweather.git
$ cd pyweather

Then we create the virtual environment and install the packages the script uses:

$ virtualenv --python=python3 env
$ source env/bin/activate
(env) $ pip install requirements.txt

And only then, we can execute the script:

(env) $ ./pyweather.py	

We created a virtual environment and saved it in the root folder of our project. While being in that folder we had to activate the environment with the source command.

When we finish working, we need to leave the Virtual Environment by executing deactivate:

(env) $ deactivate

All those steps are our responsibility. How many times we may have forgotten to activate an environment and installed a package globally!

Let's see how direnv helps us automate this.

direnv

direnv was made mainly to load environment variables, depending on the current directory and has an extension for many shells.

In this example, we'll be using bash, but direnv supports many other shells as well. And what's more important for us, it allows us to manage Python Virtual Environments.

To install it we'll run the bash installer they provide. We could use the package manager of our distribution, but the bash installer will ensure we install the latest version available:

$ curl -sfL https://direnv.net/install.sh | bash

Now we need to hook direnv to bash. We'll edit ~/.bashrc and then reload it:

$ echo 'eval "$(direnv hook bash)"' >> ~/.bashrc
$ source ~/.bashrc

This way direnv will link itself to the shell and will be executed always before each prompt. We will never notice it's working in the background.

direnv will check if something needs to be loaded on the current folder. It checks the existence of a file named .envrc, with instructions on what should be loaded.

To load Python Virtual Environments we run the layout command, followed by the Python version:

$ echo 'layout python' > .envrc

Or if we want to use Python 3:

$ echo 'layout python3' > .envrc

Running these will tell direnv to look for a python or python3 executable on the path.

As soon as we create the .envrc file we'll be warned that we need to allow direnv to access that folder. Let's do that right now:

$ direnv allow

direnv: loading .envrc
...
New python executable in /home/myuser/untitled/.direnv/python-3.6.9/bin/python3
...
Installing setuptools, pkg_resources, pip, wheel...direnv:
done.
direnv: export +VIRTUAL_ENV ~PATH

As we can see in the output, the Virtual Environment was immediately created. The prompt is not modified though, so we won't see the name of the environment written at the beginning.

Now we can install the packages we need as we did on the environment we created in the previous section:

$ pip install -r requirements.txt

direnv will silently activate the environment in the background. Whenever we move out of the directory, the environment will be deactivated:

$ cd ..

direnv: unloading

If we can use any Python version that is installed on the system, direnv is all we need.

Let's suppose now that our pyweather script requires a very specific version though.

pyenv

pyenv is a version management utility for Python. It allows, among other things, to change Python versions on a per-project basis. direnv provides support for it since version 2.21.0, so together they can give us a higher level of control on the version we use in our environment.

Let's start by installing pyenv:

$ curl -L https://pyenv.run | bash

And then ensuring it will always be accessible to our terminal:

$ echo 'export PATH="~/.pyenv/bin:$PATH"' >> ~/.bashrc
$ echo 'eval "$(pyenv init -)"' >> ~/.bashrc
$ echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc
$ source ~/.bashrc

Now let's suppose our pyweather script requires a very specific Python version, 3.6.2.

First, we need to install that version of Python:

$ pyenv install 3.6.2

And now we can configure our project to use the specific version:

$ echo 'layout pyenv 3.6.2' > .envrc
$ direnv allow

We can confirm all works as expected by checking the Python version in the environment:

$ python --version

Python 3.6.2

If we ever need to change the Python version, it will be enough for us to change the layout in the .envrc file.

Thanks to both utilities we can change the layout to any Python version, and our virtual environment will be updated right away.

Another advantage of using both direnv and pyenv is that we can version our .envrc file in our project repository.

That way all contributors will be able to configure their environment as intended by the project, as long as they install the utilities and Python version needed.

Conclusion

Virtual Environments are in a way detached from the Python development workflow. We need to remember to configure and activate it before working with our project. Thanks to direnv and pyenv we can automate all this, and entering the project folder will do all the work for us in the background.

Installation of both utilities is not straightforward, but after being done once we will save ourselves a lot of time. We will also always have the certainty that we are working with the right virtual environment and Python version.