Python: How to Specify a GitHub Repo in requirements.txt

Introduction

In Python the requirements.txt file helps manage dependencies. It's a simple text file that lists the packages that your Python project depends on. But did you know you can also specify a direct GitHub repo as a source in your requirements.txt? In this Byte, we'll explore how and why to do this.

Why specify a direct GitHub repo?

There are a few reasons why you might want to specify a direct GitHub repo in your requirements.txt file. Maybe the package you need isn't available on PyPI, or perhaps you need a specific version of a package that's only available on GitHub (after all, in some older packages, updates don't always get published on PyPI). Or, you could be collaborating on a project and want to use the most recent changes that haven't been pushed to PyPI yet.

For instance, there have been a few times where I needed a feature from a Python library that was only available in the development branch on GitHub. By specifying the direct GitHub repo in our requirements.txt, we were able to use this feature before it was officially released.

And lastly, you can use direct URLs as a way to use private repos from GitHub.

How to Use a Direct GitHub Source in requirements.txt

To specify a direct GitHub repo in your requirements.txt, you'll need to use the following format:

git+https://github.com/username/repo.git

Let's say we want to install the latest code from the requests library directly from GitHub. We would add the following line to our requirements.txt:

git+https://github.com/psf/requests.git

Then, we can install the dependencies from our requirements.txt as usual:

$ pip install -r requirements.txt

You'll see that pip clones the requests repo and installs it.

Variations of the Repo URL

There are a few variations of the repo URL you can use, depending on your needs.

If you want to install a specific branch, use the @ symbol followed by the branch name:

git+https://github.com/username/repo.git@branch-name

To install a specific commit, use the @ symbol followed by the commit hash:

git+https://github.com/username/repo.git@commit-hash

And of course, another commonly used version is for private repos. For those, you can use an access token:

git+https://<token>@github.com/username/repo.git

Wait! Be careful with access tokens, they're similar to passwords in that they give access to your account. Don't commit them to your version control system.

I'd recommend using environment variables to keep them secure. When using environment variables (i.e. ${GH_ACCESS_TOKEN}), pip will replace it when installing from requirements.txt.

Conclusion

Being able to specify a direct GitHub source in your requirements.txt gives you more flexibility in managing your Python project's dependencies. Whether you need a specific version of a package, want to use a feature that hasn't been officially released yet, or are working with private repos, this technique can be a very useful tool in your development workflow.