Effortless Python Development with Nix
Episode #18: After struggling with various tools and suggestions for years, I finally streamlined setting up a new Python project by using Nix.
If you ask 100 developers how they set up their Python development environment, getting many different answers is not uncommon.
Why hasn't the problem been solved yet?
In the last decade, many tools have promised to simplify your life. At best, they solve one issue and introduce many more.
I have tried and abandoned many tools like:
pyenv
Pipenv
virtualenv
pyenv-virtualenv
This article will show you how I finally found what I consider the most efficient and automated way to set up a development environment for my Python projects.
We will cover:
Developing inside a container.
What are ephemeral environments?
What is Nix, and why do you need it?
What can go wrong when setting up a Python environment?
A "simple" setup for Python.
An optimized setup with Nix.
Dev environments for other programming languages.
The lessons learned here are not limited to programming in Python.
We will demonstrate how to set up a development environment that can be used for installing command-line tools or programming in other languages, such as Golang.
Python has been chosen because of the extra complexity of working with multiple versions of the interpreter, using virtual environments for isolation, and pinning down dependencies.
You don't have to be a Python developer to read this article, but being one helps you appreciate how Nix makes things much more manageable.
If you are not a Python developer, feel free only to read the sections that are relevant to you.
Want to connect?
👉 Follow me on LinkedIn and Twitter.
If you need 1-1 mentoring sessions, please check my Mentorcruise profile.
Developing inside a container
In a previous article Develop inside a container for easy developer onboarding, I suggested using containers for quickly creating new development environments with a technology called Devcontainers introduced by Microsoft.
Since that article, I have changed my mind a little.
Containers are great for packaging and deploying an application but are less than ideal for local development.
Who knows, maybe in the future, our laptops will be dumb terminals, and remote environments like GitHub Codespaces or GitPod will be mainstream.
Until then, working locally is the fastest and most economical way to develop.
So, should you completely abandon dev containers? Maybe not.
You can reach a good compromise.
You can use a Dev container to avoid installing Nix directly on your laptop. Then, follow the rest of the article to discover how to install your application dependencies with Nix.
More info about combining Nix and Devcontainers can be found at Devenv and devcontainers.
What are ephemeral environments?
I've been meaning to write this article for a while, but I finally decided to do it after watching a video by Victor Farcic.
The video in question is called Say Goodbye to Containers - Ephemeral Environments with Nix Shell and talks about the concept of creating "ephemeral environments" with Nix.
In short, an ephemeral environment is a development environment where you can install command line tools that can be easily removed once you are done with them.
Similar results can be achieved with containers.
In my opinion, containers bring extra complexity to local development that is not worth the trouble—especially given the alternatives suggested in this article.
Furthermore, bloated containers with entire operating systems are still the norm. I don't want to download an operating system worth hundreds of megabytes to avoid installing curl on my laptop.
Even if you can have smaller containers, as I suggest in my article Shrink to Secure: Kubernetes and Compact Containers what you need at the end of the day is just a package manager
Victor's video explains better than me the concept of ephemeral containers and why Nix is better than containers in this context.
What is Nix, and why do you need it?
If I wanted to introduce Nix in a single sentence, I would say:
Nix is a package manager, an operating system, a programming language, a packaging tool, and much more.
The challenge with Nix is that due to its extensive capabilities, it comes with a steep learning curve.
What I need from Nix is not much more than a package manager that seamlessly works on all major OSes—no more separate scripts to install a tool in a different Operating System.
Instead of using Nix directly, I use a tool called Devenv, which is based on Nix but has a straightforward learning curve.
If you need to install some command line tools, you can provide the names of the Nix packages in a list.
Do you want to install the Python interpreter? Just toggle a config.
It also solves many problems highlighted by Victor's video above, like configuring Nix to work with your current shell instead of the default one.
This is not the first time that I talk about Devenv. I wrote about it in a previous article called Taskfile: a modern alternative to Makefile.
In that article, I have already introduced Devenv and another tool called Direenv, which will also be used here.
Later in this chapter, we will cover how to set up a project by using both tools.
What can go wrong when setting up a Python environment?
There are many ways to set up your Python environment.
You can install the Python interpreter with Conda, from Linux repositories, on Mac with Homebrew, and in many other ways.
Some of those methods are faster than others. Some require you to compile the interpreter, and they might take a lot of time and third-party dependencies.
So many things can go wrong with this step that using an already compiled package for your laptop architecture is always preferable.
Then, there is the option of installing many different versions of the interpreter. Should you install them side by side? Should you use pyenv? Is it ok to use more than three different interpreter versions? Should you use the latest and greatest version?
What about creating virtual environments? Should you create global virtual environments in the same root folder on your laptop? Should you create a virtual environment per project?
A "simple" setup for Python
The most simplistic setup for Python development is the following.
First, install the Python environment from binaries from Python.org; then, you can use the following commands to set up your environment and install Python libraries.
# create a virtual environment in the same folder of your code
python -m venv .venv
# Activate the virtual environment
source .venv/bin/activate
# Install a package
python -m pip install <package>
# Freeze your dependencies and store them into a file
python -m pip freeze > requirements.txt
# Use the file requirements.txt to replicate the virtual environment on other laptops
Python -m pip install -r requirements.txt
# when you are done, deactivate the virtual environment
deactivate
This method is easier than other tools like pyenv, virtualenv, pyenv-virtualenv, and many other tools. But it has some drawbacks:
The version of the interpreter in the virtual environment needs to match the interpreter's version on your laptop. What if you want to have multiple versions of the interpreter?
You need to activate the virtual environment before you can install a package or run a Python application
if you want to replicate the virtual env on another laptop, you first need to freeze the Python dependencies in a requirements.txt and then use it later to install those libraries
requirements.txt contains a flattened dependency tree, mixing direct and indirect dependencies. There is also no way to distinguish development and production dependencies.
Resources for this simple setup can be found as follows:
Some of those issues have been bothering me for a while, and I never found a solution that was sane enough.
Well, until very recently.
An optimized setup with Nix
In this setup, I use a combination of Devenv and Direnv.
To get started, you need to:
Install Devenv as introduced at Getting started with Devenv. This will take care of installing Nix as well.
Install and setup Direnv as recommended at Automatic shell activation
If you are starting from an empty project, you first need to setup Devenv with the command:
devenv init
This will generate some files like:
devenv.nixwhere to define the Nix packages to installdevenv.yamlMost of the time, you won't interact with this file. Except here and in a few other occasions for supporting extra features like installing the Python interpreter..envrcto make Direnv integrate with Devenv. More about this in the official docs about Devenv.
To install Python and set up our dev environment, you must change the content of devenv.nix as follows.
{ pkgs, ... }:
{
packages = [
# pkgs.git
];
languages.python.enable = true;
languages.python.version = "3.12.0";
languages.python.venv.enable = true;
languages.python.poetry.enable = true;
languages.python.poetry.install.enable = true;
}Furthermore, the content of devenv.yaml needs to match the following:
inputs:
nixpkgs:
url: github:NixOS/nixpkgs/nixpkgs-unstable
nixpkgs-python:
url: github:cachix/nixpkgs-python
Once this is done, you need to issue the following command from the terminal to allow Direnv to interact with your file system:
direnv allow
The first time that you run the previous command, Devenv will
install Python as a Nix package with the pinned version provided. Here it is 3.12.0
create a virtualenv inside the hidden folder
.venvInstall Poetry into the virtual environment
Initialize Poetry by creating a
poetry.tomland apoetry.lock, which will contain the pinned Python dependencies
To install Python libraries in Poetry, you can issue the command.
# install a package as a production dependency
poetry add <package>
# install a package only for development
poetry add --dev <package>
The version of the installed packages is stored in the poetry.toml automatically. This makes the setup easily reproducible on another laptop. Forget about syncing the libraries in your virtual environment to the requirements.txt file.
To learn more about Poetry, you can visit their docs.
Now, imagine that you want your colleague to contribute to this project.
Devenv and Direnv are the only dependencies for them to install and configure.
After that, he only needs to use the terminal to visit the root folder where the project lives, issue the command direnv allow and, Devenv will take care of the rest.
In a couple of minutes, without doing anything else, they will have a virtual environment already set up with all the Python dependencies installed.
Direnv also automatically activates the virtual environment where it finds a .venv folder with a virtual environment. No more activating and deactivating the virtual environment.
One more thing, if you are using Git for version control, I suggest you have a .gitignore file in the root directory with the following content:
.devenv*
.direnv
.venv
So that you avoid committing folders and files to Git that should be automatically regenerated instead.
Dev environments for other programming languages
What we covered in this article only scratches the surface of what you can achieve with Devenv, Nix and Direnv.
For example, you could work with Golang with devenv.nix as follows.
{ pkgs, ... }:
{
languages.golang.enable = true;
}
Maybe more interestingly, you could install many command line tools with a similar devenv.nix:
{ pkgs, ... }:
{
# https://devenv.sh/packages/
packages = [
pkgs.git
pkgs.air
pkgs.gofumpt
pkgs.golangci-lint
pkgs.go-swag
pkgs.gocyclo
pkgs.curl
pkgs.delve
];
languages.go.enable = true;
}
You can also choose between over 80,000 packages from Nix packages, and Devenv supports many different languages and other features.
Conclusion
The topic discussed in this article is quite complex and involves several components that require careful attention.
There are multiple factors to consider, and the possibility of things going wrong cannot be ignored.
Nevertheless, I am confident that once you become familiar with the tools Devenv and Direnv, your work experience will become much simpler and easier to manage.