Goodbye Dependency Headaches: Discover the Power of Pipx for Python Package Management
Episode #21: Escape Python dependency hell with Pipx by creating an isolated virtual environment for each application.
If you have been a Software Developer as long as I have (more than ten years now), you might have encountered dependency hell at some point in your career.
If you haven't (lucky you), let me introduce the concept:
Dependency hell describes the complex web of software dependencies that can arise, making it challenging to install, update, or run applications due to version conflicts and compatibility issues. It is a common problem in software development and deployment, especially across different platforms and ecosystems.
As the term describes, it is not a great state of mind.
Software Engineers have written entire programming languages and countless tools out of the frustration of dealing with some sort of dependency hell.
In this article, I would like to describe my recent experience of dependency hell in Python and how I solved the problem once and for all—at least, I hope.
I'll introduce a tiny tool called Pipx, which allows you to install a CLI Python application (and its dependency) in a totally isolated virtual environment, avoiding any dependency conflict.
I’ll explain how to use Pipx and what are its benefits over Pip and other similar tools.
This article is specifically tailored for Python developers, both experienced and inexperienced.
Whether you're a seasoned pro or just starting out, the insights and solutions I'll be sharing can be invaluable in your Python development journey.
As a Python developer, I hope you can gain valuable insights from this article that you might apply at some point in your career, further enhancing your skills and knowledge.
In this article:
Meaning of the front page image
💰 (paid content) My encounter with Pipx
💰 (paid content) What is Pipx?
💰 (paid content) Pip vs Pipx
💰 (paid content) How to get started
💰 (paid content) Example usage
💰 (paid content) Pipx run
Meaning of the front page image
When exploring a Midjourney prompt for this article, I wanted to convey the concept of "Dependency Hell".
What better metaphor for Dependency Hell than a sea of lava?
So, a Python snake avoids the sea of lava (the dependency hell) by crawling over a precarious tree log hanging above it.
I generated the above picture with Midjourney using the prompt below.
A vibrant, comic-book inspired image of a green Python snake slithering across a weathered log, its head tilted curiously as it navigates a fiery, lava-filled chasm beneath.
Want to connect?
👉 Follow me on LinkedIn and Twitter.
If you need 1-1 mentoring sessions, please check my Mentorcruise profile.
My encounter with Pipx
My first (and most horrific) experience of dependency hell goes back to the time that I was writing Java code early on in my professional career.
Things didn't change much later when I began writing Scala code (mainly for Spark) but it got a lot better when after a few years I started writing Python code.
I love creating virtual environments in Python to isolate each project I am working on.
So, I stayed away from dependency hell for a while.
Last week, I had my first experience of dependency hell in Python. I had to install at least four different CLI applications in Python.
As usual, I added them all to my Poetry config. Soon, Poetry started complaining that those tools were using incompatible library versions.
As years of experience solving similar problems in Java taught me, I started checking if there was a way to solve those conflicts, but after 30 minutes of struggling, I gave up.
Since I knew the maintainers of those tools, I reached out to check if they were willing to update one ancient legacy dependency library. I even opened a public GitHub issue to get this fixed.
To my surprise, they suggested I use a separate virtual environment for each tool.
My first thought was... Are you crazy? One virtual environment for each of those? It will take me ages to set everything up. Are you sure you cannot upgrade this tiny dependency? I could fork the tool and update it myself; that would have been much faster, but potentially, it could still have taken hours.
After a few minutes of complaining, they finally suggested a tool I never heard of: Pipx.
What the hell is pipx? was my first reaction. I have never heard of it.
Surprisingly, this tool did precisely that: installing each Python CLI application in an isolated virtual environment.
Here I am, with this new piece of knowledge, trying to teach you about the benefits of Pipx for your Python development.
What is Pipx?
Pipx simplifies your life by enabling you to install CLI applications written in Python (and their dependencies) in isolated virtual environments.
It's as easy as pipx install <package>.
It creates one virtual environment for each CLI application to avoid dependency conflicts.
It looks similar to brew or other package managers you might have used in different languages.
Between the benefits, you can count on a clean uninstall of your packages, no need to use sudo to install global packages and the ability to list packages already installed by pipx.
Behind the scenes, pipx uses the Python module venv to create a virtual environment and pip to install those dependencies.
Those virtual environments are installed under a fixed location, ~/.local/share/pipx/venvs/<package>, and the Python applications installed have symlinks added to the ~/.local/bin.
Those two locations differ on Windows but are the same on macOS and Linux.
More information about pipx can be found at the pipx—GitHub repository.
Some GitHub stats
Pipx is open source and actively developed at the time of writing.
It counts 146 contributors and 8.9k GitHub stars.
The number of stars has grown exponentially since the beginning of the year.
The latest version 1.5.0 has been released on Mar 29, 2024.
Pip vs Pipx
As the name implies, "Pipx" has a similar purpose to Pip.
The main difference is that while Pip installs a package globally or in the current virtual environment, Pipx instead creates a new virtual environment for each package and then adds a symlink to the package entry point to a dedicated location.
Similarly to Pip, Pipx installs packages from PyPi by default, but it can optionally install from Git repositories, folders, and any other place Pip can install from.
How to get started
Getting started with Pipx is quite simple.
You install the tool first, then add the default location ~/.local/bin to your PATH, and finally, install your packages.
Installing pipx varies for all operating systems.
If you, like me, are a MacOS user, you can use brew to install pipx. Otherwise, for Linux, you can use one of the many package managers for your operating system.
Warning: Installing pipx with pipx is not suggested.
The pipx official documentation provides more information on how to install pipx.
Example usage
Let's say you are installing the famous Python formatter called Black.
How would you install this package? And what does it get created under the hood?
You can install Black with Pipx using the command:
pipx install blackBehind the scenes, Pipx will go through these three steps:
Creates a virtual environment via the Python module
venvat~/.local/share/pipx/venvs/blackInstall the package and its dependencies in that venv
Create a symlink at
~/.local/bin/blackto the command installed at~/.local/share/pipx/venvs/black/bin/black
Since pipx ensurepath has already added the location "~/.local/bin" to your PATH, the command black should be available immediately.
Pipx run
Pipx allows you to avoid installing a package if you only need to run a command once.
Instead of using, pipx install <package>, you can use pipx run <package> <package params>.
This command will create a temporary virtual environment that will be destroyed when the command finishes running.
I love this feature since it reminds me how Nix shell works.
If you want only run a command in Nix once without installing it, you can run nix-shell -p <command> --run <command and params>
nix-shell -p jq --run jq .If you have never heard of Nix before I have written an article about it at Effortless Python Development with Nix.
This article explains how I use Nix to install Python, create a virtual environment, install Poetry and much more.