On new Debian 12 Bookworm and Ubuntu 24.04 Noble Numbat installs, when I try running pip3 install [something] (whether that's Ansible or some other Python tool), I get the following error message:
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.
If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
See /usr/share/doc/python3.11/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
The error message says you can pass in the flag --break-system-packages but that sounds terrifying. I just want pip to stop nagging me, but let me manage my system dependencies like I have for many years.
I think some Python developers really want people like me to use virtual environments, but that's way too much effort when I don't really care to do that, thankyouverymuch. If you want to use venv more power to you. I just like getting stuff done on my little servers.
So the solution for Debian 12, at least, is to delete the EXTERNALLY-MANAGED file in your system Python installation:
sudo rm -rf /usr/lib/python3.11/EXTERNALLY-MANAGED
If using Ansible, you can add this task to your playbook prior to running any pip tasks:
- name: Ignore PEP 668 because it's silly.
ansible.builtin.file:
path: /usr/lib/python3.11/EXTERNALLY-MANAGED
state: absent
Note that the python3.11 version number should match whatever you have installed—it was 3.11 at the time of this blog post's writing. For Ubuntu 24.02, the version is python3.12.
See this answer on Stack Overflow for more. Another interesting option is to install and use pipx, which does the grunt work of managing the venvs for you.
Comments
This merges your development environment with the OS's working environment and its tools, some of which are dependent on python. This can break things in ways that can be difficult to recover. See the Motivation section of the PEP that implemented the check for a more thorough explanation. https://peps.python.org/pep-0668/
Venvs aren't hard, but you do you.
They're not hard, just annoying.
And 99% of my work is inside containers or VMs, where rebuilding is done at the drop of a hat. Having to add on any extra stuff to manage virtual Python environments is just annoying and takes more space + time in the build.
Just like with PHP installs, Java, etc., I will rely on the OS install and if I absolutely need more separation, I throw the extra environment inside Docker. Some people also like to install and manage multiple versions of PHP, Node.js, or Java, but with modern tooling, it feels redundant to try to manage environments using separate tools for each programming language.
I get/respect their motivation, but if i'm doing a pip3 install $WHATEVER --user as non-root, that's not going to blow up the whole OS and i'd like to be left alone, thanks.
Jeff's nudge towards tracking down the "EXTERNALLY-MANAGED" file under the python dirs and removing worked for me. Thanks!
It's a pain if you want to install some command line utility via pip into your $PATH for general use. Think awscli for example (although their v2 has its own installer), or ansible as you mention. I feel like this case got overlooked in the PEP.
Yeah, I think the PEP authors are considering mostly the deployment of big Python applications to different systems.
But for most sysadmin types, we just want to throw a few CLI tools on the system (ansible, awscli, etc.), and throwing another layer of abstraction on top of system Python is just annoying (and requires extra automation steps to manage).
If it works for you, great. As a warning, you can end up with some interesting problems when you try to update the system. There's a reason the file you deleted has the same effect as using `--break-system-packages`. (Of course you also lose the ability to use multiple versions of a dependency too).
As another alternatives, for a one-off thing I just use `python3 -m venv venv` to create a local `./venv` directory in the project, and `. ./venv/bin/activate` to use it (or just do `./venv/bin/{pip,python}` commands directly). If you want still use those system packages you installed, make it `python -m venv --system-site-packages venv` instead.
Do you update the code in github for your internet-pi Project??
Not frequently, I typically update it once a year or so (sometimes more frequently if there's a major bug).
I habe asked, because i have exactly these error in your internet-pi playbook. I have also tried to delete the file EXTERNALLY-MANAGED… also nof working :-(
Sadly same here... :-(
Thanks for this, and for the nay sayers out there, there are situations where there is no apt install for a module and setting up this virtual env does not make sense. For example, I have a Raspi 2, that monitors my brine tank and emails updates on level and room temp, that all, no development etc. So it really is a case by case situation that we need to examine each situation to see what makes sense. (wish Adafruit would do an apt install for circuit python)
On a side note, is pip going the way of the do do ? Ive been seeing similar direction with octoprint development seeming trying to avoid pip.
No-no-no, this is not a "solution", this is a workaround... The only solution is to do what the message is telling you, which is to install stuff in virtual environments so that they don't risk messing with your distro's python packages
No.
It's my system and I should be allowed to run it the way I want to.
I'm getting back to an INKY Impression hat on a Pi Zero, and had this issue. Wasted two hours. I saw Jeff Geerling had this issue, and I am completely unsurprised that the issue is now resolved. Thanks, Jeff!
This seems to break the --user option too, which can be worked around by deleting/moving the /usr/lib/python3.11/EXTERNALLY-MANAGED file.
Jeff... you are The Dude. Love your posts. Love your YouTube channel. Really value everything you git back to the community.
Thanks alot haha
I am running into this problem trying to install the latest paho-mqtt or gpiozero on a new pi running bookworm. So I temporarily renamed the EXTERNALLY-MANAGED file. Not fully understanding the “breaking” mechanics I assume the danger simply lies in new versions of dependancies that might come along with paho-mqtt and gpiozero packages. Assuming that is correct (??), I might also assume such widely used packages might rely on well churned dependancies and therefore carry little risk. Am I correctly looking at this?
This was a beneficial and entertaining forum. Thank you for your help! :)
-Luke G
Tons of comments all over the place on this breaking change.
IMHO I'm with Jeff on this. I just want to get something installed and don't want to figure anything new and different out.
Effectively the distro's are the root of the problem having made half a change. They figured out how to keep people from running pip install. Great for them.
But not great for us. They need to solve the second half. Install a pip modified to install in a separate venv that is used by default and the path is added to $PATH already.
Then pip install would work, mostly. There still would be the issue of needing to do pip install for each user. But that is far simpler than forcing everyone to figure out a new process etc etc.
> The only solution is to do what the message is telling you, which is to install stuff in virtual environments so that they don't risk messing with your distro's python packages
What distro packages? What risk? Many use Python inside docker containers. Installing this for the whole distro inside the container, is intentional, and the risk is zero. Adding a second layer of protection via virtual environments, with all the cost that adds to both container build, container usage, to protect against a non-existing risk, makes no sense.
If you have this the issue on Raspberry pi 2 model B, you must do a
pip cache purge
sudo apt update
pip install --upgrade pip
in order to download ansible in raspbian but i got problems with
sys-platform (=="darwin") ; extra == 'objc'