Package updates on the system affecting virtual environment?

Question

I am in a virtual environment. I verified that in my active terminal using which python and can confirm that I am in fact in a virtual environment. Printing pip list , according to the official documentation, should list the packages in this virtual environment. Here's an output:

Package                            Version            
---------------------------------- -------------------
alabaster                          0.7.10             
anaconda-client                    1.6.5         
... (truncated)
pip                                10.0.1             

You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

When I fire up another terminal, this time making sure I'm outside of that virtual environment and proceed to upgrade pip :

pip install --upgrade pip

Upon doing that, I verified that my pip package on the system has been updated to version 18.0. Here is the confusing part: I switched back into the virtual environment and use pip list and the pip version in my virtual environment is now pip 18.0.

Why is it that upgrading the pip version outside of that environment subsequently update the pip in my virtual environment from 10.0.1 at all? Have I misunderstood how virtual environments work? I'm not new to python but have not used virtual environment so forgive me if it's something very fundamental. In my understanding the primary value in using virtual envs is that I can be upgrading my system-wide packages (such as pip, flask etc) without any of that changes being affected in my virtual environments. Virtual environments should be isolated environment at all?

If it matters, I'm using the default venv and not virtualenvwrapper or any other wrapper tool.

Answer 1

The Problem

tl;dr : The problem is with conda. I use conda and apparently that causes some issues with managing and installing packages in virtual environment created via venv because a local instance of pip was not present.

Solution A: conda install -n myenv pip where myenv refers to the name of your virtual environment

Solution B: Use conda list , conda create to work with environments in a way that is 100% compatible with conda

/end tl;dr

Here's a breakdown of the problem. When I'm using an anaconda's version of python and I decide to look at the list of packages I have in my environment; Supposed I were to call pip freeze or pip list , whether I'm in a virtual environment or not wouldn't matter. It returns exactly the same list of packages from conda's corresponding site-packages folder.

When I'm inside a virtual environment, running which python does seem to point to an isolated version of the python instance ( acco is the name of that instance):

(acco) Samuels-MacBook-Pro:Accomplish Samuel$ which python
/Users/Samuel/Dropbox/Projects/Python/Acco/acco/bin/python 

However, while in this virtual environment, running pip list or pip list --local would still refer to the same set of packages - and that is because it is still point to the conda's version of package directory (Yes, even in a virtual environment).

Specifically, with or without, inside or outside of a virtual environment, the pip list points to packages installed in the /anaconda3/lib/../site-packages directory:

import sys
sys.prefix
'/Users/Samuel/anaconda3'    

import site
site.getsitepackages()
['/Users/Samuel/anaconda3/lib/python3.6/site-packages']

The really problematic part of this is that while in the virtual environment, you have essentially no libraries installed. Installing from requirements.txt' using pip install -r requirements.txt or just installing packages at all using plain old pip install` wouldn't work if any of the packages you're attempting to install already exist at the conda's directory. Instead, you'll get a message that doesn't look like an error, and it stops just right there. The package you're attempting to install is not installed into the local directory.

# same as pip install Flask==0.12.2
pip install -r requirements.txt 
Requirement already satisfied: Flask==0.12.2 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (0.12.2)
Requirement already satisfied: Werkzeug>=0.7 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (0.12.2)
Requirement already satisfied: Jinja2>=2.4 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (2.9.6)
Requirement already satisfied: itsdangerous>=0.21 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (0.24)
Requirement already satisfied: click>=2.0 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (6.7)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Jinja2>=2.4->Flask==0.12.2->-r requirements.txt (line 1)) (1.0)

As a reminder, we're still executing your python commands using the python in our virtual environment ( ../Dropbox/Projects/Python/Acco/acco/bin/python ) and not the conda distribution. This virtual environment has no packages in its isolated lib folder, and you can't install any libraries into it as pip will be stopped with the "requirement already satisfied" message and exit (or terminate its attempt).

This means that while in that virtual environment, trying to run a python script or app that has dependencies will certainly fail. Building from the above example, your Flask app app.py won't run because it couldn't find flask . Goes without saying, since your virtual environment has no packages and it won't let you install any.

---

The Solution

The solution is that if you're using a conda distribution of python, check the packages you have installed on your system using conda list instead of pip list for maximal consistency.

(acco) Samuels-MacBook-Pro:Acco Samuel$ conda list
# packages in environment at /Users/Samuel/anaconda3:
#
# Name                    Version                   Build  Channel
_ipyw_jlab_nb_ext_conf    0.1.0            py36h2fc01ae_0
_r-mutex                  1.0.0                     mro_2
alabaster                 0.7.10           py36h174008c_0
anaconda                  custom           py36ha4fed55_0

Notice that conda tells you the environment from which the libraries are listed from. At this stage:

1. Deactivate the environment using deactive
2. Make virtual environment using conda create --name acco python=3.6.3 flask sqlite , here we're using acco as the virtual environment's name, a specific version of python, and optionally some other dependencies
3. Activate the environment using source activate acco

Now, when you do conda list again, being in your virtual environment:

# packages in environment at /Users/Samuel/anaconda3/envs/acco:
#
# Name                    Version                   Build  Channel
ca-certificates           2018.03.07                    0
certifi                   2018.8.13                py36_0
click                     6.7              py36hec950be_0
flask                     1.0.2                    py36_1

If you want to manage the dependencies of your app using an old-school requirement.txt file, then simplest way is to conda list --export > requirements.txt . From this point on, use conda list in place of pip list and use conda create as well as source activate in place of its venv counterpart.

The anaconda's main documentation suggest:

To use pip in your environment, in your Terminal window or an Anaconda Prompt, run:

 conda install -n acco pip source activate acco pip <pip_subcommand> 

This solution worked too. Following the steps above, I ended up with:

(acco) Samuels-MacBook-Pro:Accomplish Samuel$ pip list
Package      Version
------------ ---------
certifi      2018.8.13
click        6.7
Flask        1.0.2
itsdangerous 0.24
Jinja2       2.10
MarkupSafe   1.0
pip          10.0.1
setuptools   40.0.0
Werkzeug     0.14.1
wheel        0.31.1

As a bonus, I double-checked to see that a conda environment created this way would be able to handle pip list or pip install : same error. However, when using conda update (for example: conda update sqlite ), conda install or any of the conda-counterpart code, they all work as fully intended. Updating sqlite in your local environment update exactly that copy of sqlite and not the one in your anaconda system. Whew!