python3 subprocess pip “ImportError: cannot import name main” in terminal

Question

I created a script (see below) to upgrade all my pip packages. I successfully executed my script via idle3 , ie open the script using idle3 and pressing F5 to run the script as a module. However, I am not able to execute it in the terminal; got the below error. How do I overcome this error? Why does the import error happen in terminal but not in idle3 ?

$ python3 -m upgrade_pip_packages 
====================================================
UPGRADING ALL --USER PIP PACKAGES TO LATEST VERSION:
====================================================
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main
ERROR: Command 'pip list' returned non-zero exit status 1

My script : upgrade_pip_packages.py

#!/bin/python3
import subprocess
from pprint import pprint


def get_pkgs():
    try:
        cmd = 'pip list'
        completed = subprocess.run( cmd, shell=True, check=True,
                                    stdout=subprocess.PIPE )
    except subprocess.CalledProcessError as err:
        print( 'ERROR:', err )
    else:
        for line in completed.stdout.decode('utf-8').splitlines()[2:]:
            yield line


def update_pkgs(piplist):
    npackages = 0
    nupgrades = 0
    nerrors = 0
    upgradelist = []
    errorlist = []
    for i in piplist:
        npackages += 1
        pkgname, ver = i.split()
        print('\n',pkgname)
        try:
            cmd = 'pip install --user {} --upgrade'.format(pkgname)
            completed = subprocess.run( cmd, shell=True, check=True,
                                        stdout=subprocess.PIPE )
        except subprocess.CalledProcessError as err:
            nerrors += 1
            errorlist.append(pkgname)
            print( 'ERROR: {}'.format(err) )
        else:
            for line in completed.stdout.decode('utf-8').splitlines():
                print(line)
                if 'Successfully installed' in line:
                    nupgrades +=1
                    upgradelist.append(pkgname)
    return npackages, nupgrades, nerrors, upgradelist, errorlist


def main():
    print('====================================================')
    print('UPGRADING ALL --USER PIP PACKAGES TO LATEST VERSION:')
    print('====================================================')
    pip_pkgs = get_pkgs() # created a generator
    npackages, nupgrades, nerrors, upgradelist, errorlist \
               = update_pkgs(pip_pkgs)
    print('\nNo. of --user pip packages = {}'.format(npackages))
    print('No. of upgrades            = {}'.format(nupgrades))
    print('No. of upgrade errors      = {}'.format(nerrors))
    if upgradelist:
        print('Package(s) upgraded:')
        pprint(upgradelist)
    if errorlist:
        print('Package(s) with upgrade error:')
        pprint(errorlist)
    print()


if __name__ == '__main__':
    main()

Answer 1

Final upgrade_pip_packages.py .

I found the answer to my question. Essentially, my script had to:

import sys

and make the following amendments:

cmd = [sys.executable, '-m', 'pip', 'list'] #Change here
completed = subprocess.run( cmd,
                            #shell=True, #switch this off
                            check=True,
                            stdout=subprocess.PIPE )

and

cmd = [sys.executable, '-m', 'pip', 'install', '--user', pkgname, '--upgrade'] #Change here
completed = subprocess.run( cmd,
                            #shell=True, #switch this off
                            check=True,
                            stdout=subprocess.PIPE )

PyPA documentation explanation:

It's recommended to write {sys.executable} rather than plain python in order to ensure that commands are run in the Python installation matching the currently running notebook (which may not be the same Python installation that the python command refers to).

$ pip --version
pip 18.1 from ~/.local/lib/python3.5/site-packages/pip (python 3.5)
$ pip3 --version
pip 18.1 from ~/.local/lib/python3.5/site-packages/pip (python 3.5)