nrfutil - “ImportError: No module named main” on Nixos

Question

I'm using the tool nrfutil which is implemented in Python. To be able to use it under NixOS I was using a default.nix file, that installed nrfutil into a venv. This worked for some time very well. (The last build on the build server using Nix within an alpine container could build the software I'm working on 11 days ago successfully.) When I do exactly the same things (ie restarting the CI server build without changes), the build fails now complaining about pip being incorrect:

$ nix-shell 
New python executable in /home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7
Not overwriting existing python script /home/matthias/source/tbconnect/bootloader/.venv/bin/python (you must use /home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7)
Installing pip, wheel...
done.
Traceback (most recent call last):
  File "/home/matthias/source/tbconnect/bootloader/.venv/bin/pip", line 6, in <module>
    from pip._internal.main import main
ImportError: No module named main

To me it seems that the module main should exist:

$ ls -l .venv/lib/python2.7/site-packages/pip/_internal/main.py
-rw-r--r-- 1 matthias matthias 1359 10月 15 12:27 .venv/lib/python2.7/site-packages/pip/_internal/main.py

I'm not very much into the Python environment, so I don't know any further. Has somebody any pointer for me where to continue debugging? How is Python resolving modules? Why doesn't it find the module, that seems to be present to me?

This is my default.nix that I use to install pip :

with import <nixpkgs> {};
with pkgs.python27Packages;

stdenv.mkDerivation {
  name = "impurePythonEnv";
  buildInputs = [
    automake
    autoconf
    gcc-arm-embedded-7
    # these packages are required for virtualenv and pip to work:
    #
    python27Full
    python27Packages.virtualenv
    python27Packages.pip
    # the following packages are related to the dependencies of your python
    # project.
    # In this particular example the python modules listed in the
    # requirements.txt require the following packages to be installed locally
    # in order to compile any binary extensions they may require.
    #
    taglib
    openssl
    git
    stdenv
    zlib ];
  src = null;
  shellHook = ''
    # set SOURCE_DATE_EPOCH so that we can use python wheels
    SOURCE_DATE_EPOCH=$(date +%s)
    virtualenv --no-setuptools .venv
    export PATH=$PWD/.venv/bin:$PATH
    #pip install nrfutil
    pip help

    # the following is required to build micro_ecc_lib_nrf52.a in the SDK
    export GNU_INSTALL_ROOT="${gcc-arm-embedded-7}/bin/"

    unset CC
  '';
}

• I replaced pip install nrfutil with pip help to make sure the problem is not the package I try to install itself.
• I'm still using python 2.7 as the nrfutil still is not fit for Python 3.
• Anyway replacing python27 with python37 did not change the error I get when trying to start pip .)
• NixOS version used locally is 19.09. Nix in the CI docker container is nixos/nix:latest which is the nix package manager on Alpine Linux.

Update:

Actually it works when I replace the call to pip install nrfutil with python2.7 -m pip install nrfutil . This actually confuses me even more. python2.7 is exactly the binary that is in the shebang of pip:

[nix-shell:~/source/tbconnect/bootloader]$ type python2.7
python2.7 is /home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7

[nix-shell:~/source/tbconnect/bootloader]$ type pip
pip is /home/matthias/source/tbconnect/bootloader/.venv/bin/pip

[nix-shell:~/source/tbconnect/bootloader]$ head --lines 2 .venv/bin/pip
#!/home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7
# -*- coding: utf-8 -*-

Update 2: I found out that another way to fix the problem is to edit .venv/bin/pip . This script tried the following import:

from pip._internal.main import main

Which I think is the new module path starting with pip 19.3. But I still have pip 19.2. When I change this line to:

from pip._internal import main

Running pip by typing pip is working.

The thing is I have no idea why the pip script is trying to load the new module path while NixOS still has the old version of pip .

I also opened an issue for NixOS on GitHub: https://github.com/NixOS/nixpkgs/issues/71178

Answer 1

I got your shell derivation to work by dropping the Python27Packages.pip ,

(nix-shell) 2d [azul:/tmp/lixo12333] $ 
 >>> pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package          Version
---------------- -------
behave           1.2.6  
Click            7.0    
crcmod           1.7    
ecdsa            0.13.3 
enum34           1.1.6  
future           0.18.2 
intelhex         2.2.1  
ipaddress        1.0.23 
libusb1          1.7.1  
linecache2       1.0.0  
nrfutil          5.2.0  
parse            1.12.1 
parse-type       0.5.2  
pc-ble-driver-py 0.11.4 
piccata          1.0.1  
pip              19.3.1 
protobuf         3.10.0 
pyserial         3.4    
pyspinel         1.0.0a3
PyYAML           4.2b4  
setuptools       41.6.0 
six              1.12.0 
tqdm             4.37.0 
traceback2       1.4.0  
virtualenv       16.4.3 
wheel            0.33.6 
wrapt            1.11.2 
(nix-shell) 2d [azul:/tmp/lixo12333] $ 

and my default.nix

with import <nixpkgs> {};
with pkgs.python27Packages;

stdenv.mkDerivation {
  name = "impurePythonEnv";
  buildInputs = [
    automake
    autoconf
    gcc-arm-embedded-7
    # these packages are required for virtualenv and pip to work:
    #
    python27Full
    python27Packages.virtualenv
    # the following packages are related to the dependencies of your python
    # project.
    # In this particular example the python modules listed in the
    # requirements.txt require the following packages to be installed locally
    # in order to compile any binary extensions they may require.
    #
    taglib
    openssl
    git
    stdenv
    zlib ];
  src = null;
  shellHook = ''
    # set SOURCE_DATE_EPOCH so that we can use python wheels
    SOURCE_DATE_EPOCH=$(date +%s)
    virtualenv .venv
    export PATH=$PWD/.venv/bin:$PATH
    pip install nrfutil
    #pip help

    # the following is required to build micro_ecc_lib_nrf52.a in the SDK
    export GNU_INSTALL_ROOT="${gcc-arm-embedded-7}/bin/"

    unset CC
  '';
}