Why can't I install cudatookkit=10.1 using pip but can using conda?

Question

$ pip install cudatoolkit==10.1
Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not find a version that satisfies the requirement cudatoolkit==10.1 (from versions: none)
ERROR: No matching distribution found for cudatoolkit==10.1

This failed. However, the conda works:

$ conda install cudatoolkit=10.1
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /data0/congmin/miniconda

  added / updated specs:
    - cudatoolkit=10.1


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2021.5.25  |       h06a4308_1         112 KB
    certifi-2021.5.30          |   py38h06a4308_0         138 KB
    conda-4.10.1               |   py38h06a4308_1         2.9 MB
    cudatoolkit-10.1.243       |       h6bb024c_0       347.4 MB
    openssl-1.1.1k             |       h27cfd23_0         2.5 MB
    ------------------------------------------------------------
                                           Total:       353.1 MB

I need to use pip to install, because I want to put it in requirements.txt.

Answer 1

pip relies on https://pypi.org/ as the default package repository.

Anaconda has it's own repository

cudatoolkit is not a package in pypi. So pip cannot find any such package.

It is available in the conda repository.

The reason why cudatoolkit is not available in pypi is because it's not a python packge. It is a toolkit from nvidia that needs a C compiler to exist in your system. Pip was never intended to handle such cases, whereas Anaconda is.

See this blog post from Anaconda for details:

This highlights a key difference between conda and pip. Pip installs Python packages whereas conda installs packages which may contain software written in any language.

As far as adding to requirements.txt is concerned. If you are using conda in your target system as well you could just export your conda environment to an environment.yml file and then clone the envrionment in your target machine.