使用 Rapids.ai 0.11+ 版将 cuDF 和 cuML 安装到 Colab

Question

我正在尝试将带有 cuDF 和 cuML 的 Rapids 库安装到 Colab 会话，并根据此示例执行代码：从在 Googe Colab 笔记本上安装 RAPIDS 库

!wget -nc https://raw.githubusercontent.com/rapidsai/notebooks-contrib/890b04ed8687da6e3a100c81f449ff6f7b559956/utils/rapids-colab.sh
!bash rapids-colab.sh

import sys, os

dist_package_index = sys.path.index("/usr/local/lib/python3.6/dist-packages")
sys.path = sys.path[:dist_package_index] + ["/usr/local/lib/python3.6/site-packages"] + sys.path[dist_package_index:]```
sys.path
if os.path.exists('update_pyarrow.py'): ## This file only exists if you're using RAPIDS version 0.11 or higher
  exec(open("update_pyarrow.py").read(), globals())

在安装过程中，我收到此错误：

  - cudf=0.11

Current channels:

  - https://conda.anaconda.org/rapidsai-nightly/label/xgboost/linux-64
  - https://conda.anaconda.org/rapidsai-nightly/label/xgboost/noarch
  - https://conda.anaconda.org/rapidsai-nightly/linux-64
  - https://conda.anaconda.org/rapidsai-nightly/noarch
  - https://conda.anaconda.org/nvidia/linux-64
  - https://conda.anaconda.org/nvidia/noarch
  - https://conda.anaconda.org/conda-forge/linux-64
  - https://conda.anaconda.org/conda-forge/noarch
  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/free/linux-64
  - https://repo.anaconda.com/pkgs/free/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/pro/linux-64
  - https://repo.anaconda.com/pkgs/pro/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

我曾尝试分别安装 cuDF 和 cuML

conda install -c rapidsai -c nvidia -c conda-forge \
    -c defaults cudf=0.12 python=3.6 cudatoolkit=10.0

但仍然收到错误：

ModuleNotFoundError Traceback (most recent call last)

<ipython-input-10-a95ca25217db> in <module>()
----> 1 import cudf
      2 import io, requests
      3 
      4 # download CSV file from GitHub
      5 url="https://github.com/plotly/datasets/raw/master/tips.csv"

ModuleNotFoundError: No module named 'cudf'

如何解决这个错误？

Answer 1

更新（2020 年 12 月 21 日）：要直接在线使用GPU 驱动的 RAPIDS 笔记本，您可以使用BlazingSQL （RAPIDS 0.15+）或继续使用 Colabatory（仅限 RAPIDS 0.14）

更新（2/19/2020）：回到这个问题，Colab 正在工作 @try。 玩得开心！

如果您有任何其他问题，请告诉我们。 如果您需要更新您的个人 Colab 笔记本，请使用此脚本安装 RAPIDS：

# Install RAPIDS
!git clone https://github.com/rapidsai/rapidsai-csp-utils.git
!bash rapidsai-csp-utils/colab/rapids-colab.sh

import sys, os

dist_package_index = sys.path.index('/usr/local/lib/python3.6/dist-packages')
sys.path = sys.path[:dist_package_index] + ['/usr/local/lib/python3.6/site-packages'] + sys.path[dist_package_index:]
sys.path
exec(open('rapidsai-csp-utils/colab/update_modules.py').read(), globals())

之前的回复：

我们正在中间或将我们的 Colab 脚本转换为新的存储库。 我们应该尽快更新所有笔记本，并尝试帮助其他人迁移。 就像在 24 小时内一样，如果不是今天 PST 的 EOD。

Answer 2

运行@TaureanDyerNV RAPIDS 的代码后，建议更改以下代码。 运行需要15分钟。

!git clone https://github.com/rapidsai/rapidsai-csp-utils.git
!bash rapidsai-csp-utils/colab/rapids-colab.sh 0.19

import sys, os, shutil

sys.path.append('/usr/local/lib/python3.7/site-packages/')
os.environ['NUMBAPRO_NVVM'] = '/usr/local/cuda/nvvm/lib64/libnvvm.so'
os.environ['NUMBAPRO_LIBDEVICE'] = '/usr/local/cuda/nvvm/libdevice/'
os.environ['CONDA_PREFIX'] = '/usr/local'
for so in ['cudf', 'rmm', 'nccl', 'cuml', 'cugraph', 'xgboost', 'cuspatial']:
    fn = 'lib'+so+'.so'
    source_fn = '/usr/local/lib/'+fn
    dest_fn = '/usr/lib/'+fn
    if os.path.exists(source_fn):
        print(f'Copying {source_fn} to {dest_fn}')
        shutil.copyfile(source_fn, dest_fn)
if not os.path.exists('/usr/lib64'):
    os.makedirs('/usr/lib64')
for so_file in os.listdir('/usr/local/lib'):
    if 'libstdc' in so_file:
        shutil.copyfile('/usr/local/lib/'+so_file, '/usr/lib64/'+so_file)
        shutil.copyfile('/usr/local/lib/'+so_file, '/usr/lib/x86_64-linux-gnu/'+so_file)