在 databricks 集群中使用初始化腳本安裝 python 包

Question

我已經通過運行以下命令安裝了 databricks cli 工具

pip install databricks-cli 。 如果您使用的是 Python 3，請運行 pip3。

然后通過創建 PAT（Databricks 中的個人訪問令牌）我運行以下.sh bash 腳本：

# You can run this on Windows as well, just change to a batch files
# Note: You need the Databricks CLI installed and you need a token configued
#!/bin/bash
echo "Creating DBFS direcrtory"
dbfs mkdirs dbfs:/databricks/packages

echo "Uploading cluster init script"
dbfs cp --overwrite python_dependencies.sh                     dbfs:/databricks/packages/python_dependencies.sh

echo "Listing DBFS direcrtory"
dbfs ls dbfs:/databricks/packages

python_dependencies.sh 腳本

#!/bin/bash
# Restart cluster after running.

sudo apt-get install applicationinsights=0.11.9 -V -y
sudo apt-get install azure-servicebus=0.50.2 -V -y
sudo apt-get install azure-storage-file-datalake=12.0.0 -V -y
sudo apt-get install humanfriendly=8.2 -V -y
sudo apt-get install mlflow=1.8.0 -V -y
sudo apt-get install numpy=1.18.3 -V -y
sudo apt-get install opencensus-ext-azure=1.0.2 -V -y
sudo apt-get install packaging=20.4 -V -y
sudo apt-get install pandas=1.0.3 -V -y
sudo apt update
sudo apt-get install scikit-learn=0.22.2.post1 -V -y
status=$?
echo "The date command exit status : ${status}"

我使用上面的腳本在集群的 init-scripts 中安裝 python 庫

我的問題是，即使一切似乎都很好並且集群已成功啟動，但庫沒有正確安裝。 當我單擊集群的庫選項卡時，我得到以下信息：

僅安裝了 10 個 python 庫中的 1 個。

感謝您的幫助和評論。

Answer 1

我根據@RedCricket 的評論找到了解決方案，

#!/bin/bash

pip install applicationinsights==0.11.9
pip install azure-servicebus==0.50.2
pip install azure-storage-file-datalake==12.0.0
pip install humanfriendly==8.2
pip install mlflow==1.8.0
pip install numpy==1.18.3
pip install opencensus-ext-azure==1.0.2
pip install packaging==20.4
pip install pandas==1.0.3
pip install --upgrade scikit-learn==0.22.2.post1

above.sh 文件將安裝集群啟動時引用的所有 python 依賴項。 因此，重新執行筆記本時不必重新安裝庫。

Answer 2

根據文檔，對於 azure 數據塊

https://docs.microsoft.com/en-us/azure/databricks/dev-tools/cli/

# Set up authentication using an Azure AD token
export DATABRICKS_AAD_TOKEN=$(jq .accessToken -r <<< "$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d)")
# Databricks CLI configuration 
databricks configure --host "https://https://<databricks-instance>" --aad-token

現在，將腳本文件復制到 databricks 文件系統

databricks fs cp "./cluster-scoped-init-scripts/db_scope_init_script.sh" "dbfs:/databricks/init-scripts/db_scope_init_script.sh"

確保“db_scope_init_script.sh”shell 腳本具有所需的安裝命令。

最后，使用 DBFS REST API 配置集群范圍的初始化腳本

curl -n -X POST -H 'Content-Type: application/json' -d '{
  "cluster_id": "1202-211320-brick1",
  "num_workers": 1,
  "spark_version": "7.3.x-scala2.12",
  "node_type_id": "Standard_D3_v2",
  "cluster_log_conf": {
    "dbfs" : {
      "destination": "dbfs:/cluster-logs"
    }
  },
  "init_scripts": [ {
    "dbfs": {
      "destination": "dbfs:/databricks/scripts/db_scope_init_script.sh"
    }
  } ]
}' https://<databricks-instance>/api/2.0/clusters/edit