如何使用 databricks 上的 init bash 脚本从私有 git repo 安装 python package？

Question

I'm trying to pip install a python package from a private github repo using a init.sh script that I uploaded to my s3 bucket.

This is my init.sh file

#!/bin/bash
TOKEN={{secrets/private-repo/github}}
pip install git+https://${TOKEN}@github.com/<path-to-repo>

When I try to create my cluster I get the following error messsage. Init script failure: Cluster scoped init script s3://<s3_bucket>/init.sh failed: Script exit status is non-zero

I create a secret through the API with scope and key as private-repo and github . I tested this using a notebook and it worked fine.

Documentation Used: https://docs.databricks.com/security/secrets/secrets.html#reference-a-secret-in-an-environment-variable

Answer 1

The problem is that you're trying to refer to the secret using the {{secrets/private-repo/github}} syntax, but it doesn't work from the inside of the init script.

You need to define an environment variable on the cluster level and use that secret syntax there, and then it will be available inside your init script. See documentation on that topic.