需要为 Airflow 与 Azure 连接的有效负载获取正确的 URL

Question

我有四个文件main.py 、 jobs.zip 、 libs.zip和params.yaml ，这些我存储在 Azure 存储帐户容器中。

现在我有了这段代码，它正在制作一个有效负载，并将尝试使用该有效负载运行一个 spark 作业。 该有效负载将具有这 4 个文件的位置链接。

hook = AzureSynapseHook(
            azure_synapse_conn_id=self.azure_synapse_conn_id, spark_pool=self.spark_pool
        )

payload = SparkBatchJobOptions(
            name=f"{self.job_name}_{self.app_id}",
            file=f"abfss://{Variable.get('ARTIFACT_BUCKET')}@{Variable.get('ARTIFACT_ACCOUNT')}.dfs.core.windows.net/{self.env}/{SPARK_DIR}/main.py",
            arguments=self.job_args,
            python_files=[
                f"abfss://{Variable.get('ARTIFACT_BUCKET')}@{Variable.get('ARTIFACT_ACCOUNT')}.dfs.core.windows.net/{self.env}/{SPARK_DIR}/jobs.zip",
                f"abfss://{Variable.get('ARTIFACT_BUCKET')}@{Variable.get('ARTIFACT_ACCOUNT')}.dfs.core.windows.net/{self.env}/{SPARK_DIR}/libs.zip",
            ],
            files=[
                f"abfss://{Variable.get('ARTIFACT_BUCKET')}@{Variable.get('ARTIFACT_ACCOUNT')}.dfs.core.windows.net/{self.env}/{SPARK_DIR}/params.yaml"
            ],
        )

self.log.info("Executing the Synapse spark job.")
response = hook.run_spark_job(payload=payload)

我已经检查了正确的位置链接，但是当我在 airflow 上运行它时，它会抛出一个与有效负载相关的错误，我认为它试图说它无法获取链接。

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/azure/core/pipeline/transport/_base.py", line 579, in format_url
    base = self._base_url.format(**kwargs).rstrip("/")
KeyError: 'endpoint'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/airflow/dags/operators/spark/__init__.py", line 36, in execute
    return self.executor.execute()
  File "/usr/local/airflow/dags/operators/spark/azure.py", line 60, in execute
    response = hook.run_spark_job(payload=payload)
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/microsoft/azure/hooks/synapse.py", line 144, in run_spark_job
    job = self.get_conn().spark_batch.create_spark_batch_job(payload)
  File "/usr/local/lib/python3.9/site-packages/azure/synapse/spark/operations/_spark_batch_operations.py", line 163, in create_spark_batch_job
    request = self._client.post(url, query_parameters, header_parameters, **body_content_kwargs)
  File "/usr/local/lib/python3.9/site-packages/azure/core/pipeline/transport/_base.py", line 659, in post
    request = self._request(
  File "/usr/local/lib/python3.9/site-packages/azure/core/pipeline/transport/_base.py", line 535, in _request
    request = HttpRequest(method, self.format_url(url))
  File "/usr/local/lib/python3.9/site-packages/azure/core/pipeline/transport/_base.py", line 582, in format_url
    raise ValueError(err_msg.format(key.args[0]))
ValueError: The value provided for the url part endpoint was incorrect, and resulted in an invalid url

我还想知道abfss和wasbs的区别以及我应该在哪里上传我的文件以便代码能够获取链接？

也许我在错误的地方上传文件。

Answer 1

连接self.azure_synapse_conn_id有问题，其中主机（Synapse Workspace URL）无效，这是连接示例：

Connection(
    conn_id=DEFAULT_CONNECTION_CLIENT_SECRET,
    conn_type="azure_synapse",
    host="https://testsynapse.dev.azuresynapse.net",
    login="clientId",
    password="clientSecret",
    extra=json.dumps(
        {
            "extra__azure_synapse__tenantId": "tenantId",
            "extra__azure_synapse__subscriptionId": "subscriptionId",
        }
    ),
)

对于abfss和wasbs之间的区别，这里有关于该主题的详细解答。