自定义 Azure Synapse Workspace Spark 集群的节点

Question

When creating a Spark cluster within an Azure Synapse workspace , is there a means to install arbitrary files and directories onto it's cluster nodes and/or onto the node's underlying distributed filesystem ?

By arbitrary files and directories, I literally mean arbitrary files and directories; not just extra Python libraries like demonstrated here .

Databricks smartly provided a means to do this on it's cluster nodes (described in this document ). Now I'm trying to see if there's a means to do the same on an Azure Synapse Workspace Spark Cluster .

Thank you.

Answer 1

Unfortunately, Azure Synapse Analytics don't support arbitrary binary installs or writing to Spark local storage.

I would suggest you to provide feedback on the same:

https://feedback.azure.com/forums/307516-azure-synapse-analytics

All of the feedback you share in these forums will be monitored and reviewed by the Microsoft engineering teams responsible for building Azure.