通过 Python 中的 Databricks api 读取 Databricks 表？

Question

Using Python-3, I am trying to compare an Excel (xlsx) sheet to an identical spark table in Databricks. I want to avoid doing the compare in Databricks. So I am looking for a way to read the spark table via the Databricks api. Is this possible? How can I go on to read a table: DB.TableName?

Answer 1

I can recomend you write pyspark code in notebook, call the notebook from previously defined job, and establish connection between your local machine and databricks workspace.

You could perfom comaprision directly on spark or convert data frames to pandas if you wish. If noteebok will end comaprision, could retrun result from particular job. I think that sending all databricks tables could be impossible because of API limitation you have spark cluster to perform complex operation, API should be use to send small messages.

Officical documentation: https://docs.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/jobs#--runs-get-output

Retrieve the output and metadata of a run. When a notebook task returns a value through the dbutils.notebook.exit() call, you can use this endpoint to retrieve that value. Azure Databricks restricts this API to return the first 5 MB of the output. For returning a larger result, you can store job results in a cloud storage service.

Answer 2

There is no way to read the table from the DB API as far as I am aware unless you run it as a job as LaTreb already mentioned. However, if you really wanted to, you could use either the ODBC or JDBC drivers to get the data through your databricks cluster.

Information on how to set this up can be found here .

Once you have the DSN set up you can use pyodbc to connect to databricks and run a query. At this time the ODBC driver will only allow you to run Spark-SQL commands.

All that being said, it will probably still be easier to just load the data into Databricks, unless you have some sort of security concern.