使用 databricks-connect 连接到 python 中的数据块时出错

Question

I'm using databricks-connect on mac using pycharm but after I finished the configuration and tried to run databricks-connect test , I got the following error and have no idea what the problem is. I followed this documentation: https://docs.databricks.com/user-guide/dev-tools/db-connect.html

The error message is as below:

scala> spa
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/bin/databricks-connect", line 11, in 
    load_entry_point('databricks-connect==5.3.1', 'console_scripts', 'databricks-connect')()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pyspark/databricks_connect.py", line 244, in main
    test()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pyspark/databricks_connect.py", line 213, in test
    raise ValueError("Scala command failed to produce correct result")
ValueError: Scala command failed to produce correct result

Answer 1

Maybe your Java/Python version does not comply. Check your cluster, which Python version does it use (in my case it was 3.5).

And what's most important: check which JDK version do you have on your computer. In my case, I've had the latest one, which was not supported by databricks-connect . It needs to run on JDK 8.

Answer 2

I would make sure you are using the right version of the Databricks Runtime (DB Connect only currently supports 5.1-5.5). Since these is a limit on the DBR that works with DB connect, you'll have to make sure you match the python version as well (for the base Databricks runtime, I believe it is 3.5.x).

Answer 3

To ignore the RUNTIME version, export an environment variable that resolves:

export DEBUG_IGNORE_VERSION_MISMATCH=1