Open file on dbfs while using databricks-connect

Question

I am using databricks-connect to connect my local PyCharm IDE to an Azure Databricks cluster.

This is working completely fine until I try to access files without a Spark Context.

For example

dbutils.fs.put('dbfs:/tmp/test_file.txt','line_1')
with open('/dbfs/tmp/test_file.txt') as f:
  print(f.read())

is working fine when being run directly in the databricks notebook. When I try to run this code snippet via databricks-connect in PyCharm I get a FileNotFoundError. This also happens for other file system operations (shutil, gzip, ...).

I assume that the program ("open") is trying to access the file system on my local client running PyCharm, but I want "open" and similar functions to access the dbfs.

Is there a way to achieve this?

Answer 1

the open function belongs to the Python file API, so it works only with local files, and in case of the databricks-connect - this is a file system of your computer. The /dbfs/ mount point is available only on the cluster nodes.