使用我自己的Azure Pyspark中的python模块，该模块读取并准备数据

Question

My jupyter notebook, reads a module I created, in Jupyter I add

sc.addPyFile('wasb:///HdiNotebooks/PySpark/project/read_test_data.py')

and it loads the module ok,

However, my "py" file, opens the data as:

data_file= open('wasb:///example/data/fruits.txt', 'rU') 
to prepare it and do different calculations.

However, I get the following error

[Errno 2] No such file or directory: 'wasb:///example/data/fruits.txt'
Traceback (most recent call last):
FileNotFoundError: [Errno 2] No such file or directory: 'wasb:///example/data/fruits.txt'

If I tried to create a dataframe with the same data in jupyter I run

df=sqlContext.read.csv('wasb:///example/data/fruits.txt',header='true', inferSchema='true')

And I dont get any errors. What I'm doing wrong?

Answer 1

Python API open does not support the wasb protocol for HDInsight DFS based on Azure Blob Storage.

If you want to directly read the file on HDInsight without pyspark, the only way is using Azure Storage SDK for Python to do it with account_name & account_key of Azure Blob Storage Account for HDInsight like the document said, and please refer to the offical tutorial for Azure Storage in Python.

Hope it helps.