使用Python从MS Azure读取和写入文件

Question

I am newbie to Python and Spark, I am trying to load file from Azure to table. Below is my simple code.

 import os import sys os.environ['SPARK_HOME'] = "C:\\spark-2.0.0-bin-hadoop2.74" sys.path.append("C:\\spark-2.0.0-bin-hadoop2.7\\python") sys.path.append("C:\\spark-2.0.0-bin-hadoop2.7\\python\\lib\\py4j-0.10.1-src.zip") from pyspark import SparkContext from pyspark import SparkConf from pyspark.sql.types import * from pyspark.sql import * sc = SparkContext("local", "Simple App") def loadFile(path, rowDelimeter, columnDelimeter, firstHeaderColName): loadedFile = sc.newAPIHadoopFile(path, "org.apache.hadoop.mapreduce.lib.input.TextInputFormat", "org.apache.hadoop.io.LongWritable", "org.apache.hadoop.io.Text", conf={"textinputformat.record.delimiter": rowDelimeter}) rddData = loadedFile.map(lambda l:l[1].split(columnDelimeter)).filter(lambda f: f[0] != firstHeaderColName) return rddData Schema= StructType([ StructField("Column1", StringType(), True), StructField("Column2", StringType(), True), StructField("Column3", StringType(), True), StructField("Column4", StringType(), True) ]) rData= loadFile("wasbs://Storagename@Accountname.blob.core.windows.net/File.txt", '\\r\\n',"#|#","Column1") DF = sc.createDataFrame(Data,Schema) DF.write.saveAsTable("Table1") 

I am getting error like FileNotFoundError: [WinError 2] The system cannot find the file specified

Answer 1

@Miruthan, As far as I know, If we'd like to read data from WASB into Spark, the URL syntax is as following :

wasb[s]://<containername>@<accountname>.blob.core.windows.net/<path>

Meanwhile, due to Azure Storage Blob (WASB) is used as the storage account associated with an HDInsight cluster.Could you please double check it? Any update, please let me know.