Azure HDInsight - SparkSql：如何从数据帧中的 github 加载 csv 文件

Question

I am new to spark. I am trying to import csv data kept in github into dataframe in PySpark notebook in azure.

from pyspark.sql import SparkSession
url = "https://raw.githubusercontent.com/MicrosoftLearning/20775_Performing-Data-Engineering-on-Microsoft-HDInsight/master/Allfiles/Demofiles/Mod04/clidata/hvac/HVAC.csv"

spark = SparkSession \
    .builder \
    .appName("how to read csv file") \
    .getOrCreate()  
df = spark.read.csv(url)

I am getting below error message

An error occurred while calling o1298.csv.

: java.lang.UnsupportedOperationException
    at org.apache.hadoop.fs.http.AbstractHttpFileSystem.listStatus(AbstractHttpFileSystem.java:91)

I am seeking help from community.

Answer 1

Try this one

>>> from pyspark import SparkFiles
>>> url = "https://raw.githubusercontent.com/MicrosoftLearning/20775_Performing-Data-Engineering-on-Microsoft-HDInsight/master/Allfiles/Demofiles/Mod04/clidata/hvac/HVAC.csv"

>>> spark.sparkContext.addFile(url)

>>> df = spark.read.csv("file://"+SparkFiles.get("HVAC.csv"), header='True')

>>> df.count()
8000