火花错误地读取CSV

Question

我想在spark中读取train.csv，但似乎spark以某种方式错误地读取了文件。 我使用python将csv读入了pandas，它显示了正确的值1作为project_is_approved中的第一项。 当我用spark（scala）读取csv时，我得到了一个字符串，大概是从数据集中的其他地方获取的。

为什么会这样呢？ 大多数示例使用我用来读取csv的语法。

jakeu123@azure3:~$ python
Python 2.7.12 (default, Dec  4 2017, 14:50:18) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> f = requests.get("https://www.dropbox.com/s/2hdbltrl8bh6kbu/train.csv?raw=1", stream=True)
>>> with open("train.csv", "w") as csv:
...     csv.write(f)
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
TypeError: expected a string or other character buffer object
>>> with open("train.csv", "w") as csv:
...     csv.write(f.content)
... 
>>> import pandas as pd
>>> df = pd.read_csv("train.csv")
>>> df[["project_is_approved"]].head(1)
   project_is_approved
0                    1
>>> 
jakeu123@azure3:~$ ./spark/bin/spark-shell
2018-06-07 23:55:02 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
2018-06-07 23:55:09 WARN  Utils:66 - Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
2018-06-07 23:55:09 WARN  Utils:66 - Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
Spark context Web UI available at http://azure3:4042
Spark context available as 'sc' (master = local[*], app id = local-1528415709241).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.3.0
      /_/

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_171)
Type in expressions to have them evaluated.
Type :help for more information.

scala> val df = spark.read.option("header", true).csv("train.csv")
2018-06-07 23:55:27 WARN  ObjectStore:568 - Failed to get database global_temp, returning NoSuchObjectException
df: org.apache.spark.sql.DataFrame = [id: string, teacher_id: string ... 14 more fields]

scala> df.select($"project_is_approved").show(1)
+--------------------+                                                          
| project_is_approved|
+--------------------+
|I currently have ...|
+--------------------+
only showing top 1 row


scala> :quit

Answer 1

据我所知，spark无法直接读取URL格式的文件。 因此，与其使用python读取CSV文件并将其写入磁盘以供以后使用spark读取，还可以使用pandas，然后再将其转换为Spark Dataframe（通过使用dataframe可以在spark上获得分布式计算的好处）

我对Scala不太熟悉，所以我尝试使用pyspark解决它

import pandas as pd
from pyspark.sql.types import StringType, LongType

frame = pd.read_csv("https://www.dropbox.com/s/2hdbltrl8bh6kbu/train.csv?raw=1", index_col=None, header=0, encoding='utf-8') # Prod
frame = frame.where(pd.notnull(frame), None) # Fill NaN with Null

schema = StructType([
  StructField("id", StringType(), True),
  StructField("teacher_id", StringType(), True),
  StructField("teacher_prefix", StringType(), True),
  StructField("school_state", StringType(), True),
  StructField("project_submitted_datetime", StringType(), True),
  StructField("project_grade_category", StringType(), True),
  StructField("project_subject_categories", StringType(), True),
  StructField("project_subject_subcategories", StringType(), True),
  StructField("project_title", StringType(), True),
  StructField("project_essay_1", StringType(), True),
  StructField("project_essay_2", StringType(), True),
  StructField("project_essay_3", StringType(), True),
  StructField("project_essay_4", StringType(), True),
  StructField("project_resource_summary", StringType(), True),
  StructField("teacher_number_of_previously_posted_projects", LongType(), True),
  StructField("project_is_approved", LongType(), True),
                    ])
df = spark.createDataFrame(frame, schema)

哦，顺便说一句，我认为必须提供读取CSV文件的模式，因为它不会踢任何火花工作，因此您可以避免浪费计算资源，并且火花将以正确的格式读取文件

Answer 2

您需要定义转义字符，以便在解析时可以忽略文本内的逗号（，）

这可以做到

spark.read.option("escape","\"")

工作示例：

scala> val df = spark.read.option("header",true).option("escape","\"").csv("train.csv");

            df: org.apache.spark.sql.DataFrame = [id: string, teacher_id: string ... 14 more fields]

            scala> df.select($"project_is_approved").show
            +-------------------+
            |project_is_approved|
            +-------------------+
            |                  1|
            |                  0|
            |                  1|
            |                  0|
            |                  1|
            |                  1|
            |                  1|
            |                  1|
            |                  1|
            |                  1|
            |                  1|
            |                  1|
            |                  1|
            |                  0|
            |                  1|
            |                  1|
            |                  1|
            |                  1|
            |                  1|
            |                  0|
            +-------------------+
            only showing top 20 rows