Exception while reading csv in spark

Question

I am very new to spark. I have csv file, with 2 columns only. The csv file is large (have 30 million rows). I am trying to use spark-csv_2.10:1.2.0 to load it to dataframes.

I am using the following code:

df = sqlContext.read.load('file:///path/file_third.csv', 
                           format='com.databricks.spark.csv', 
                           header='true', 
                           inferSchema='true')

I am getting following error:

[Stage 2:>                                                         (0 + 8) / 10]16/08/04 15:32:57 ERROR Executor: Exception in task 1.0 in stage 2.0 (TID 3)
com.univocity.parsers.common.TextParsingException: Length of parsed input (1000001) exceeds the maximum number of characters defined in your parser settings (1000000). 
Identified line separator characters in the parsed content. This may be the cause of the error. The line separator in your parser settings is set to '\n'. Parsed content:

And the following hint is coming. I am guessing it is not able understand line break.

Hint: Number of characters processed may have exceeded limit of 1000000 characters per column. Use settings.setMaxCharsPerColumn(int) to define the maximum number of characters a column can have
Ensure your configuration is correct, with delimiters, quotes and escape sequences that match the input format you are trying to parse
Parser Configuration: CsvParserSettings:
    Auto configuration enabled=true
    Autodetect column delimiter=false
    Autodetect quotes=false
    Column reordering enabled=true
    Empty value=null
    Escape unquoted values=false
    Header extraction enabled=null
    Headers=[1235187239212711042, 0006]
    Ignore leading whitespaces=false
    Ignore trailing whitespaces=false
    Input buffer size=128
    Input reading on separate thread=false
    Keep escape sequences=false
    Line separator detection enabled=false
    Maximum number of characters per column=1000000
    Maximum number of columns=20480
    Normalize escaped line separators=true
    Null value=
    Number of records to read=all
    Row processor=none
    RowProcessor error handler=null
    Selected fields=none
    Skip empty lines=true
    Unescaped quote handling=STOP_AT_DELIMITERFormat configuration:
    CsvFormat:
        Comment character=\0
        Field delimiter=,
        Line separator (normalized)=\n
        Line separator sequence=\n
        Quote character="
        Quote escape character=\
        Quote escape escape character=null

How to solve this error?

Answer 1

尝试将参数maxCharsPerColumn设置为更高的值。