Python =达人vs熊猫，read_csv中的错误

Question

I've got an error on reading a file with dask, which work with pandas :

import dask.dataframe as dd
import pandas as pd
pdf = pd.read_csv("./tous_les_docs.csv")
pdf.shape
(20140796, 7)

while dask gives me an error :

df = dd.read_csv("./tous_les_docs.csv")
df.describe().compute()
ParserError: Error tokenizing data. C error: EOF inside string starting at line 192999

Answer : Adding "blocksize=None" make it work :

df = dd.read_csv("./tous_les_docs.csv", blocksize=None)

Answer 1

The documentation says that this could happen

It should also be noted that this function may fail if a CSV file includes quoted strings that contain the line terminator. To get around this you can specify blocksize=None to not split files into multiple partitions, at the cost of reduced parallelism.

http://docs.dask.org/en/latest/dataframe-api.html#dask.dataframe.read_csv

It seems Dask chops the file in chunks by line terminator but without scanning the whole file from the start, to see if a line terminator is in a string.