使用 to_csv 和 dask 忽略不匹配的列类型

Question

I am trying to export a dataframe using dask with the dask.dataframe.to_csv(datframe_name, file etc..) command that was listed in the Dask manual: https://docs.dask.org/en/latest/dataframe-api.html#dask.dataframe.read_csv

I'm using dask because the original csv file was very large (20Gb) and it was very slow to use pandas to read the file.

However, everything I try to export the dataframe I keep getting the following error:

ValueError: Mismatched dtypes found in `pd.read_csv`/`pd.read_table`.

+----------+--------+----------+
| Column   | Found  | Expected |
+----------+--------+----------+
| DeviceID | object | int64    |
| Lat      | object | float64  |
| Long     | object | float64  |
+----------+--------+----------+

It is strange that the dataframe is finding the columns as objects, when their dtypes are integer and float.

Is there a way to ignore reading the columns' types and just export the dataframe as is?

Answer 1

The error message is telling you that when reading your data from the original CSV files Dask dataframe found that your data actually wasn't numeric as it originally guessed. A common cause for this is that you have a few rows of your data that actually aren't numeric in some way. Perhaps you have a custom NA value or some of the rows of your data are mismatched in some way.