How to correct error when saving dask dataframe to csv?

Question

I keep getting an error when I try and save a dask dataframe to csv. In short, I have a pandas df that is made up of 10 columns and 20 rows, and then I loaded a dask df that is 350 columns and 6+ million rows (~6GB). I needed to do a rather simple left join onto the pandas df. After doing that join, I look at the datatypes of the final dask df using final.dtypes and it shows 12 columns, as I hoped. However, when I try to convert the dask df called final to .csv I keep getting an error that refers to columns in the dask_df even though they are not in the final table. What is going on and how can I correct this? I can provide sample data if necessary.

Error message:

Usually this is due to dask's dtype inference failing, and
*may* be fixed by specifying dtypes manually by adding:
dtype={'Authorized Official Telephone Number': 'object',
       'Other Provider Identifier Issuer_33': 'object',
       'Other Provider Identifier Issuer_34': 'object',
       'Other Provider Identifier Issuer_35': 'object',
       'Other Provider Identifier Issuer_36': 'object',
       'Other Provider Identifier Issuer_37': 'object',
       'Other Provider Identifier Issuer_39': 'object',
       'Other Provider Identifier Issuer_40': 'object',
       'Other Provider Identifier Issuer_41': 'object',
       'Other Provider Identifier Issuer_42': 'object',
       'Other Provider Identifier Issuer_43': 'object',
       'Other Provider Identifier Issuer_44': 'object',
       'Other Provider Identifier Issuer_45': 'object',
       'Other Provider Identifier Issuer_46': 'object',
       'Other Provider Identifier Issuer_47': 'object',
       'Other Provider Identifier Issuer_48': 'object',
       'Other Provider Identifier Issuer_49': 'object',
       'Other Provider Identifier_37': 'object',
       'Other Provider Identifier_48': 'object',
       'Other Provider Identifier_49': 'object',
       'Provider Business Mailing Address Fax Number': 'object',
       'Provider Business Practice Location Address Fax Number': 'object'}

to the call to `read_csv`/`read_table`.

My code:

import dask.dataframe as dd
import pandas as pd

pandas_df = dd.read_csv('small_table.csv')

dask_df = dd.read_csv('npidata_pfile_20050523-20190407.csv',low_memory=False,dtype=str)

final= dd.merge(pandas_df, dask_df[['NPI','Provider First Name']], how='left', left_on='Physician NPI',right_on='NPI')

final.to_csv('e.csv')

Answer 1

You're passing dtype=str, but I think that perhaps you should pass dtype=object, which is what Pandas uses to represent really any non-numeric data.

The dask.dataframe.read_csv function is giving you an error message encouraging you to use dtype=object. It's actually giving you the full dtype={...} dict that you could pass in to make things work in the error message.

Answer 2

如果你真的不需要这些列中的任何一个，你可以通过将columns=[...]传递给dd.read_csv来简单地排除它们，只包括你确实需要的列。