Pandas read_fwf: specify dtype

Question

I am reading in a huge fixed width text file in chunks and export the data as csv. Because pandas.read_fwf does not allow to specify the dtypes, I am wondering what other way there exists to force the columns to be strings. The reason is that pandas infers some columns as float even though they are not and I do not want a .0 within a column.

Using data[column] = data[column].astype(str) does not help as it will not get rid of decimals. Converting columns of float64 dtype to int doesn't work either since NAs cannot be converted. Any ideas?

Here's a snippet of my code:

dat = pd.read_fwf(file_to_read,colspecs=cols,header=None,chunksize=100000,names=header)
#First chunk
data.info()
Int64Index: 100000 entries, 0 to 99999
Columns: 562 entries,
dtypes: float64(405), int64(4), object(153)
memory usage: 429.5+ MB

for column in data.columns:
    if data[column].dtype == 'float64':
        data[column] = data[column].astype(int)
    else:
        pass

I could do str().replace('.0','') , but I want to find an easier way than iterating through the column which takes a lot of time.

Answer 1

The converter parameter can be used to preserve the data as strings since pd.read_fwf does not try to guess the dtype if a converter is specified:

import pandas as pd
try:
    # for Python2
    from cStringIO import StringIO 
except ImportError:
    # for Python3
    from io import StringIO

content = '''\
1.0    2    A
3.0    4    B
5      X    C
M      Y    D
'''
header = ['foo', 'bar', 'baz']

for df in pd.read_fwf(StringIO(content), header=None, chunksize=2, names=header,
                      converters={h:str for h in header}):
    print(df)
df.info()

yields

   foo bar baz
0  1.0   2   A
1  3.0   4   B

  foo bar baz
0   5   X   C
1   M   Y   D

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 3 columns):
foo    2 non-null object
bar    2 non-null object
baz    2 non-null object
dtypes: object(3)
memory usage: 120.0+ bytes