pandas dataframe read csv with rows that have/not have comma at the end

Question

my sample file looks like this:

1.50424e+09,164.84,164.94,163.4,164.07,09:30:00,1.50424e+12,eAAPL,1.38904e+07,0,22.45,2.64333e+07,847097,18.49,1.54
1.50459e+09,163.8,164.25,158.26,162.2,09:30:00,1.50459e+12,eAAPL,2.54615e+07,0,22.44,2.64646e+07,847097,18.49,1.54
1.50467e+09,162.71,162.99,160.52,162.01,09:30:00,1.50467e+12,eAAPL,1.67919e+07,0,22.67,2.61136e+07,837180,18.27,1.55
1.50485e+09,160.9,161.15,158.62,158.7,09:30:00,1.50485e+12,eAAPL,2.02651e+07,0,22.73,2.48843e+07,832945,18.18,1.56,
1.50511e+09,160.51,162.05,159.89,161.48,09:30:00,1.50511e+12,eAAPL,2.44948e+07,0,22.54,2.50082e+07,819360,17.88,1.59,
1.50476e+09,162.17,163.69,160.36,161.175,09:30:00,1.50476e+12,eAAPL,1.88933e+07,0,22.68,2.58778e+07,836302,18.25,1.56,

please scroll to the very right, you will see first 3 rows do not have comma at the end but the last 3 have comma. When i do

f = pd.read_csv("AAPL.csv", header=None)

it gives error saying:

CParserError: Error tokenizing data. C error: Expected 15 fields in line 4, saw 16

how do i fix it?

Answer 1

You could use usecols with np.arange(0,15) , ignoring that trailing column on the bottom three lines of your csv file:

from io import StringIO
file = StringIO("""1.50424e+09,164.84,164.94,163.4,164.07,09:30:00,1.50424e+12,eAAPL,1.38904e+07,0,22.45,2.64333e+07,847097,18.49,1.54
1.50459e+09,163.8,164.25,158.26,162.2,09:30:00,1.50459e+12,eAAPL,2.54615e+07,0,22.44,2.64646e+07,847097,18.49,1.54
1.50467e+09,162.71,162.99,160.52,162.01,09:30:00,1.50467e+12,eAAPL,1.67919e+07,0,22.67,2.61136e+07,837180,18.27,1.55
1.50485e+09,160.9,161.15,158.62,158.7,09:30:00,1.50485e+12,eAAPL,2.02651e+07,0,22.73,2.48843e+07,832945,18.18,1.56,
1.50511e+09,160.51,162.05,159.89,161.48,09:30:00,1.50511e+12,eAAPL,2.44948e+07,0,22.54,2.50082e+07,819360,17.88,1.59,
1.50476e+09,162.17,163.69,160.36,161.175,09:30:00,1.50476e+12,eAAPL,1.88933e+07,0,22.68,2.58778e+07,836302,18.25,1.56,""")

f = pd.read_csv(file, usecols=np.arange(0,15), header=None)

print(f.head())

Output:

             0       1       2       3       4         5             6   \
0  1.504240e+09  164.84  164.94  163.40  164.07  09:30:00  1.504240e+12   
1  1.504590e+09  163.80  164.25  158.26  162.20  09:30:00  1.504590e+12   
2  1.504670e+09  162.71  162.99  160.52  162.01  09:30:00  1.504670e+12   
3  1.504850e+09  160.90  161.15  158.62  158.70  09:30:00  1.504850e+12   
4  1.505110e+09  160.51  162.05  159.89  161.48  09:30:00  1.505110e+12   

      7           8   9      10          11      12     13    14  
0  eAAPL  13890400.0   0  22.45  26433300.0  847097  18.49  1.54  
1  eAAPL  25461500.0   0  22.44  26464600.0  847097  18.49  1.54  
2  eAAPL  16791900.0   0  22.67  26113600.0  837180  18.27  1.55  
3  eAAPL  20265100.0   0  22.73  24884300.0  832945  18.18  1.56  
4  eAAPL  24494800.0   0  22.54  25008200.0  819360  17.88  1.59  

Answer 2

Try the following approach if you don't know the # of columns beforehand:

import io

with open(filename) as f:
    data = f.read() + '\n' # in case there is no `\n` in the last line
    df = pd.read_csv(io.StringIO(data.replace(',\n','\n'))