pandas数据帧问题与特殊字符

Question

I am struggling with the following issue with pandas data frame Python 2.7.12 pandas 0.18.1

df = pd.read_csv(file_name, encoding='utf-16', header=0, index_col=False, error_bad_lines=False,
                     names=['Package_Name','Crash_Report_Date_And_Time','Crash_Report_Millis_Since_Epoch','Device', 'Android_OS_Version','App_Version_Name','App_Version_Code','Exception_Class_Name','Exception_Message','Throwing_File_Name','Throwing_Class_Name','Throwing_Method_Name','Throwing_Line_Number','Native_Crash_Library','Native_Crash_Location','Crash_Link'])

I debug the code and found following data is not inserting properly in the dataframe.

There are some special characters in the Exception_Message field which is telling pandas to move rest of the data on that row to the next row.

Some how Pandas is not reading the file properly.

Following is the output for both the rows. 137 and 138 are the row numbers.

Package_Name Crash_Report_Date_And_Time  \
137  com.vcast.mediamanager        2016-09-05 14:54:13   
138                     NaN                 Class.java   

    Crash_Report_Millis_Since_Epoch        Device  Android_OS_Version  \
137                   1473087253130       victara                  22   
138                 java.lang.Class  classForName                  -2   

    App_Version_Name  App_Version_Code  \
137          14.3.34      1.503050e+09   
138              NaN               NaN   

                                  Exception_Class_Name  \
137                   java.lang.ClassNotFoundException   
138  https://play.google.com/apps/publish?dev_acc=0...   

                     Exception_Message Throwing_File_Name Throwing_Class_Name  \
137  Invalid name: com.strumsoft.appen                NaN                 NaN   
138                                NaN                NaN                 NaN   

    Throwing_Method_Name  Throwing_Line_Number Native_Crash_Library  \
137                  NaN                   NaN                  NaN   
138                  NaN                   NaN                  NaN   

    Native_Crash_Location Crash_Link account_id  
137                   NaN        NaN       NONE  
138      

         NaN        NaN       NONE  

Row 138 is created erroneously with some data from row 137. Exception Message field in 137 has some value which is breaking that row to the next row. Which is wrong.

I tried different encoding nothing helped. Can anyone please help?

Answer 1

So the solution was really simple. In Pandas 0.18 I had to specify the lineterminator='n'

df = pd.read_csv(file_name,lineterminator='\n', encoding='utf-16', delimiter=',', header=0, index_col=False, error_bad_lines=False,...

This simple flag fixed my issue.