使用“ |”将带有推文的文本文件解析为csv 分隔符

Question

I have a .txt file containing geotagged tweets. The information is delimited by '|' character. The information (which can be perceived as columns) are datetime, latitude, longitude and tweet_text.

Date_time|latitude|longitude|tweet_text
Mon Jan 01 09:09:57 +0000 2018|-37.8140362|144.9644232|terima kasih 2017 yang ohsem. semoga 2018 akan lebih baik lagi.-selamat tahun baru rakan-rakan… 
Mon Jan 01 09:15:54 +0000 2018|-37.81639|144.9655|we love christmas and new year proposals! happy new year to everyone celebrating love this year… 
Mon Jan 01 09:42:08 +0000 2018|-37.818|144.985|@michaelpaynter entertaining everyone at yarra park nye event #melbourne| #nye #music… 
Mon Jan 01 09:45:16 +0000 2018|-37.818|144.985|@emilyurbandiva and brother @jwilliamsimusik entertaining everyone at yarra park nye eventâ€|¦

Initially, I used

data = pd.read_csv('MelbCBD_scs2018_new.txt',sep="|", header=None)

, but it threw parsing error whenever the tweet_text had '|'.

I tried manually cleaning tweet_text but it is too much work for large files. Hence I changed the argument parameter of read_csv.

data = pd.read_csv('MelbCBD_scs2018_new.txt',sep="|", header=None, quoting=csv.QUOTE_NONE,error_bad_lines=False)

But it displays the following warning and skips those lines (essentially the tweets) which I do not want.

b'Skipping line 340: expected 4 fields, saw 5

I would ideally like a code that removes any special character after encountering 3 '|' characters in each line of the .txt file ie the tweet_text column and parses it into the .csv file, without skipping any line.

Answer 1

So it was just a matter of specifying the number of columns by specifying the column names.

data = pd.read_csv('MelbCBD_scs2018_new.txt', sep="|", names = ["Date_time", "latitude", "longitude","tweet_text"], header=None, quoting=csv.QUOTE_NONE,error_bad_lines=False)

Now, this returns every single line without skipping and stores it in the dataframe named 'data'.