在导入带有额外逗号的熊猫的csv文件时，如何使用正则表达式作为分隔符？

Question

The csv file was sent to me/ I can not re delimit the columns

239845723,28374,2384234,AEVNE EFU 5 GN OR WNV,Owinv Vnwo Badvw 5 VIN,Ginq 2 jnwve wef evera wve 6 vwe as fgsb bfd bdfwd dsf (sdv seves 4-6), sebsbe sve(sevsev esvse 7-10) fsesef fesevsesv PaVvin (1 evesve vEV VEWee, 2 for WVEee VEWE. paper tuff as sWEFEWoon as VEWeew.).,2011-07-13 00:00:00,2011-07-13 00:00:00

I replaced the string letters to cover sensitive info, however the problem is apparent.

This is an example "problem row" in my csv. It should be sorted into 8 columns as follows:

col1: 239845723
col2: 28374
col3: 2384234
col4: AEVNE EFU 5 GN OR WNV
col5: Owinv Vnwo Badvw 5 VIN
col6: Ginq 2 jnwve wef evera wve 6 vwe as fgsb bfd bdfwd dsf (sdv seves 4-6), sebsbe sve(sevsev esvse 7-10) fsesef fesevsesv PaVvin (1 evesve vEV VEWee, 2 for WVEee VEWE. paper tuff as sWEFEWoon as VEWeew.).
col7: 2011-07-13 00:00:00
col8: 2011-07-13 00:00:00

As you can see, column 6 is where the problem occurs as there are commas in the string that cause pandas to delimit and create new columns incorrectly. How can I solve this problem? I was thinking regex would help, perhaps with the below setup. Any help is appreciated!

    csvfile = open(filetrace) 
    reader = csv.reader(csvfile)
    new_list=[]
    for line in reader:
        for i in line:
            #not sure

Answer 1

Istead of going to regex, read the csv with delimiter ',', You can extract the last two dates and store it in a list. Then fill the dates with '' then join the columns you want and the delete the rest. Example

If you have a csv file :

239845723,28374,2384234,AEVNE EFU 5 GN OR WNV,Owinv Vnwo Badvw 5 VIN,Ginq 2 jnwve wef evera wve 6 vwe as fgsb bfd bdfwd dsf (sdv seves 4-6), sebsbe sve(sevsev esvse 7-10) fsesef fesevsesv PaVvin (1 evesve vEV VEWee, 2 for WVEee VEWE. paper tuff as sWEFEWoon as VEWeew.).,2011-07-13 00:00:00,2011-07-13 00:00:00
239845723,28374,2384234,AEVNE EFU 5 GN OR WNV,Owinv Vnwo Badvw 5 VIN,Ginq 2 jnwve wef evera wve 6 vwe as fgsb bfd bdfwd dsf (sdv seves 4-6), sebsbe sve(sevsev esvse 7-10) fsesef fesevsesv PaVvin (1 evesve vEV VEWee 2 for WVEee VEWE.).,2011-07-13 00:00:00,2011-07-13 00:00:00
239845723,28374,2384234,AEVNE EFU 5 GN OR WNV,Owinv Vnwo Badvw 5 VIN sebsbe sve(sevsev esvse 7-10) fsesef fesevsesv PaVvin (1 evesve vEV VEWee 2 for WVEee VEWE. paper tuff as sWEFEWoon as VEWeew.).,2011-07-13 00:00:00,2011-07-13 00:00:00

Then

df = pd.read_csv('good.txt',delimiter=',',header=None)
# Get the Dates from all the DataFrame 
dates = [[item] for i in df.values for item in i if '2011-' in str(item)]
# Merge two Dates for each column
dates = pd.DataFrame([x+y for x,y in zip(dates[0::2], dates[1::2])])
# Remove the dates present 
df = df.replace({'2011-': np.nan}, regex=True).replace(np.nan,'')

#Get the columns you want to merge 
cols = df.columns[4:]
# Merge the columns 
df[4] = df[cols].astype(str).apply(lambda x: ','.join(x), axis=1)
df[4] = df[4].replace('\,+$', '',regex=True)
#Drop the Columns 
df = df.drop(df.columns[5:],axis=1)
#Concat the dates 
df = pd.concat([df,dates],axis=1)

Output : print(df)

0      1        2                      3  \
0  239845723  28374  2384234  AEVNE EFU 5 GN OR WNV   
1  239845723  28374  2384234  AEVNE EFU 5 GN OR WNV   
2  239845723  28374  2384234  AEVNE EFU 5 GN OR WNV   

                                                   4                    0  \
0  Owinv Vnwo Badvw 5 VIN,Ginq 2 jnwve wef evera ...  2011-07-13 00:00:00   
1  Owinv Vnwo Badvw 5 VIN,Ginq 2 jnwve wef evera ...  2011-07-13 00:00:00   
2  Owinv Vnwo Badvw 5 VIN sebsbe sve(sevsev esvse...  2011-07-13 00:00:00   

                     1  
0  2011-07-13 00:00:00  
1  2011-07-13 00:00:00  
2  2011-07-13 00:00:00

Ouput of the 4th column :

['Owinv Vnwo Badvw 5 VIN,Ginq 2 jnwve wef evera wve 6 vwe as fgsb bfd bdfwd dsf (sdv seves 4-6), sebsbe sve(sevsev esvse 7-10) fsesef fesevsesv PaVvin (1 evesve vEV VEWee, 2 for WVEee VEWE. paper tuff as sWEFEWoon as VEWeew.).',

 'Owinv Vnwo Badvw 5 VIN,Ginq 2 jnwve wef evera wve 6 vwe as fgsb bfd bdfwd dsf (sdv seves 4-6), sebsbe sve(sevsev esvse 7-10) fsesef fesevsesv PaVvin (1 evesve vEV VEWee 2 for WVEee VEWE.).',

'Owinv Vnwo Badvw 5 VIN sebsbe sve(sevsev esvse 7-10) fsesef fesevsesv PaVvin (1 evesve vEV VEWee 2 for WVEee VEWE. paper tuff as sWEFEWoon as VEWeew.).']

If you want to change column index

df.columns = [i for i in range(df.shape[1])]

Hope it helps

Answer 2

So, without knowing the specifics of the file or the data I can offer a regex solution that could work if the data is consistent (and has the period at the end of column 6). We can do it without using the csv module and just the regex module.

import re

# make the regex pattern here
pattern = r"([\d\.]*),([\d\.]*),([\d\.]*),([^,]*),([^,]*),(.*\.?),([\d\-\s:]*),([\d\-\s:]*)"

# open the file with 'with' so you don't have to worry about closing it
with open(filetrace) as f:
    for line in f:  # iterate through the lines
        values = re.findall(pattern, line)[0]  # re.findall returns a list 
                                               # literal of a tuple
        # record the values somewhere

values here is an 8-tuple containing the values from each of the columns that you had in your original csv, just use/store them however you want.

Best of luck with it!

Answer 3

Since you know exactly how many columns you need and there is only one problematic column, we can split the first few off from the left and rest from the right. In other words, you don't need regex

Read file into single string

csvfile = open(filetrace).read()

Make pd.Series

s = pd.Series(csvfile.split('\n'))

Split s and limit it to 5 splits, which should be 6 columns

df = s.str.split(',', 5, expand=True)

Now split the right side limited to 2 splits

df = df.iloc[:, :-1].join(df.iloc[-1].str.rsplit(',', 2, expand=True))

---

Another way starting from s

left = s.str.split(',', 5)
right = left.str[-1].str.rsplit(',', 2)

df = pd.DataFrame(left.str[:-1].add(right).tolist())

---

I ran this and took the transpose to make it easier to read on screen

df.T



                                                   0
0                                          239845723
1                                              28374
2                                            2384234
3                              AEVNE EFU 5 GN OR WNV
4                             Owinv Vnwo Badvw 5 VIN
5  Ginq 2 jnwve wef evera wve 6 vwe as fgsb bfd b...
6                                2011-07-13 00:00:00
7                                2011-07-13 00:00:00