将列表列表转换为结构化的 Pandas 数据框

Question

I am trying to convert the following data structure;

To the format below in python 3;

Answer 1

if your data looks like:

array = [['PIN: 123 COD: 222 \n', 'LOA: 124 LOC: Sea \n'],
        ['PIN:456 COD:555 \n', 'LOA:678 LOC:Chi \n']]

You can do this:

1 Step: use regular expressions to parse your data, because it is string.

see more about reg-exp

raws=list()
for index in range(0,len(array)):    
    raws.append(re.findall(r'(PIN|COD|LOA|LOC): ?(\w+)', str(array[index])))

Output:

[[('PIN', '123'), ('COD', '222'), ('LOA', '124'), ('LOC', 'Sea')], [('PIN', '456'), ('COD', '555'), ('LOA', '678'), ('LOC', 'Chi')]]

2 Step: extract raw values and column names.

columns = np.array(raws)[0,:,0]
raws = np.array(raws)[:,:,1]

Output:

raws -

[['123' '222' '124' 'Sea']
 ['456' '555' '678' 'Chi']]

columns -

['PIN' 'COD' 'LOA' 'LOC']

3 Step: Now we can just create df.

df = pd.DataFrame(raws, columns=columns)

Output:

   PIN  COD  LOA  LOC
0  123  222  124  Sea
1  456  555  678  Chi

Is it what you want?

I hope it helps, I'm not sure about your input format.

And don't forget import libraries! (I used pandas as pd, numpy as np, re).

UPD: another way I have created log file like you have:

array = open('example.log').readlines()

Output:

['PIN: 123 COD: 222 \n',
 'LOA: 124 LOC: Sea \n',
 'PIN: 12 COD: 322 \n',
 'LOA: 14 LOC: Se \n']

Then split by ' ' , drop '\\n' and reshape:

raws = np.array([i.split(' ')[:-1] for i in array]).reshape(2, 4, 2)

In reshape, first number is raws count in your future dataframe, second - count of columns and last - you don't need to change. It won't works if you don't have whitespace between info and '\\n' in each raw. If you don't, I will change an example. Output:

array([[['PIN:', '123'],
        ['COD:', '222'],
        ['LOA:', '124'],
        ['LOC:', 'Sea']],

       [['PIN:', '12'],
        ['COD:', '322'],
        ['LOA:', '14'],
        ['LOC:', 'Se']]], 
      dtype='|S4')

And then take raws and columns:

columns = np.array(raws)[:,:,0][0]
raws = np.array(raws)[:,:,1]

Finally, create dataframe (and cat last symbol for columns):

pd.DataFrame(raws, columns=[i[:-1] for i in columns])

Output:

   PIN  COD  LOA  LOC
0  123  222  124  Sea
1   12  322   14   Se

If you have many log files, you can do that for each in for-loop, save each dataframe at array (example, array calls DF_array) and then use pd.concat to do one dataframe from array of dataframes.

pd.concat(DF_array)

If you need I can add an example.

UPD: I have created a dir with log files and then make array with all files from PATH:

PATH = "logs_data/"
files = [PATH + i for i in os.listdir(PATH)]

Then do for-loop like in last update:

dfs = list()
for f in files:
    array = open(f).readlines()
    raws = np.array([i.split(' ')[:-1] for i in array]).reshape(len(array)/2, 4, 2)
    columns = np.array(raws)[:,:,0][0]
    raws = np.array(raws)[:,:,1]
    df = pd.DataFrame(raws, columns=[i[:-1] for i in columns])
    dfs.append(df)
result = pd.concat(dfs)

Output:

     PIN   COD    LOA  LOC
0    123   222    124  Sea
1     12   322     14   Se
2      1    32      4  Ses
0  15673  2324  13464  Sss
1  12452  3122  11234   Se
2     11   132      4  Ses
0    123   222    124  Sea
1     12   322     14   Se
2      1    32      4  Ses