How to create a pandas dataframe from a file with no common separator?

Question

I have routing table data that looks something like this:

Valid  Network Next-Hop Path Protocol
0  1.0.128.0/17 80.249.208.85 3257 38040 9737 i
0       80.249.209.150 6939 4766 38040 9737 i
1       80.249.209.37 3491 38040 9737 i
0       80.249.211.42 6762 38040 9737 i
0       80.249.208.85 3257 38040 9737 i
1       80.249.209.37 3491 38040 9737 i
0       80.249.211.42 6762 38040 9737 i

I want to create DataFrame with those same column names as the header and the prefix in the network column. The problem here is that not all lines have a prefix so I need to add the latest prefix (most recently seen). This is what I did:

f = open('initial_data')
current_prefix = None
for index,line in enumerate(f):
    if index != 0 and index != 1058274: #ignoring first and last line
        if line.split(' ')[2].startswith(str(1)) or line.split(' ')[2].startswith(str(2)):
            current_prefix = np.asarray(line.split(' ')[2]) #storing the most recent prefix
            #print(current_prefix)#.shape,type(current_prefix))
            df2 = pd.DataFrame([[current_prefix]], columns=list('Network'))
            df.append(df2,ignore_index = True)#current_prefix)
        else:
            pass#df['Network'].append(current_prefix)
            df2 = pd.DataFrame([[current_prefix]], columns=list('Network'))
            df.append(df2,ignore_index = True)#current_prefix

But the prefix (eg 1.0.128.0/17) is interpreted as having 7 columns and I get this error:

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-100-f2ee3d75b5c4> in <module>()
      6             current_prefix = np.asarray(line.split(' ')[2])
      7             #print(current_prefix)#.shape,type(current_prefix))
----> 8             df2 = pd.DataFrame([[current_prefix]], columns=list('Network'))
      9             df.append(df2,ignore_index = True)#current_prefix)
AssertionError: 7 columns passed, passed data had 1 columns

So is there any better/cleaner way to deal with this? To be more precise, I would like the DataFrame to look something like this:

Valid | Network        | Next-Hop      | Path                 | Protocol
0     | 1.0.128.0/17   | 80.249.208.85 | 3257 38040 9737      | i
0     |NaN/aboveprefix | 80.249.209.150| 6939 4766 38040 9737 | i

and so on. Any leads?

Answer 1

I Solved this first by writing a new file with \\t separations and then using pandas.read_csv('file_name',sep='\\t').

The main trick here was to count the '/' and '.'s in each split of line:

def classify(strx):
#call this on each line
next_hop = 0
prefix = ""
#path = ""
path = []
for i in strx.split(' '):
    slash_counter,dot_counter = i.count('/'),i.count('.')
    #print(i.count('.'),i.count('/'))
    if dot_counter == 3 and slash_counter == 1:
        prefix = i
    elif dot_counter == 3 and slash_counter == 0:
        next_hop = i
    elif len(i) > 1 and dot_counter == 0 and slash_counter == 0:

        #path = path.join(i)
        path.append(i)
        #print("Sanity check on paths",path,"\t",i)
        #path = path.join(' ')
path_ = path
path = " ".join([str(i) for i in path_])
protocol = strx[-1]
#print(protocol)
#print("prefix = {0}, next hop = {1}, path = {2}".format(prefix,next_hop,path))
return(prefix,next_hop,path,protocol)

an example: original line:

'0 1.0.128.0/17 80.249.208.85 3257 38040 9737 i'

after converting with above function:

1.0.128.0/17 80.249.208.85 3257 38040 9737 i

Answer 2

The accepted answer is certainly a way to do this, but pandas provides a way to extract fields and fill in missing values exactly as you've described by passing a regex to str.extract() on a Series object to create a DataFrame , and then .fillna() with method='ffill' to fill in the missing values in the "Network" field. This approach would be accomplished by something like the following.

import io
import pandas as pd
import re

f = io.StringIO('''\
Valid  Network Next-Hop Path Protocol
0  1.0.128.0/17 80.249.208.85 3257 38040 9737 i
0       80.249.209.150 6939 4766 38040 9737 i
1       80.249.209.37 3491 38040 9737 i
0       80.249.211.42 6762 38040 9737 i
0       80.249.208.85 3257 38040 9737 i
1       80.249.209.37 3491 38040 9737 i
0       80.249.211.42 6762 38040 9737 i
''')

pattern = re.compile(r'^(?P<valid>[01])'
                     r'\s+(?P<network>\d+\.\d+\.\d+\.\d+/\d+)?'
                     r'\s+(?P<next_hop>\d+\.\d+\.\d+\.\d+)'
                     r'\s+(?P<path>(?:\d+ )+)'
                     r'(?P<protocol>[a-z])$')

next(f) #skip the first line
df = (pd.Series(f.readlines())
      .str.extract(pattern, expand=False)
      .fillna(method='ffill'))

This results in a DataFrame that looks like.

Out [26]:
  valid       network        next_hop                   path protocol
0     0  1.0.128.0/17   80.249.208.85       3257 38040 9737         i
1     0  1.0.128.0/17  80.249.209.150  6939 4766 38040 9737         i
2     1  1.0.128.0/17   80.249.209.37       3491 38040 9737         i
3     0  1.0.128.0/17   80.249.211.42       6762 38040 9737         i
4     0  1.0.128.0/17   80.249.208.85       3257 38040 9737         i
5     1  1.0.128.0/17   80.249.209.37       3491 38040 9737         i
6     0  1.0.128.0/17   80.249.211.42       6762 38040 9737         i

If you don't want the missing "Network" values filled in, you can remove the call to .fillna() .