Python script to Grep strings from input file and output in csv format

Question

I am writing a python script to grep string from file and display output in csv file in below format

enter image description here

enter image description here

Input file(result_EPFT_config_device) :

Hostname SIM-MPL-LTE-PE-RTR-134
loopback 22.13.7.34
lpts punt excessive-flow-trap
penalty-rate arp 10
penalty-rate icmp 50
penalty-rate igmp 50
penalty-rate ip 100
exclude interface Bundle-Ether6
exclude interface Bundle-Ether8
exclude interface Bundle-Ether15
exclude interface Bundle-Ether16
exclude interface Bundle-Ether53
exclude interface TenGigE0/0/1/1
exclude interface TenGigE0/1/1/0
exclude interface Bundle-Ether6.2
exclude interface Bundle-Ether6.4
exclude interface Bundle-Ether8.2
exclude interface Bundle-Ether8.4
exclude interface Bundle-Ether16.2
exclude interface Bundle-Ether16.4
exclude interface Bundle-Ether53.2
exclude interface TenGigE0/0/1/3.100
exclude interface TenGigE0/0/1/3.102
exclude interface TenGigE0/0/1/3.103
exclude interface TenGigE0/0/1/3.104
exclude interface TenGigE0/1/1/0.100
exclude interface GigabitEthernet0/0/0/1
exclude interface GigabitEthernet0/0/0/6
exclude interface GigabitEthernet0/0/0/9
dampening.
non-subscriber-interfaces
report-threshold 10

Below is the python script i have prepared as of now. Only able to grep string and print it

import sys
import telnetlib
import os
import subprocess
import re
import csv

fh = open("result_EPFT_config_device", "r")
fh1 = open("testingAjay", "w+")
line = fh.readlines()
for lines in line:
        if re.search("(lpts punt excessive-flow-trap)", lines):
                m =  (lines.split(' '))
                print m[0], m[1], m[2]
        if re.search("(penalty-rate arp)", lines):
                n =  (lines.split(' '))
                print n[0], n[1], n[2]
        if re.search("(penalty-rate icmp)", lines):
                a =  (lines.split(' '))
                print a[0], a[1], a[2]
        if re.search("(penalty-rate igmp)", lines):
                b =  (lines.split(' '))
                print b[0], b[1], b[2]
        if re.search("(penalty-rate ip)", lines):
                c =  (lines.split(' '))
                print c[0], c[1], c[2]
        if re.search("(dampening)", lines):
                c =  (lines.split(' '))
                print c[0]
        if re.search("(non-subscriber-interfaces)", lines):
                c =  (lines.split('-'))
                print c[0], c[1], c[2]
        if re.search("(report-threshold 10)", lines):
                c =  (lines.split(' '))
                print c[0], c[1]

My script output :

lpts punt excessive-flow-trap
penalty-rate arp 10
penalty-rate icmp 50
penalty-rate igmp 50
penalty-rate ip 100
dampening.
non subscriber interfaces

report-threshold 10

Now here i want to put the output in csv file as shown below

enter image description here

Hostname|loopback|lpts punt excessive-flow-trap|penalty-rate arp|penalty-rate icmp|penalty-rate igmp|penalty-rate ip|dampening|non-subscriber-interfaces|report-threshold
SIM-MPL-LTE-PE-RTR-134|1.1.1.1|yes|10|50|50|100|Yes|Yes|10
NDL-MPL-PE-RTR-195|2.2.2.2|No|No|No|20|50|NO|20Yes

As shown in above screenshot, column lpt spunt excessive flow trap has to mark as YES if its present in Input File else mark NO. Similar logic needs to apply for column dampening and non subscriber interface column

Could you please help me to achieve require output in csv format as shown above

Answer 1

Here we go! So as the comment above said you can just use "startswith" rather then regex to match the lines.

I have used python3 rather then python2 here.

If you run this in a directory using "python3 main.py" it will search the "inputs" subdirectory for all files to parse.

We then build a dictionary for each file with the relevant fields and load their values. We add these dictionaries to a list. At the end we just write the headers to the csv then loop through the rows and write values. You could probably write the rows while reading the files but I find separating parsing and output cleaner mentally.

I changed the ordering of your "for lines in line" to "for line in lines" as you want to loop through each line in the lines.

import os
import csv

def parseFile(fileName):

    # We are using a dictionary to store info for each file
    data = dict()

    # Set all Yes/Nos to NO by default
    data["lpts punt excessive-flow-trap"] = "NO"
    data["dampening"] = "NO"
    data["non-subscriber-interfaces"] = "NO"

    fh = open(fileName, "r")
    lines = fh.readlines()
    for line in lines:

        # We need this so we don't end up with newline characters in our CSV
        line = line.rstrip("\n")

        # We dont need regular expressions here as matching whole line
        # Do YES/NO first
        if line == "lpts punt excessive-flow-trap":
            data["lpts punt excessive-flow-trap"] = "YES"
            continue;

        if line == "dampening":
            data["dampening."] = "YES"
            continue;

        if line == "non-subscriber-interfaces":
            data["non-subscriber-interfaces"] = "YES"
            continue;

        # Now do the rest
        if line.startswith("Hostname"):
            splitted = line.split(' ')
            data["Hostname"] = splitted[1]
            continue;

        if line.startswith("loopback"):
            splitted = line.split(' ')
            data["loopback"] = splitted[1]
            continue;

        if line.startswith("penalty-rate arp"):
            print("ARP")
            splitted = line.split(' ')
            data["penalty-rate arp"] = splitted[2]
            continue;

        if line.startswith("penalty-rate icmp"):
            splitted = line.split(' ')
            data["penalty-rate icmp"] = splitted[2]
            continue;

        if line.startswith("penalty-rate igmp"):
            splitted = line.split(' ')
            data["penalty-rate igmp"] = splitted[2]
            continue;

        if line.startswith("penalty-rate ip"):
            splitted = line.split(' ')
            data["penalty-rate ip"] = splitted[2]
            continue;

        if line.startswith("report-threshold"):
            splitted = line.split(' ')
            data["report-threshold"] = splitted[1]
            continue;

    return data


if __name__ == "__main__":
    inputsDirectory = "inputs"
    path = os.path.abspath(inputsDirectory)
    fileList = ["{}/{}".format(path,x) for x in os.listdir(inputsDirectory)]
    print(fileList)

    # Load Each File and Build Dictionary
    csvRows = []
    for file in fileList:
        newRow = parseFile(file)
        csvRows.append(newRow)

    print(csvRows)

    # Output CSV using dictionaries for each file
    outputFile = "output.csv"
    with open(outputFile, 'w') as csvfile:
        fieldnames = ["Hostname",
                      "loopback",
                      "lpts punt excessive-flow-trap",
                      "penalty-rate arp",
                      "penalty-rate icmp",
                      "penalty-rate igmp",
                      "penalty-rate ip",
                      "dampening",
                      "non-subscriber-interfaces",
                      "report-threshold"]
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)

        writer.writeheader()
        for row in csvRows:
            writer.writerow(row)