从日志文件python创建csv头文件

Question

My log file contains some info in every row like below

Info1:NewOrder|key:123 |Info3:10|Info5:abc
Info3:10|Info1:OldOrder| key:456| Info6:xyz
Info1:NewOrder|key:007

I want to change it to a csv like below (if i give key,Info1,Info3 as required headers)

key,Info1.Info3
123,NewOrder,10
456,OldOrder,10
007,NewOrder,

Earlier I used awk to get field values, but logging can change the order of info and key printed in a row. So I cannot be sure that Info3 would always be in some particular column. Everytime,logging changes, the script needed to be changed.

I intend then to load csv in pandas dataframe. So a python solution would be better. This is more of a data cleaning task to generate a csv from logfile.

This is what I have used after reading the answers

import csv
import sys
with open(sys.argv[1], 'r') as myLogfile:
        log=myLogfile.read().replace('\n', '')

requested_columns = ["OrderID", "TimeStamp", "ErrorCode"]

def wrangle(string, requested_columns):
        data = [dict([element.strip().split(":") for element in row.split("|")]) for row in string.split("\n")]
        body = [[row.get(column) for column in requested_columns] for row in data]
        return [requested_columns] + body

outpath = sys.argv[2]
open(outpath, "w", newline = "") with open(outpath, 'wb')
        writer = csv.writer(file)
        writer.writerows(wrangle(log, requested_columns))

Sample logfile= https://ideone.com/cny805

Answer 1

The bulk of it is just using useful string methods like strip and split, plus list comprehensions.

import csv

string = """Info1=NewOrder|key=123 |Info3=10|Info5=abc
Info3=10|Info1=OldOrder| key=456| Info6=xyz
Info1=NewOrder|key=007"""

requested_columns = ["key", "Info1", "Info3"]

def wrangle(string, requested_columns):
    data = [dict([element.strip().split("=") for element in row.split("|")]) for row in string.split("\n")]
    body = [[row.get(column) for column in requested_columns] for row in data]
    return [requested_columns] + body

outpath = "whatever.csv"

with open(outpath, "w", newline = "") as file:
    writer = csv.writer(file)
    writer.writerows(wrangle(string, requested_columns))

Answer 2

You could use a csv reader with a | delimiter to get you started, then split using : to give you a per row dictionary as follows:

import csv

with open('input.csv', 'rb') as f_input, open('output.csv', 'wb') as f_output:
    csv_output = csv.writer(f_output)
    cols = ["OrderID", "TimeStamp", "ErrorCode"]
    csv_output.writerow(cols)

    for row in csv.reader(f_input, delimiter='|'):
        # Remove any entries that do not have a colon
        row = [c for c in row if c.find(':') != -1]
        # Convert remaining columns into a dictionary
        entries = {c.split(':')[0].strip() : c.split(':')[1].strip() for c in row}
        csv_output.writerow([entries.get(c, "") for c in cols])

Giving you an output file:

OrderID,TimeStamp,ErrorCode
3000000,1488948188555841641,
3000000,1488948188556444675,0

---

To read the data directly into a Pandas dataframe:

import pandas as pd
import csv

cols = ["OrderID", "TimeStamp", "ErrorCode"]
data = []

with open('input.csv', 'rb') as f_input:
    csv_output = csv.writer(f_output)

    for row in csv.reader(f_input, delimiter='|'):
        # Remove any entries that do not have a colon
        row = [c for c in row if c.find(':') != -1]
        # Convert remaining columns into a dictionary
        entries = {c.split(':')[0].strip() : c.split(':')[1].strip() for c in row}
        data.append([entries.get(c, "") for c in cols])

df = pd.DataFrame(data, columns=cols)
print df

Giving you:

   OrderID            TimeStamp ErrorCode
0  3000000  1488948188555841641          
1  3000000  1488948188556444675         0