How should I print formated text from a txt file with python?

Question

I got this file with this info pattern:

# Query 1: 204.60k QPS, 230.79x concurrency, ID XXXXXXXXXX at byte 19XXX9318
# This item is included in the report because it matches --limit.
# Scores: V/M = 0.00
# Time range: 2020-01-29 18:18:59.073995 to 18:18:59.074005
# Attribute    pct   total     min     max     avg     95%  stddev  median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count          7       2
# Exec time     10     2ms     1ms     1ms     1ms     1ms    12us     1ms
# Rows affecte   0       0       0       0       0       0       0       0
# Query size     7      74      37      37      37      37       0      37
# Warning coun   0       0       0       0       0       0       0       0
# String:
# Hosts        10.1.1.5 (1/50%), 10.8.0.2 (1/50%)
# Query_time distribution
#   1us
#  10us
# 100us
#   1ms  ################################################################
#  10ms
# 100ms
#    1s
#  10s+
SHOW SESSION STATUS LIKE 'XXXXX'\G
\n break line 
repeat

I want to run a script in python to get only some information from that file. There will be multiple querys.

Currently Im trying something like this:

#!/usr/bin/python

file = open("/etc/openvpn/logs/log-2020_01_29_06_20_PM.txt", "r")
read = file.read()
removeChar = read.replace("#", "")
for item in removeChar.split("\n"):
        if "Hosts" and "Time range" in item:
                print item.strip()

The output is:

Time range: 2020-01-29 18:18:59.073995 to 18:18:59.074005
Time range: 2020-01-29 18:18:58.489162 to 18:18:59.188582
Time range: 2020-01-29 18:18:58.666020 to 18:18:58.666028

I want it to be something like this:

['Query 1, 2020-01-29 18:18:59, 10.1.1.5, 10.8.0.2, SHOW SESSION STATUS LIKE 'XXXXX'\G']
['Query 2, 2020-01-29 18:19:59, 10.1.1.5, 10.8.0.2, SHOW FROM BLA * LIKE 'BLA'\G']

Im tired of trying to find how to do this, also I'm learning python because it is a good language to learn! :)

Thanks.

Answer 1

You could try without regex, using string manipulation only:

data = '''# Query 1: 204.60k QPS, 230.79x concurrency, ID XXXXXXXXXX at byte 19XXX9318
# This item is included in the report because it matches --limit.
# Scores: V/M = 0.00
# Time range: 2020-01-29 18:18:59.073995 to 18:18:59.074005
# Attribute    pct   total     min     max     avg     95%  stddev  median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count          7       2
# Exec time     10     2ms     1ms     1ms     1ms     1ms    12us     1ms
# Rows affecte   0       0       0       0       0       0       0       0
# Query size     7      74      37      37      37      37       0      37
# Warning coun   0       0       0       0       0       0       0       0
# String:
# Hosts        10.1.1.5 (1/50%), 10.8.0.2 (1/50%)
# Query_time distribution
#   1us
#  10us
# 100us
#   1ms  ################################################################
#  10ms
# 100ms
#    1s
#  10s+
SHOW SESSION STATUS LIKE 'XXXXX'\G
\n break line 
repeat
'''

data = data.split('\n')

all_results = []

result = []

for row in data:
    if row.startswith('# Query ') and not row.startswith('# Query size'):
        row = row.split(':')[0].split('# ')[1]
        result.append(row)

    elif row.startswith('# Hosts'):
        row = row.replace('# Hosts', '').replace(' ', '').split(',')
        result.append(row[0].split('(')[0])
        result.append(row[1].split('(')[0])

    elif row.startswith('# Time range:'):
        row = row.replace('# Time range:', '').split('.')[0].strip()
        result.append(row)

    elif row.startswith('SHOW') and row.endswith('\G'):
        result.append(row)
        result = ', '.join(result)
        all_results.append(result)
        result = []

print(all_results)

    # output: "Query 1, 2020-01-29 18:18:59, 10.1.1.5, 10.8.0.2, SHOW SESSION STATUS LIKE 'XXXXX'\\G"

Answer 2

If you want to use regex and hardcode, here is an example:

import re

file_path = 'file.txt'

query_pattern = r'Query \d+'
date_time_pattern = r'\d\d\d\d-(0?[1-9]|1[0-2])-(0?[1-9]|[12][0-9]|3[01]) ' \
                    r'(00|[0-9]|1[0-9]|2[0-3]):([0-9]|[0-5][0-9]):([0-5][0-9]|[0-9])'
hosts_pattern = r'\d+\.\d+\.\d\.\d+'
final_pattern = r"SHOW.+\\G"

result = ''

with open(file_path, 'r', encoding='utf-8') as f:
    try:
        text = f.read()
        result += '{}, '.format(str(re.search(query_pattern, text)[0]))
        result += '{}, '.format(str(re.search(date_time_pattern, text)[0]))
        hosts = re.findall(hosts_pattern, text)
        result += '{}, {}, '.format(hosts[0], hosts[1])
        result += re.search(final_pattern, text)[0]
        print('Done!')
    except Exception as e:
        print('Error: \n{}'.format(e))


print(result)

Output:

Query 1, 2020-01-29 18:18:59, 10.1.1.5, 10.8.0.2, SHOW SESSION STATUS LIKE 'XXXXX'\G