Python - Add matched square bracket with numbers regular expression to 2d Array

Question

I am trying to write some python to parse through a text log. There is a time stamp on most lines in the log, and if the log contains certain text, I would like to capture the time stamps in a 2d array so that I can later compare a value to the values in the 2d array.

An example log file looks like:

[1542053213 s] Starting Program TEE:
-----------------------------------------
[1542053213 s] PROGRAM ERROR
ERRHAND: 1033
ERRHAND: 233545
ERRHAND: 1
[1542053213 s] Program completed!

[1542053300 s] Ending Program. Stopping!


[1542053500 s] Starting Program FOO:
-----------------------------------------
[1542060100 s] Program completed!   

[1542060105 s] Ending Program. Stopping!

My code currently is able to appropriately detect the start and stop lines:

from os.path import expanduser
import re

#Setting variables
filepath = expanduser('~/LogAutomation/programlog.txt')
timeArray = []
tempStart = ''
tempEnd = ''

with open(filepath) as myFile:
    line = myFile.readline()
    cnt = 1
    while line:
        if ("Starting Program") in line:
            print("MATCH FOUND FOR STARTING PROGRAM")
            print("Line {}: {}".format(cnt, line.strip()))
            #Add timestamp to time array
                #tempStart = matched regular expression

        elif ("Ending Program. Stopping!") in line:
            print("MATCH FOUND FOR ENDING PROGRAM")
            print("Line {}: {}".format(cnt, line.strip()))
            #Add timestamp to time array
                #tempEnd = matched regular expression

        #Add [tempStart, tempEnd] to timeArray

        line = myFile.readline()
        cnt +=1

myFile.close()

print("Completed Script")

However, I am trying to add the commented functionality. I am not sure where to begin to do so. I have tried using https://pythex.org/ in order to create a regular expression and am trying to edit the following in order to extract the number sequence out of the [digits(space)s] sequence: (\\[[,\\d]*) but it is not matching the example timestamps in the log above. It is currently including the beginning bracket, which I don't want.

Any assistance with determining how to extract the digits, and add items to the 2d array, would be greatly appreciated.

NOTE: I am a brand new Python developer so please excuse me if this is an easy problem. I am trying quickly to learn.

EDIT --- Here is an updated code base using a suggestion below:

from os.path import expanduser
import re

#Setting variables
filepath = expanduser('~/LogAutomation/programlog.txt')
timeArray = []
tempStart = ''
tempEnd = ''

# with open(filepath) as myFile:
#     line = myFile.readline()
#     cnt = 1
#     while line:
#         if ("Starting Program") in line:
#             print("MATCH FOUND FOR STARTING PROGRAM")
#             print("Line {}: {}".format(cnt, line.strip()))
#             #Add timestamp to time array
#                 #tempStart = matched regular expression
#
#         elif ("Ending Program. Stopping!") in line:
#             print("MATCH FOUND FOR ENDING PROGRAM")
#             print("Line {}: {}".format(cnt, line.strip()))
#             #Add timestamp to time array
#                 #tempEnd = matched regular expression
#
#         #Add [tempStart, tempEnd] to timeArray
#
#         line = myFile.readline()
#         cnt +=1
#
# myFile.close()

import re

with open(filepath) as myFile:
    all_logs = myFile.read()
    starting_regex = re.compile(r'\[(\d+)\s+s\]\s+Starting\s+Program')
    ending_regex = re.compile(r'\[(\d+)\s+s\]\s+Ending\s+Program\.\s+Stopping')

    start_times = list(map(int, starting_regex.findall(all_logs)))
    end_times = list(map(int, ending_regex.findall(all_logs)))

    timeArray = list(map(list, zip(start_times, end_times)))

    for x in timeArray:
        print(x)
        print(timeArray[x])

print("Completed Script")

Answer 1

To capture just the digits, you need to put only the characters matching the digits between the parentheses, like so: (\\d+)

To match the full square bracket at the beginning, with the space ( \\s denotes a whitespace character): \\[(\\d+)\\s

If you want to be even more specific (while still only capturing digits) by matching the full time stamp up to the closing square bracket: \\[(\\d+) s\\] , where the space between the ) paren and the s is significant.

The complete regex expression (assuming the use of .match() ) could then be:

regex = "\[(\d+)\s"
tempStart = re.match(regex, line).group(1)

To put the values in a 2d array, you're right in that [tempStart, tempEnd] basically forms a row. To add this to timeArray , we can use:

timeArray.append([tempStart, tempEnd])

Answer 2

Let's first create a regex for Starting Program:

\[(\d+)\s+s\]\s+Starting\s+Program

This is just a simple regex which will match strings like [1542053213 s] Starting Program .

But if you notice, I have added a pair of parenthesis with \\d+ (the digits part), adding the parenthesis makes it a group in the regex, and this is the only group we have in this regex.

Similarly, I will create a regex for Ending Program, making the digits a group:

\[(\d+)\s+s\]\s+Ending\s+Program\.\s+Stopping

Now, let's use them in a python program, and pick the groups:

import re

with open(filepath) as myFile:
    all_logs = myFile.read()
    starting_regex = re.compile(r'\[(\d+)\s+s\]\s+Starting\s+Program')
    ending_regex = re.compile(r'\[(\d+)\s+s\]\s+Ending\s+Program\.\s+Stopping')

    start_times = list(map(int, starting_regex.findall(all_logs)))
    end_times = list(map(int, ending_regex.findall(all_logs)))

    timeArray = list(map(list, zip(start_times, end_times)))

In the program above, I have read all of the log file at once, and used findall method to find all of the occurrences in the all_logs that the regex will match.

Now, the interesting part is: findall only returns the list of values matching to groups, and we only have one group in the regex, which is (\\d+) , so, start_times and end_times will just contain the time values.

The corresponding values of start_times and end_times contain the time of starting program and ending program.

So, to make a pair of each, let's just zip both lists. It will create tuple s of each of the pair, and let's just convert each tuple to list using map method, and then convert the map object to list to make a list of lists.

timeArray will be a list of lists now. Each sub-list containing only two elements, first being the starting time of program, and second being the ending time.

Edit :

You don't actually need a loop to print out the values of timeArray . You can just use a print statement to see the values, it'll print the entire list:

print(timeArray)