逐行解析文本文件并将数据组织到嵌套字典中？

Question

I have a file that runs a CMD trace route command whenever I get a ping timeout, and I print that out to a file. I have a file formatted as follows:

Sun 02/17/2019 13:20:44.27 PING ERROR 1

Tracing route to _____________ [IP_REDACTED]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2    <1 ms    <1 ms    <1 ms  [IP_REDACTED]
  3     1 ms    <1 ms     1 ms  [IP_REDACTED]
  4     *        *        *     Request timed out.
  5     7 ms    10 ms     6 ms  [IP_REDACTED]
  6     8 ms     4 ms     6 ms  [IP_REDACTED]
  7     5 ms     7 ms     6 ms  [IP_REDACTED]

Trace complete.

Sun 02/17/2019 13:45:59.27 PING ERROR 2

Tracing route to _____________ [IP_REDACTED]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2    <1 ms    <1 ms    <1 ms  [IP_REDACTED]
  3     1 ms    <1 ms     1 ms  [IP_REDACTED]
  4    23 ms     *        *     [IP_REDACTED]
  5     7 ms    10 ms     6 ms  [IP_REDACTED]
  6     8 ms     4 ms     6 ms  [IP_REDACTED]
  7     5 ms     7 ms     6 ms  [IP_REDACTED]

Trace complete.

Sun 02/17/2019 15:45:59.27 PING ERROR 3

Tracing route to _____________ [IP_REDACTED]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2    <1 ms    <1 ms    <1 ms  [IP_REDACTED]
  3     1 ms    <1 ms     1 ms  [IP_REDACTED]
  4    23 ms    12 ms    11 ms  [IP_REDACTED]
  5     7 ms    10 ms     6 ms  [IP_REDACTED]
  6     8 ms     *        6 ms  [IP_REDACTED]
  7     5 ms     7 ms     6 ms  [IP_REDACTED]

Trace complete.

The first line has the timestamp of the trace route command. I want to use Python to graph the number of times hop # 4 loses a packet (signaled by a '*' character) over time. Before I cross that road, I need to organize the data. I figured nested dictionaries is the way to go with python.

I am new to Python and the syntax is confusing.

The code below shows my attempt. This is the basic flow I was going for:

1. look at a line in the file.

2. If line has the word "ERROR", save that line

3. Look at the other lines. if line starts with "4", parse the data from step #2

4. Grab month, day, hour, and min and put those into separate variables

5. Create nested dictionary with this data.

6. Repeat these steps for all the errors in the file. Add to dictionary of step #5

7. At the end, be able to print the data from any scope (eg number of errors in a day, or number of errors in a specific hour of a day)

For example, a dictionary might look like this:

day{ 6 : hour{ 2 : min{ 15 : 2, 30 : 1, 59 : 1 }, 9 : min{ 10: 1 }}}

There were 4 errors in hour 2 of day 6. These errors happened in minute 15, 20, and 59.

day_d = {}

with open("2019-02-17_11-54-AM.log", "r") as fo:

    for line in fo:
        list = line.strip() # Expected: each index in list is a word
        if list.count('ERROR'):
            # Save the line to parse if trace route reports
            # bad data on hop 4
            lineToParse = line

        if "4" in list[0]:
            # We found the line that starts with "4"
            if "*" in list[1] or "*" in list[2] or "*" in list[3]:
                # We should parse the data in lineToParse

                # Expected: lineToParse[1] = "02/17/2019"
                word  = lineToParse[1].split("/")
                month = word[0] # I don't care about month
                day   = word[1]
                year  = word[2] # I don't care about year

                # Expected: lineToParse[2] == "13:20:44.27"
                word = lineToParse[2].split(":")
                hour = word[0]
                min  = word[1]
                sec  = word[2] # I don't care about seconds


                # Keep track of number occurances in min
                if day in day_d:
                    if hour in day_d[day]:
                        if min in day_d[day[hour]]
                            day_d[day[hour[min]]] += 1
                        else:
                            day_d[day[hour[min]]] = 1
                    else:
                        min_d = { min : 1 }
                        day_d[day[hour]] = min_d
                else:
                    min_d = { min : 1 }
                    hour_d = { hour : min_d }
                    day_d[day] = hour_d


#Print number of occurances in hour "12" of day "01"
hourCounter = 0;
if "01" in day_d:
    if "12" in day:
        day["12"] = hour_d
        for min in hour_d:
            hourCounter += int(hour_d[min], 10) # Convert string to base 10 int
print(hourCounter)

EDIT: After reviewing Gnudiff's reply, I was able to accomplish what I wanted to do. My code is as follows:

from matplotlib import pyplot as plt
from matplotlib import style

style.use('ggplot')

from datetime import datetime as DT

ping_errors = dict()
data = dict()

with open("2019-02-17_02-41-PM.log", "r") as fo:
    for line in fo:
        if 'ERROR' in line: # A tracert printout will follow
            pingtime = DT.strptime(line[:23],'%a %m/%d/%Y %H:%M:%S') # fixed format datetime format allows us just to cut the string precisely
        words = line.strip().split()
        if len(words) > 0:
            if words[0] == '4':
                if '*' in line:
                    # Found packet timeout in hop # 4
                    ping_errors[pingtime] = 1


# Create key value pairs. Keys are the hours from 1 to 24
# and values are the drops for each hour.
for i in range(1,24):
    data[i] = 0
    for x in ping_errors.keys():
        if x.time().hour == i:
            data[i] += 1


# Prepare the chart         
x_axis = list(data.keys())
y_axis = list(data.values())

fig, ax = plt.subplots()

ax.bar(x_axis, y_axis, align='center')

ax.set_title('10-second drops from ___ to ____')
ax.set_ylabel('Number of drops')
ax.set_xlabel('Hour')

ax.set_xticks(x_axis)
ax.set_yticks(y_axis)

plt.show()

Answer 1

Nested dictionaries do not really look like the right tool for the job, because the syntax and the storage becomes convoluted, as you yourself have found out.

What you are having is something much more tabular in format already from the output of ping and it will become much easier to deal with, if you keep it as such.

So, you want to store ping errors and be able to find how many happened at what time.

If this was a large project, I would probably store and query the data against some external database. But let's see how that could work with Python.

Here are some things which won't work and which need to be changed:

1) as Laurent mentioned in comments, you can't use reserved words for variable names. In this case "list" should be renamed something else

2) line is a string and your line.strip() still remains a string, not a list. if you want to split the line string by spaces, you should use something like: linewords=line.split() #and use this variable instead of your list variable

3) for attempting date and time manipulation it is generally very helpful to use appropriate modules. In this case datetime.datetime

So, the start of your loop could go something like this:

from datetime import datetime as DT

ping_errors=dict()

with open("2019-02-17_11-54-AM.log", "r") as fo:
    firstline=fo.readline()
    if 'ERROR' in firstline: # this file has an error, so we will process it
       pingtime=DT.strptime(firstline[:23],'%a %m/%d/%Y %H:%M:%S') # fixed format datetime format allows us just to cut the string precisely
       ping_errors[pingtime]=list()
       for line in fo:
           words=line.strip().split()
           if words[1]=='*':
              # this is the hop with error, add its info to this
              ping_errors[pingtime].append(words[0]) # add the number of hop which had the error

after this you have a nice unnested dict ping_errors , which is indexed on datetime with precision to second (yes, you needn't index dict on strings although it might be more useful usually), so you can use filter on the dict to query the times you are interested in.

The dict will look like this:

{datetime.datetime(2019, 2, 17, 13, 20, 44): [4], datetime.datetime(2019, 2, 17, 13, 33, 11): [7, 8]}

which means that at 17/feb/2019, 13:20:44 we had 1 error in hop 4 and at 17/feb/2019, 13:33:11 we had 2 errors, in hop 7 & 8 respectively.

Then, for example, a query to select how many pings have had errors been at 13 o'clock (any minute and second), would be (for the above fictional dict):

sum([len(ping_errors[x]) for x in ping_errors.keys() if x.time().hour==13])

which hops were affected during this time?

[ping_errors[x] for x in ping_errors.keys() if x.time().hour==13]

what if we are only interested in seconds between 30 and 45?

sum([len(ping_errors[x]) for x in ping_errors.keys() if x.time().second >= 30 and x.time().second <= 45 ])