将多个字典与 python 合并

Question

I have a list of strings from which I want to extract all relevant information using regex . I have written a pattern to extract the information I need. the pattern is as follows

pattern1 = "(?P<host>\d*\.\d*\.\d*\.\d*[^0-9]*) - (?P<user_name>\w*\d*)|(?P<time>\d*/\w*/\d*:\d*:\d*:\d* -\d*)|\"(?P<request>[A-Z]* (/[a-z+]*)+ [A-Z]*/\d\.\d)"

result = [item.groupdict() for item in re.finditer(pattern1,logdata)]

Multiple dictionaries are generated as follows. This is the answer I am looking for

[{'host': '146.204.224.152',
  'user_name': 'feest6811',
  'time': None,
  'request': None},
 {'host': None,
  'user_name': None,
  'time': '21/Jun/2019:15:45:24 -0700',
  'request': None},
 {'host': None,
  'user_name': None,
  'time': None,
  'request': 'POST /incentivize HTTP/1.1'},

 ...

]

In the output 3 dictionaries are formed each containing a piece of the required information. I want a single dictionary with all the information as follows

{
  'host': '146.204.224.152',
  'user_name': 'feest6811',
  'time': '21/Jun/2019:15:45:24 -0700',
  'request': 'POST /incentivize HTTP/1.1'
}

I am not sure what I am doing wrong. Is there any way to merge the dictionaries as they are formed?

these are few samples from logdata

'146.204.224.152 - feest6811 [21/Jun/2019:15:45:24 -0700] "POST /incentivize HTTP/1.1" 302 4622',
 '197.109.77.178 - kertzmann3129 [21/Jun/2019:15:45:25 -0700] "DELETE /virtual/solutions/target/web+services HTTP/2.0" 203 26554',
 '156.127.178.177 - okuneva5222[21/Jun/2019:15:45:27 -0700] "DELETE /interactive/transparent/niches/revolutionize HTTP/1.1" 416 14701',

Answer 1

This is one approach using groupdict

Ex:

logdata = ['146.204.224.152 - feest6811 [21/Jun/2019:15:45:24 -0700] "POST /incentivize HTTP/1.1" 302 4622',
 '197.109.77.178 - kertzmann3129 [21/Jun/2019:15:45:25 -0700] "DELETE /virtual/solutions/target/web+services HTTP/2.0" 203 26554',
 '156.127.178.177 - okuneva5222[21/Jun/2019:15:45:27 -0700] "DELETE /interactive/transparent/niches/revolutionize HTTP/1.1" 416 14701']

result = []
pattern1 = re.compile('(?P<host>\d*\.\d*\.\d*\.\d*[^0-9]*)\s*-\s*(?P<user_name>\w*\d*)\s*\[(?P<time>\d*/\w*/\d*:\d*:\d*:\d* -\d*)\]\s*"(?P<request>[A-Z]* (/[a-z+]*)+ [A-Z]*/\d\.\d)')
for log in logdata:
    m = pattern1.match(log)
    if m:
        result.append(m.groupdict())

print(result)

Output:

[{'host': '146.204.224.152',
  'request': 'POST /incentivize HTTP/1.1',
  'time': '21/Jun/2019:15:45:24 -0700',
  'user_name': 'feest6811'},
 {'host': '197.109.77.178',
  'request': 'DELETE /virtual/solutions/target/web+services HTTP/2.0',
  'time': '21/Jun/2019:15:45:25 -0700',
  'user_name': 'kertzmann3129'},
 {'host': '156.127.178.177',
  'request': 'DELETE /interactive/transparent/niches/revolutionize HTTP/1.1',
  'time': '21/Jun/2019:15:45:27 -0700',
  'user_name': 'okuneva5222'}]

Answer 2

You can just modify your regex pattern slightly, so that it matches all groups at the same time instead of matching either of the 3 capturing groups.

pattern1 = "(?P<host>\d*\.\d*\.\d*\.\d*[^0-9]*) - (?P<user_name>\w*\d*) ?\[(?P<time>\d*/\w*/\d*:\d*:\d*:\d* -\d*)\] \"(?P<request>[A-Z]* (/[a-z+]*)+ [A-Z]*/\d\.\d)"

Answer 3

After changing your pattern like this, it seems to work:

# modified matching pattern
pattern1 = "(?P<host>\d*\.\d*\.\d*\.\d*[^0-9]*) - (?P<user_name>\w*\d*).*(?P<time>\d*/\w*/\d*:\d*:\d*:\d* -\d*).*\"(?P<request>[A-Z]* (/[a-z+]*)+ [A-Z]*/\d\.\d)"
logdata = ['146.204.224.152 - feest6811 [21/Jun/2019:15:45:24 -0700] "POST /incentivize HTTP/1.1" 302 4622', '197.109.77.178 - kertzmann3129 [21/Jun/2019:15:45:25 -0700] "DELETE /virtual/solutions/target/web+services HTTP/2.0" 203 26554', '156.127.178.177 - okuneva5222[21/Jun/2019:15:45:27 -0700] "DELETE /interactive/transparent/niches/revolutionize HTTP/1.1" 416 14701']

result = []

for log in logdata:
     result.append([item.groupdict() for item in re.finditer(pattern1, log)][0])

gives

{'host': '146.204.224.152', 'user_name': 'feest6811', 'time': '/Jun/2019:15:45:24 -0700', 'request': 'POST /incentivize HTTP/1.1'}
{'host': '197.109.77.178', 'user_name': 'kertzmann3129', 'time': '/Jun/2019:15:45:25 -0700', 'request': 'DELETE /virtual/solutions/target/web+services HTTP/2.0'}
{'host': '156.127.178.177', 'user_name': 'okuneva5222', 'time': '/Jun/2019:15:45:27 -0700', 'request': 'DELETE /interactive/transparent/niches/revolutionize HTTP/1.1'}

Answer 4

You can make the pattern a bit more specific using {n} as a quantifier instead of mostly * with only 4 named capture groups.

(?P<host>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s*-\s*(?P<user_name>\w+)\s*\[(?P<time>\d+/[a-zA-Z]+/\d{4}:\d{2}:\d{2}:\d{2} -\d{4})\]\s*"(?P<request>[A-Z]+ /[^"]*)"

Regex demo

import re
pattern1 = r"(?P<host>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s*-\s*(?P<user_name>\w+)\s*\[(?P<time>\d+/[a-zA-Z]+/\d{4}:\d{2}:\d{2}:\d{2} -\d{4})\]\s*\"(?P<request>[A-Z]+ /[^\"]*)\""
logdata = ("146.204.224.152 - feest6811 [21/Jun/2019:15:45:24 -0700] \"POST /incentivize HTTP/1.1\" 302 4622\n"
            "197.109.77.178 - kertzmann3129 [21/Jun/2019:15:45:25 -0700] \"DELETE /virtual/solutions/target/web+services HTTP/2.0\" 203 26554\n"
            "156.127.178.177 - okuneva5222[21/Jun/2019:15:45:27 -0700] \"DELETE /interactive/transparent/niches/revolutionize HTTP/1.1\" 416 14701")
result = [item.groupdict() for item in re.finditer(pattern1, logdata)]
print(result)

Output

[
{'host': '146.204.224.152',
'user_name': 'feest6811',
'time': '21/Jun/2019:15:45:24 -0700',
'request': 'POST /incentivize HTTP/1.1'
},
{
'host': '197.109.77.178',
'user_name': 'kertzmann3129',
'time': '21/Jun/2019:15:45:25 -0700',
'request': 'DELETE /virtual/solutions/target/web+services HTTP/2.0'
},
{
'host': '156.127.178.177',
'user_name': 'okuneva5222',
'time': '21/Jun/2019:15:45:27 -0700',
'request': 'DELETE /interactive/transparent/niches/revolutionize HTTP/1.1'
}
]