I want my file to get parsed by url but some of the url have missing parameters and when I am iterating through lines of log I am getting error of missing parameter. I need to append blank or null value into in parse list so that I can transform it into data frame
My Data File : log file
"GET /pixel.gife=heartbeat&creative_id=33548&in_view_time=290"
"GET/pixel.gife=heartbeat&creative_id=33548&in_view_time=23988"
"GET /pixel.gif?e=heartbeat&creative_id=33548&in_view_time=19183"
"GET /pixel.gif?e=ad_load&creative_id=33548"
I want Output as :
E | Creative ID | IN VIEW TIME
heartbeat 33548 290
heartbeat 33548 23988
ad_load 33548 null
My Code:
parselist = []
for eachline in log.readlines():
ip_regex = re.findall(r'(\d{18})', eachline)
date = re.findall(r'([0-9]{4}\-[0-9]{2}\-[0-9]{2})',eachline)
url = eachline
parsed = urlparse.urlparse(url)
parselist.append(ip_regex)
parselist.append(date)
parselist.append(urlparse.parse_qs(parsed.query)['e'])
parselist.append(urlparse.parse_qs(parsed.query)['account_id'])
parselist.append(urlparse.parse_qs(parsed.query)['impression_id'])
parselist.append(urlparse.parse_qs(parsed.query)['campaign_id'])
parselist.append(urlparse.parse_qs(parsed.query)['creative_id'])
parselist.append(urlparse.parse_qs(parsed.query)['in_view_time'])
Error I am getting as in_view_time parameter is missing in third line:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-6-405c1bfb329e> in <module>()
12 parselist.append(urlparse.parse_qs(parsed.query)['campaign_id'])
13 parselist.append(urlparse.parse_qs(parsed.query)['creative_id'])
---> 14 parselist.append(urlparse.parse_qs(parsed.query)['in_view_time'])
KeyError: 'in_view_time'
You can use try
and except
:
parselist = []
for eachline in log.readlines():
ip_regex = re.findall(r'(\d{18})', eachline)
date = re.findall(r'([0-9]{4}\-[0-9]{2}\-[0-9]{2})',eachline)
url = eachline
parsed = urlparse.urlparse(url)
parselist.append(ip_regex)
parselist.append(date)
try:
parselist.append(urlparse.parse_qs(parsed.query)['e'])
except:
parselist.append('Null')
try:
parselist.append(urlparse.parse_qs(parsed.query)['account_id'])
except:
parselist.append('Null')
try:
parselist.append(urlparse.parse_qs(parsed.query)['impression_id'])
except:
parselist.append('Null')
try:
parselist.append(urlparse.parse_qs(parsed.query)['campaign_id'])
except:
parselist.append('Null')
try:
parselist.append(urlparse.parse_qs(parsed.query)['creative_id'])
except:
parselist.append('Null')
try:
parselist.append(urlparse.parse_qs(parsed.query)['in_view_time'])
except:
parselist.append('Null')
Or, in a more compact way:
parselist = []
for eachline in log.readlines():
ip_regex = re.findall(r'(\d{18})', eachline)
date = re.findall(r'([0-9]{4}\-[0-9]{2}\-[0-9]{2})',eachline)
url = eachline
parsed = urlparse.urlparse(url)
parselist.append(ip_regex)
parselist.append(date)
for key in ['e','account_id','impression_id','campaign_id','creative_id','in_view_time']:
try:
parselist.append(urlparse.parse_qs(parsed.query)[key])
except:
parselist.append('Null')
As a suggestion, instead of 'Null'
you can append None
.
for v in urlparse.parse_qs(parsed.query).values():
parselist.append(v)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.