Python script analyzing log files

Question

I am creating a python script to analyze a log file(example: http://www.monitorware.com/en/logsamples/apache.php ) and needs suggestions of methods that I can use to achieve this.

1. reading the log file with open method. OK.
2. counting elapsed time of the log? Which method do I use to read from specific position within the log line? Example: 64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846 How do I count elapsed time after a certain position like year in this case?
   
   I also need to display requests served per minute, total amount of data sent, average amount of data sent, largest amount of data sent in a single request.

Answer 1

I'd use re module to extract information from text (search, substr, etc.).

Then strptime could be your friend to convert those date-time values to datetime and do calculations on them.

After that I'd use numpy.ndarray to store the data and generate further reports.

I also like matplotlib and features like hist to make some nice plots.