使用bigquery分析iis日志

Question

Any preferred way/example to load and analyze IIS logs (in Extended Log File Format ) using bigquery? we will also need to auto-partition it. we can get log files periodically

we want to analyze Usage of a particular feature, which can be identified by a particular URL pattern and a conversion funnel of most popular flows that visitors take through the website, to identify where they come in and leave. Visitors can be identified with a unique ID in a cookie (stored in the logs) and pages can be linked with the referer (also stored in the logs).

Thanks in advance

Answer 1

It's easy to load CSV format files into BigQuery. Both CSV and JSON format source data is supported.

I am not an expert in using IIS, but the quickest way to load flat log data into BigQuery is to start with CSV. IIS log format is pretty straightforward t work with, but you might want save a step and export it into CSV. A quick search shows that many people use LogParser (note: I have never used it myself) to convert IIS logs into CSV. Perhaps give this or a similar tool a try.

As for "auto-partioning" your BigQuery dataset tables - BigQuery doesn't do this automatically, but it's fairly easy to create a new table for each batch of IIS logs you export.

Depending on the volume of data you are analysing, you should create a new BigQuery table per day or hour.

Scripting this on the command line is pretty easy when using the BigQuery command line tool . Create a new BigQuery load job, with a new table name based on each timeslice of log data you have.

In other words, your BigQuery tables should look something like this:

mydataset.logs_2012_10_29
mydataset.logs_2012_10_30
mydataset.logs_2012_10_31
etc...

For more information, make sure you read through the BigQuery documentation for importing data .