Storing Click Data in MongoDB

Question

My application tracks clicks from adverts shown on remote sites, and redirects users to a product sales page.

I'm currently using MySQL to store click information (date, which link was used, ip address, custom data sent from the advertiser etc). The table is getting so big that it no longer fits our needs, which are:

• High throughput (the app is processing 5 - 10M clicks per day and this is projected to grow)

• Ability to report on the data by date range (eg how many clicks for link 1 over the past month grouped by country)

My initial idea was to move clicks into Redis (we only need to store them for 30 days, at which point they expire if they don't lead to a sale) and then make a new MySQL table to store generated stats by day, where we just update a counter per link when it's clicked.

When we started using the statistics table the database quickly fell over because of the amount of queries to that table.

Would it be best to keep the clicks in Redis, and have a separate MongoDB (or other noSQL DB) for the reporting? or could Mongo be used to store the whole click (just like we've been doing in MySQL) or is the volume too high?

Also I remember reading that MongoDB is not good at reclaiming space from deleted records, would this cause us issues since 90% of the clicks would be deleted after 30 days anyway?

Thanks

Answer 1

MongoDB is enough to solve this problem as compared to store in Radis and move to MongoDB. Since, the amount of data is very large, so you can create a indexes on timestamp or field having high carnality. This make you query fast, also MongoDB provide aggregation which help in generating the report. I don't think, is there any issue with deletion.