我需要解析大型文本文件（大小为6GB）的建议

Question

I need advice on parsing a large text file - 6GB in size

What I have done is download all my Gmail using Thundervird I now have an mbox file with all my email in - this is a text file - of size 6GB

I need to parse this file and pull out specific data that follows a specific pattern

First question : what language should I use? I've searched some other threads similar to this and understand that Perl or Python (and one or 2 others) would be fine

Second question though : I read in one of the post replies that it might be better to load the text file into a database and let the database search through the text file?

I need to have a CSV generated as an output

So... is it wiser for me to go the DB route?

Third question : How long is a piece of string... erm I mean... how long will it take to go through my 6Gb file... OK, not possible to answer without some details!

I need to pull out the following data:

First Name: 
Last Name: 

Address:

Telephone: 
Mobile:
Email:

So... I need to know if I need to run the script and leave my machine running overnight I'm not sure if the above is a really dumb question or not - but I thought I'd ask anyway

ANY replies would be great

Thanks

Omar

Answer 1

1. You should use whatever language you're more familiar with. In terms of performance, Perl programs generally can parse text data faster than python.

2. You need to parse the data regardless of using database or not. If you're going to be doing a lot of queries/searches afterwards, then you should consider loading the parsed data into a database.

3. Depends on how complex the pattern you're trying to match on. Probably no more than 1 hour.

Answer 2

You can use the aperture project to query the mbox contents:

http://aperture.sourceforge.net/