Reading Large Text File

Question

i am currently encounter a problem reading a relatively huge text file using java 1.4.

i am trying to read a text file with 100 character per line and the file can be up to 1 million line or more line.

currently i am using a BufferedReader with a filereader approach to read the file line by line for processing, but my app server always hang on me after awhile. is there a better way to read the file and to process into my database?

thank

Answer 1

Firstly, I wouldn't use Java 1.4 if at all possible. eg If you are writing code for a Blackberry, you have no choice. Java 5.0 was released 7 years ago and even it is End Of Life for free support. The latest version is Java 6 update 26.

Reading 100 MB of text shouldn't take very long, I should takes more than 5 seconds. If you are running slowly it is likely you are running low on memory. Before Java 6, running low on memory could cause the system to run slower and slower rather than fail.

If you need to load the data into a database, I suggest you load modest portions of data at a time. This way you will load the last lines as fast as the first.

If you are loading one million entries into a database, you need to check how fast your database is. Loading batches at a time you might be able to insert 10K records per second and one million records in two minutes. However a poorly configured server adding one row at a time could perform only 100 records per second taking 3.5 hours.

Say you want to process some data every 1000 lines.

List<String> lines = new ArrayList<String>();
String line;
while((line = br.readLine()) != null) {
    lines.add(line);
    if(lines.size() >= 1000) {
       process(lines);
       lines.clear();
    }
}
process(lines); // get the last lines.