使用 java 从一个大文件读取并写入多个文件

Question

I have an A.txt file of 100,000,000 records from 1 to 100000000, each record is one line. I have to read file A then write to file B and C, provided that even line writes to file B and the odd line writes to file C. Required read and write time must be less than 40 seconds. Below is the code that I already have but the runtime takes more than 50 seconds. Does anyone have any other solution to reduce runtime?

Threading.java

import java.io.*;
import java.util.concurrent.LinkedBlockingQueue;

public class Threading implements Runnable {
    LinkedBlockingQueue<String> queue = new LinkedBlockingQueue<>();
    String file;
    Boolean stop = false;
    
    public Threading(String file) {
        this.file = file;
    }

    public void addQueue(String row) {
        queue.add();
    }
    
    public void Stop() {
        stop = true;
    }
    
    public void run() {
        try {
            BufferedWriter bw = new BufferedWriter(new FileWriter(file));
            while(!stop) {
                try {
                    String rơ = queue.take();
                    bw.while(row + "\n");
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
            bw.close();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

ThreadCreate.java

// I used 2 threads to write to 2 files B and C

import java.io.*;
import java.util.List;

public class ThreadCreate {
    public void startThread(File file) {
        Threading t1 = new Threading("B.txt");
        Threading t1 = new Threading("B.txt");
        Thread td1 = new Thread(t1);
        Thread td1 = new Thread(t1);
        td1.start();
        td2.start();
        
        try {
            BufferedReader br = new BufferedReader(new FileReader(file));
            String line;
            long start = System.currentTimeMillis();
            while ((line = br.readLine()) != null) {
                if (Integer.parseInt(line) % 2 == 0) {
                    t1.addQueue(line);
                } else {
                    t2.addQueue(line);
                }
            }
            t1.Stop();
            t2.Stop();
            br.close();
            long end = System.currentTimeMillis();
            System.out.println("Time to read file A and write file B, C: " + ((end - start)/1000) + "s");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Main.java

import java.io.*;

public class Main {
    public static void main(String[] args) throws IOException {
        File file = new File("A.txt");
        
        //Write file B, C
        ThreadCreate t = new ThreadCreate();
        t.startThread(file);
    }
}

Answer 1

Why are you making threads? That just slows things down. Threads are useful if the bottleneck is either the calculation itself or the blocking nature of the operation, and they only hurt if it is not. Here, it isn't: The CPU is just idling (the bottleneck will be the disk), and the nature of what it is blocking on means that multithreading does not help either: Telling a single SSD to write 2 boatloads of bytes in parallel is probably no faster (only slower, as it needs to bounce back and forth). If the target disk is a spinning disk, it is way slower - the write head cannot make clones of itself to go any faster, and by making it multithreaded, you are wasting a ton of time by asking the write head to bounce back and forth between the different write locations.

There's nothing that immediately strikes me as ripe for significant speedups.

Sometimes, writing a ton of data to a disk just takes 50 seconds. If that's not acceptable, buy a faster disk.

Answer 2

try memory mapped files

   byte[] buffer = "foo bar foo bar text\n".getBytes();
int number_of_lines = 100000000;

FileChannel file = new RandomAccessFile("writeFIle.txt", "rw").getChannel();
ByteBuffer wrBuf = file.map(FileChannel.MapMode.READ_WRITE, 0, buffer.length * number_of_lines);
for (int i = 0; i < number_of_lines; i++)
{
    wrBuf.put(buffer);
}
file.close();

Took to my computer (Dell, I7 processor, with SSD, 32GB RAM) a little over half a minute to run this code)