寫入大量數據時，部分數據丟失/每個數據都存在時，寫入過程非常緩慢

Question

將大量字符串寫入文件時，緩沖寫入器出現問題。

情況：我必須讀取一個大文本文件（> 100k 行）並對每一行進行一些修改（刪除空格、檢查可選命令等）並將修改后的內容寫入新文件。

我嘗試了兩種寫入文件的可能性，但只得到以下兩種結果之一：

1. 寫入過程非常緩慢，但所有行都已處理
2. 在寫入過程中，幾行代碼被咀嚼，留下不完整的修改結果。

方法和結果：

1. 非常緩慢但完整

// read file content and put it in List<String> fileContent
for (String line : fileContent)
{
  try(BufferedWriter writer = new BufferedWriter(new OutputStreamwriter(new FileOutputStream(filename, true))))
    {
      writer.write(modifyFileContent(fileContent));
    }
}

我已經知道，打開一個文件寫一行然后直接關閉它非常擅長表現不佳。 修改一個大約有 4M 行的文件需要大約 4 小時左右，這是不可取的。 至少，它有效......

2. 更快但不完整的寫入

// read file content and put it in List<String> fileContent
// This is placed in a try/catch block, I'm omitting it here for brevity
BufferedWriter writer = new BufferedWriter(new OutputStreamwriter(new FileOutputStream(filename, true);
for (String line : fileContent)
{
  writer.write(modifyFileContent(fileContent));
}
writer.close();

這工作得更快，但我在結果文件中得到以下內容（我使用原始文件中的行號進行調試）：

...
Very long line with interesting content // line nb 567
Very long line with interesting content // line nb 568
Very long line with interesting content // line nb 569
Very long line wi
Very long line with interesting content // line nb 834
Very long line with interesting content // line nb 835
Very long line with interesting content // line nb 836
...

將此字符串打印到控制台時，我在行號中看不到任何間隙！ 所以看起來，有一個緩沖問題......

其他方法：我也試過NIO版的newBufferedWriter，同樣省略了幾行。

問題：我在這里缺少什么？ 有沒有辦法在這里獲得正確的良好寫入性能？ 輸入文件通常在幾個 100MB 和數百萬行的區域內......任何提示都非常感謝:)

[編輯]

感謝洛佩茲爵士，我找到了一個可行的解決方案。 我以前從未偶然發現過RandomAccessFile ......

現在有了這些信息，我想我遇到了競爭條件或其他與線程相關的問題......因為我最近才開始使用線程，我想，這本來可以預料的......

為了給出正確的觀點，我做了一個最小的例子，它顯示了我的問題最初發生的上下文。 歡迎任何反饋:) :

package minex;

import java.awt.EventQueue;
import java.awt.event.ActionEvent;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.FileReader;
import java.io.IOException;
import java.io.LineNumberReader;
import java.io.OutputStreamWriter;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.swing.GroupLayout;
import static javax.swing.GroupLayout.Alignment.BASELINE;
import static javax.swing.GroupLayout.Alignment.LEADING;
import javax.swing.JButton;
import javax.swing.JFileChooser;
import javax.swing.JFrame;
import javax.swing.JProgressBar;
import javax.swing.SwingWorker;
import javax.swing.UIManager;
import javax.swing.WindowConstants;

/**
 * Read a file line by line, modify its content and write it to another file.
 * @author demo
 */
public class gui extends JFrame {

  /**
   * Back ground task, so the gui isn't blocked and the progress bar can be updated.
   */
  class fileConversionWorker extends SwingWorker<Integer, Double>
  {
    private final File file;
    
    public fileConversionWorker(File file)
    {
      this.file = file;
    }
 
    /**
     * Count the lines in the provided file. Needed to set the boundary
     * settings for the progress bar.
     * @param aFile File to read.
     * @return Number of lines present in aFile
     * @throws IOException 
     * @see quick and dirty taken from https://stackoverflow.com/a/1277955
     */
    private int countLines(File aFile) throws IOException {
    LineNumberReader reader = null;
    try {
        reader = new LineNumberReader(new FileReader(aFile));
        while ((reader.readLine()) != null);
        return reader.getLineNumber();
    } catch (Exception ex) {
        return -1;
    } finally { 
        if(reader != null) 
            reader.close();
    }
}
    
    /**
     * Reads a file line by line, modify the line
     * content and write it back to a different file immediately.
     * @return 
     */
    @Override
    public Integer doInBackground()
    {
      int totalLines = 0;
      try {
        // Indicate, that something is happening
        barProgress.setIndeterminate(true);
        totalLines = countLines(file);
        barProgress.setIndeterminate(false);
      } catch (IOException ex) {
        Logger.getLogger(gui.class.getName()).log(Level.SEVERE, null, ex);
      }
      
      // only proceed, when we at least have 1 line to manipulate.
      if (totalLines > 0)
      {
        BufferedReader br = null;
        BufferedWriter writer = null;
        try {
          barProgress.setMaximum(totalLines);
          br = new BufferedReader(new FileReader(file));
          String filename =  file.getAbsolutePath() + ".mod";
          long lineNb = 0;
          
          writer = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(filename, true)));
          
          String line;
          // Read original file, modify line and immediately write to new file
          while ((line = br.readLine()) != null)
          {
            writer.write(line + " // " + lineNb);
            writer.newLine();

            publish((double)(lineNb / totalLines));
            lineNb++;
          }
        } catch (FileNotFoundException ex) {
          Logger.getLogger(gui.class.getName()).log(Level.SEVERE, null, ex);
        } catch ( IOException ex) {
          Logger.getLogger(gui.class.getName()).log(Level.SEVERE, null, ex);
        }
        finally {
          // Tidying up
          try {
            if (br != null)
              br.close();
            if (writer != null)
              writer.close();
          } catch (IOException ex) {
            Logger.getLogger(gui.class.getName()).log(Level.SEVERE, null, ex);
          }
        }
      }
      return 0;
    }
    
    /**
     * Prevent any interaction, which could interrupt the worker
     */
    @Override 
    public void done()
    {
      butLoadFile.setEnabled(true); 
    }
    
    /**
     * Update progress the progress bar,
     * @param aDoubles
     */
    @Override
    protected void process(java.util.List<Double> aDoubles) {    
      int amount = barProgress.getMaximum() - barProgress.getMinimum();
      barProgress.setValue( ( int ) (barProgress.getMinimum() + ( amount * aDoubles.get( aDoubles.size() - 1 ))) );
    }
  }
  
  /**
   * Start the gui.
   */
  public static void main()
  {
    EventQueue.invokeLater(() -> {
      new gui().setVisible(true);
    });
  }
  
  /**
   * Initialize all things needed.
   */
  public gui()
  {
    initComponents();
  }
  
  /**
   * Load a file and immediately begin processing it.
   * @param evt 
   */
  private void butLoadFileActionListener(ActionEvent evt)
  {
    javax.swing.JFileChooser fc = new javax.swing.JFileChooser("/home/demo/fileFolder");
    int returnVal = fc.showOpenDialog(gui.this);
    
    if (returnVal == JFileChooser.APPROVE_OPTION) {
      File file = fc.getSelectedFile();
      butLoadFile.setEnabled(false);
      fileConversionWorker worker = new fileConversionWorker(file);
      worker.execute();
    }
  }
  
  /**
   * Paint the canvas.
   */
  private void initComponents()
  {
    setDefaultCloseOperation(WindowConstants.EXIT_ON_CLOSE);
    setResizable(false);
    setTitle("Min Example");
    
    butLoadFile = new JButton("Load file");
    butLoadFile.addActionListener((ActionEvent evt) -> {
      butLoadFileActionListener(evt);
    });
    
    barProgress = new JProgressBar();
    barProgress.setStringPainted(true);
    barProgress.setMinimum(0);
    
    javax.swing.GroupLayout layout = new GroupLayout(getContentPane());
    getContentPane().setLayout(layout);
    
    layout.setHorizontalGroup(
    layout.createParallelGroup(LEADING)
            .addComponent(butLoadFile, GroupLayout.PREFERRED_SIZE, 200, GroupLayout.PREFERRED_SIZE)
            .addComponent(barProgress, GroupLayout.PREFERRED_SIZE, 200, GroupLayout.PREFERRED_SIZE)
    );

    layout.setVerticalGroup(
    layout.createParallelGroup(BASELINE)
            .addGroup(layout.createSequentialGroup()
            .addComponent(butLoadFile, GroupLayout.PREFERRED_SIZE, 20, GroupLayout.PREFERRED_SIZE)
            .addComponent(barProgress, GroupLayout.PREFERRED_SIZE, 20, GroupLayout.PREFERRED_SIZE)            
            )
    );
    
    pack();
  }
  
  private JButton butLoadFile;        /** Button to load a file. */
  private JProgressBar barProgress;   /** Progress bar to visualize progress. */  
}

[/編輯]

Answer 1

也許這可以幫助你

在文本文件Java中寫入大量數據的最快方法

https://www.quora.com/How-do-to-read-and-write-large-size-file-in-Java-efficiently