[英]Fastest way to write to file?
I made a method that takes a File
and a String
.我做了一个采用
File
和String
的方法。 It replaces the file with a new file with that string as its contents.它用该字符串作为其内容的新文件替换该文件。
This is what I made:这是我做的:
public static void Save(File file, String textToSave) {
file.delete();
try {
BufferedWriter out = new BufferedWriter(new FileWriter(file));
out.write(textToSave);
out.close();
} catch (IOException e) {
}
}
However it is painfully slow.然而,它的速度非常缓慢。 It sometimes takes over a minute.
有时需要一分钟以上。
How can I write large files with tens of thousands to maybe up to a million characters in them?如何编写包含数万到一百万个字符的大文件?
Make sure you allocate a large enough buffer: 确保分配足够大的缓冲区:
BufferedWriter out = new BufferedWriter(new FileWriter(file), 32768);
What sort of OS are you running on? 你在运行什么样的操作系统? That can make a big difference too.
这也可以产生很大的不同。 However, taking a minute to write out a file of less-than-enormous size sounds like a system problem.
然而,花一分钟写出一个不太大的文件听起来像是一个系统问题。 On Linux or other *ix systems, you can use things like
strace
to see if the JVM is making lots of unnecessary system calls. 在Linux或其他* ix系统上,您可以使用
strace
东西来查看JVM是否正在进行大量不必要的系统调用。 (A very long time ago, Java I/O was pretty dumb and would make insane numbers of low-level write()
system calls if you weren't careful, but when I say "a long time ago" I mean 1998 or so.) (很久以前,Java I / O非常愚蠢,如果你不小心的话,会发出疯狂的低级
write()
系统调用数,但是当我说“很久以前”我的意思是1998年左右。)
edit — note that the situation of a Java program writing a simple file in a simple way, and yet being really slow, is an inherently odd one. 编辑 - 请注意,Java程序以简单的方式编写一个简单文件,但速度非常慢的情况本质上是奇怪的。 Can you tell if the CPU is heavily loaded while the file is being written?
在写入文件时,能否判断CPU是否负载过重? It shouldn't be;
不应该; there should be almost no CPU load from such a thing.
从这样的事情应该几乎没有CPU负载。
A simple test for you 一个简单的测试
char[] chars = new char[100*1024*1024];
Arrays.fill(chars, 'A');
String text = new String(chars);
long start = System.nanoTime();
BufferedWriter bw = new BufferedWriter(new FileWriter("/tmp/a.txt"));
bw.write(text);
bw.close();
long time = System.nanoTime() - start;
System.out.println("Wrote " + chars.length*1000L/time+" MB/s.");
Prints 打印
Wrote 135 MB/s.
You could look into Java's NIO capabilities. 您可以了解Java的NIO功能。 It may support what you want to do.
它可能支持你想做的事情。
Java NIO FileChannel versus FileOutputstream performance / usefulness Java NIO FileChannel与FileOutputstream的性能/实用性
Try using memory mapped files: 尝试使用内存映射文件:
FileChannel rwChannel = new RandomAccessFile("textfile.txt", "rw").getChannel();
ByteBuffer wrBuf = rwChannel.map(FileChannel.MapMode.READ_WRITE, 0, textToSave.length());
wrBuf.put(textToSave.getBytes());
rwChannel.close();
Hi I have created two approaches to create big files, run program on windows 7, 64-bit, 8 GB RAM machine, JDK 8 and below are results. 嗨我创建了两种方法来创建大文件,在Windows 7,64位,8 GB RAM机器上运行程序,JDK 8及以下是结果。
In both the cases, file of 180 MB created that contains number in each line from 1 to 20 million (2 crore in Indian system). 在这两种情况下,创建的180 MB文件包含每行中1到2千万的数字(印度系统中为2千万卢比)。
Java program memory grows gradually till 600 MB Java程序内存逐渐增长到600 MB
First output 第一个输出
Approach = approach-1 (Using FileWriter)
Completed file writing in milli seconds = 4521 milli seconds.
Second output 第二输出
Approach = approach-2 (Using FileChannel and ByteBuffer)
Completed file writing in milli seconds = 3590 milli seconds.
One observation - I am calculating position (pos variable) in approach#2, if I comment it out then only last string will be visible due to overwritten at position, but time reduced to nearly 2000 milli seconds. 一个观察 - 我在方法#2中计算位置(pos变量),如果我将其注释掉,那么由于在位置被覆盖,只有最后一个字符串可见,但时间减少到接近2000毫秒。
Attaching code. 附加代码。
import java.io.FileWriter;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.util.concurrent.TimeUnit;
public class TestLargeFile {
public static void main(String[] args) {
writeBigFile();
}
private static void writeBigFile() {
System.out.println("--------writeBigFile-----------");
long nanoTime = System.nanoTime();
String fn = "big-file.txt";
boolean approach1 = false;
System.out.println("Approach = " + (approach1 ? "approach-1" : "approach-2"));
int numLines = 20_000_000;
try {
if (approach1) {
//Approach 1 -- for 2 crore lines takes 4.5 seconds with 180 mb file size
approach1(fn, numLines);
} else {
//Approach 2 -- for 2 crore lines takes nearly 2 to 2.5 seconds with 180 mb file size
approach2(fn, numLines);
}
} catch (IOException e) {
e.printStackTrace();
}
System.out.println("Completed file writing in milli seconds = " + TimeUnit.MILLISECONDS.convert((System.nanoTime() - nanoTime), TimeUnit.NANOSECONDS));
}
private static void approach2(String fn, int numLines) throws IOException {
StringBuilder sb = new StringBuilder();
FileChannel rwChannel = new RandomAccessFile(fn, "rw").getChannel();
ByteBuffer wrBuf;
int pos = 0;
for (int i = 1; i <= numLines; i++) {
sb.append(i).append(System.lineSeparator());
if (i % 100000 == 0) {
wrBuf = rwChannel.map(FileChannel.MapMode.READ_WRITE, pos, sb.length());
pos += sb.length();
wrBuf.put(sb.toString().getBytes());
sb = new StringBuilder();
}
}
if (sb.length() > 0) {
wrBuf = rwChannel.map(FileChannel.MapMode.READ_WRITE, pos, sb.length());
wrBuf.put(sb.toString().getBytes());
}
rwChannel.close();
}
private static void approach1(String fn, int numLines) throws IOException {
StringBuilder sb = new StringBuilder();
for (int i = 1; i <= numLines; i++) {
sb.append(i).append(System.lineSeparator());
}
FileWriter fileWriter = new FileWriter(fn);
fileWriter.write(sb.toString());
fileWriter.flush();
fileWriter.close();
}
}
This solution creates 20GB file containing string "ABCD...89\n" for 10 * 200 million times using Java NIO.此解决方案使用 Java NIO 创建包含字符串“ABCD...89\n”的 20GB 文件 10 * 2 亿次。 Write performance on MacBook Pro (14-inch from 2021, M1 Pro, SSD AP1024R) is around 5.1 GB/s.
MacBook Pro(2021 年起 14 英寸,M1 Pro,SSD AP1024R)的写入性能约为 5.1 GB/s。
Code is following:代码如下:
public static void main(String[] args) throws IOException { long number_of_lines = 1024 * 1024 * 200; int repeats = 10; byte[] buffer = "ABCD...89\n".getBytes(); FileChannel rwChannel = FileChannel.open(Path.of("textfile.txt"), StandardOpenOption.CREATE, StandardOpenOption.WRITE); // prepare buffer ByteBuffer wrBuf = ByteBuffer.allocate(buffer.length * (int) number_of_lines); for (int i = 0; i < number_of_lines; i++) wrBuf.put(buffer); long t1 = System.currentTimeMillis(); for(int i = 0; i < repeats; i++) { rwChannel.write(wrBuf); wrBuf.flip(); } while (wrBuf.hasRemaining()) { rwChannel.write(wrBuf); } long t2 = System.currentTimeMillis(); System.out.println("Time: " + (t2-t1)); System.out.println("Speed: " + ((double) number_of_lines * buffer.length*10 / (1024*1024)) / ((t2-t1) / (double) 1000) + " Mb/s"); }
In Java, the BufferWriter is very slow: Use the native methods directly, and call them as little as possible (give them as much data per call as you can). 在Java中,BufferWriter非常慢:直接使用本机方法,并尽可能少地调用它们(尽可能为每个调用提供尽可能多的数据)。
try{
FileOutputStream file=new FileOutputStream(file);
file.write(content);
file.close();
}catch(Throwable e){
D.error(e);
}//try
Also, deleting the file can take a while (maybe it is being copied to the recycle bin first). 此外,删除文件可能需要一段时间(可能首先将其复制到回收站)。 Just overwrite the file, like in the above code.
只需覆盖文件,就像上面的代码一样。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.