[英]Any way to improve the performance for reading a file ,better than buffered reader
我正在通过缓冲读取器机制读取日志文件,该机制占用的总执行时间以毫秒为单位:12944,请告知我该如何提高性能并降低该时间,请注意,尼奥比缓冲读取器的性能更好。 ! 文件大小为10MB,因为它是一个日志文件。 也请告知使用nio也可以实现相同的目的.. !!
public class BufferedRedeem
{
public static void main(String[] args)
{
BufferedReader br = null;
long startTime = System.currentTimeMillis();
try
{
String sCurrentLine;
br = new BufferedReader(new FileReader("C://abc.log"));
while ((sCurrentLine = br.readLine()) != null)
{
}
long elapsedTime = System.currentTimeMillis() - startTime;
System.out.println("Total execution time taken in millis: " + elapsedTime);
}
catch (IOException e)
{
e.printStackTrace();
}
finally
{
try
{
if (br != null)
br.close();
}
catch (IOException ex)
{
ex.printStackTrace();
}
}
}
}
由于OP渴望了解如何使用NIO来完成此操作。
由于文件很小,很难看到差异,但是可以测量。
public static void main(String... args) throws IOException {
PrintWriter pw = new PrintWriter("abc.log");
for (int i = 0; i < 100 * 1000; i++) {
pw.println("0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789");
}
pw.close();
long start2 = System.nanoTime();
int count2 = 0;
BufferedReader br = new BufferedReader(new FileReader("abc.log"));
while (br.readLine() != null) count2++;
br.close();
long time2 = System.nanoTime() - start2;
System.out.printf("IO: Took %,d ms to read %,d lines%n", time2 / 1000 / 1000, count2);
long start = System.nanoTime();
FileChannel fc = new FileInputStream("abc.log").getChannel();
ByteBuffer bb = ByteBuffer.allocateDirect((int) fc.size());
fc.read(bb);
fc.close();
bb.flip();
CharBuffer cb = ByteBuffer.allocateDirect(bb.remaining() * 2).order(ByteOrder.nativeOrder()).asCharBuffer();
CharsetDecoder cd = Charset.forName("UTF-8").newDecoder();
cd.decode(bb, cb, true);
cb.flip();
StringBuilder sb = new StringBuilder();
int count = 0;
while (cb.remaining() > 0) {
char ch = cb.get();
if (isEndOfLine(cb, ch)) {
// process sb
count++;
sb.setLength(0);
} else {
sb.append(ch);
}
}
long time = System.nanoTime() - start;
System.out.printf("NIO as UTF-8: Took %,d ms to read %,d lines%n", time / 1000 / 1000, count);
long start3 = System.nanoTime();
FileChannel fc2 = new FileInputStream("abc.log").getChannel();
MappedByteBuffer bb2 = fc2.map(FileChannel.MapMode.READ_ONLY, 0, fc2.size());
bb.flip();
StringBuilder sb3 = new StringBuilder();
int count3 = 0;
while (bb2.remaining() > 0) {
char ch = (char) bb2.get();
if (isEndOfLine(bb2, ch)) {
// process sb
count3++;
sb3.setLength(0);
} else {
sb3.append(ch);
}
}
fc2.close();
long time3 = System.nanoTime() - start3;
System.out.printf("NIO as ISO-8859-1: Took %,d ms to read %,d lines%n", time3 / 1000 / 1000, count3);
}
private static boolean isEndOfLine(CharBuffer cb, char ch) {
if (ch == '\r') {
if (cb.remaining() >= 1 && cb.get() == '\n') {
return true;
}
cb.position(cb.position() - 1);
return true;
} else if (ch == '\n') {
return true;
}
return false;
}
private static boolean isEndOfLine(ByteBuffer bb, char ch) {
if (ch == '\r') {
if (bb.remaining() >= 1 && bb.get() == '\n') {
return true;
}
bb.position(bb.position() - 1);
return true;
} else if (ch == '\n') {
return true;
}
return false;
}
打印每一行的长度为102字节,因此文件约为10 MB。
IO: Took 112 ms to read 100,000 lines
NIO as UTF-8: Took 207 ms to read 100,000 lines
NIO as ISO-8859-1: Took 87 ms to read 100,000 lines
如前所述,使用NIO节省35毫秒的额外复杂性不太值得。
顺便说一句:如果您有硬盘驱动器,并且文件不在内存中,则只考虑驱动器的速度。
您有一个System.out.println(sCurrentLine);
你的循环中,这通常是非常低效的,因为它基本上包括在冲洗每次调用的输出。
您可以尝试仅将这些行放入ArrayList中,而不是先输出然后测量该时间吗? 这样会花费相似的时间吗?
您的执行时间主要归因于System.out.println(sCurrentLine);
。 我假设您不只是sysout,而是想做一些处理或过滤。
如果要检查BufferedReader的速度,请使用计数器对读取的行数进行计数,然后仅打印计数。
文件大小为10MB,因为它是一个日志文件
然后,如果您有一台不错的计算机,那么一次读取整个文件应该不是问题(需要Java 7):
public static void main(String[] args) {
try {
long start = System.nanoTime();
List<String> lines = Files.readAllLines(Paths.get("C:/temp/test.log"), Charset.
forName("UTF-8"));
System.out.println("Lines read: " + lines.size());
System.out.println("Total execution time taken in millis: "
+ ((System.nanoTime() - start) / 1000000));
} catch (IOException ex) {
ex.printStackTrace();
}
}
注意:使用该方法在我的计算机上读取6MB文件需要75毫秒。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.