Unix排序命令需要更长的时间，具体取决于它的执行位置？！ （从IDE运行程序中的ProcessBuilder最快，终端最慢）

Question

I have a java program that uses ProcessBuilder to call the unix sort command. When I run this code within my IDE (intelliJ) it only takes about a second to sort 500,000 lines. When I package it into an executable jar, and run that from the terminal it takes about 10 seconds. When I run the sort command myself from the terminal, it takes 20 seconds!

Why the vast difference in performance and any way I can get the jar to execute with the same performance? Environment is OSX 10.6.8 and java 1.6.0_26. The bottom of the sort man page says "sort 5.93 November 2004"

The command it is executing is:

sort -t'    ' -k5,5f -k4,4f -k1,1n /path/to/imput/file -o /path/to/output/file

Note that when I run sort from the terminal I need to manually escape the tab delimiter and use the argument -t$'\\t' instead of the actual tab (which I can pass to ProcessBuilder).

Looking as ps everything seems the same except when run from IDE the sort command has a TTY of ?? instead of ttys000--but from this question I don't think that should make a difference. Perhaps BASH is slowing me down? I am running out of ideas and want to close this 20x performance gap!

Answer 1

I'm going to venture two guesses:

• perhaps you are invoking different versions of sort (do a which sort and use the full absolute path to recompare?)

• perhaps you are using more complicated locale settings (leading to more complicated character set handling etc.)? Try

    export LANG=C sort -t' ' -k5,5f -k4,4f -k1,1n /input/file -o /output/file 

to compare

Answer 2

Have a look at this project: http://code.google.com/p/externalsortinginjava/

Avoid the need of calling external sort entirely.