Perl-使用线程在短时间内创建带有一些数据的文本文件

Question

Whats the best way to generate 1000K text files? (with Perl and Windows 7) I want to generate those text files in as possible in less time (possibly withing 5 minutes)? Right now I am using Perl threading with 50 threads. Still it is taking longer time.

What will be best solution? Do I need to increase thread count? Is there any other way to write 1000K files in under five minutes? Here is my code

    $start = 0;
    $end = 10000;
    my $start_run = time();
    my @thr = "";

    for($t=0; $t < 50; $t++) {
        $thr[$t] = threads->create(\&files_write, $start, $end);
        #start again from 10000 to 20000 loop
        .........
    }

    for($t=0; $t < 50; $t++) {
        $thr[$t]->join();
    }

    my $end_run = time();
    my $run_time = $end_run - $start_run;
    print "Job took $run_time seconds\n";

I don't want return result of those threads. I used detach() also but it didn't worked me. For generating 500k files (with only size of 20kb) it took 1564 seconds (26min). Can I able to achieve within 5min?

Edited: The files_write will only take values from array predefined structure and write into file. thats it.

Any other solution?

Answer 1

The time needed depends on lots of factors, but heavy threading is probably not the solution:

• creating files in the same directory at the same time needs probably locking in the OS, so it's better done not too much in parallel
• the layout how the data gets written on disk depend on the amount of data and on how many writes you do in parallel. A bad layout can impact the performance a lot, especially on HDD. But even a SDD cannot do lots of parallel writes. This all depends a lot on the disk you use, eg it is a desktop system which is optimized for sequential writes or is it a server system which can do more parallel writes as required by databases.
• ... lots of other factors, often depending on the system

I would suggest that you use a thread pool with a fixed size of threads to benchmark, what the optimal number of threads is for your specific hardware. Eg start with a single thread and slowly increase the number. My guess is, that the optimal number might be between factor 0.5 and 4 of the number of processor cores you have, but like I said, it heavily depends on your real hardware.

Answer 2

The slow performance is probably due to Windows having to lock the filesystem down while creating files.

If it is only for testing - and not critical data - a RAMdisk may be ideal. Try Googling DataRam RAMdisk.