[英]Perl Multithread Program
我是Perl的新手。 我想用thread編寫一個Perl腳本,我只有幾個文件說20個文件,並且想用4個批處理中的5個線程來處理這些文件。 我正在打印線程號。 完成一個批次后,下一個批次的線程號必須以1開頭。 但是與其創建20個線程相反,請幫助。 我的代碼如下:
#!/usr/bin/perl -w
use strict;
use warnings;
use threads;
use threads::shared;
my $INPUT_DIR="/home/Documents/myscript/IMPORTLDIF/";
opendir(DIR, $INPUT_DIR) ;
my @files = grep { /^InputFile/ } readdir DIR;
my $count = @files;
#print "Total Files: $count \n";
my @threads;
my $noofthread = 5;
my $nooffiles = $count;
my $noofbatch = $nooffiles / $noofthread;
#print "No of batch: $noofbatch \n";
my $fileIndex = 0;
my $batch = 1;
while ($fileIndex < $nooffiles) {
print "Batch: $batch \n";
for (my $i=0; $i < $noofthread && $fileIndex < $nooffiles ; $i++) {
my $t = threads->new(\&doOperation, $files[$fileIndex], $i)->join;
push(@threads, $t);
$fileIndex++;
print "FileIndex: $fileIndex \n";
}
$batch++;
}
sub doOperation () {
my $ithread = threads->tid() ;
print "Thread Index : [id=$ithread]\n" ;
foreach my $item (@_){
my $filename = $item;
print "Filename name: $filename \n";
}
使用線程隊列編輯程序:
#!/usr/bin/perl -w
# This is compiled with threading support
use strict;
use warnings;
use threads;
use Thread::Queue;
my $q = Thread::Queue->new(); # A new empty queue
# Worker thread
my $INPUT_DIR="/home/Documents/myscript/IMPORTLDIF/";
opendir(DIR, $INPUT_DIR) or die "Cannot opendir: $!";
my @thrs = threads->create(\&doOperation ) for 1..5;#for 5 threads
#my @files = `ls -1 /home/Documents/myscript/IMPORTLDIF/`;
my @files = grep { /^Input/ } readdir DIR or die "File not present present. \n";
chomp(@files);
#add files to queue
foreach my $f (@files){
# Send work to the thread
$q->enqueue($f);
print "Pending items: " + $q->pending()."\n";
}
$q->enqueue('_DONE_') for @thrs;
$_->join() for @thrs;
sub doOperation () {
my $ithread = threads->tid() ;
while (my $filename = $q->dequeue()) {
# Do work on $item
return 1 if $filename eq '_DONE_';
print "[id=$ithread]\t$filename\n";
}
return 1;
}
您正在生成一個線程,然后等待它完成,然后生成下一個線程,每個線程處理一個文件。 這就是為什么看到的線程與文件一樣多的原因。
my $t = threads->new(\&doOperation, $files[$fileIndex], $i)->join;
^^^^--- This will block
而是嘗試這樣的事情:
....
# split the workload into N batches
#
while (my @batch = splice(@files, 0, $batch_size)) {
push @threads, threads->new(\&doOperation, @batch);
}
# now wait for all workers to finish
#
for my $thr (@threads) {
$thr->join;
}
順便說一句, Thread :: Queue和Thread-Pool可能意味着您想要做的工作有更好的設計。
您可以使用Paralel:Queue並創建4個線程,然后將它們可以使用的項目傳遞給他們。
use strict;
use warnings;
use threads;
use Thread::Queue;
my $q = Thread::Queue->new(); # A new empty queue
# Worker thread
my @thrs;
push @thrs, threads->create(\&doOperation ) for 1..5;#for 5 threads
my @files = `ls -1 /tmp/`;chomp(@files);
#add files to queue
foreach my $f (@files){
# Send work to the thread
$q->enqueue($f);
print "Pending items: "$q->pending()."\n";
}
$q->enqueue('_DONE_') for @thrs;
$_->join() for threads->list();
sub doOperation () {
my $ithread = threads->tid() ;
while (my $filename = $q->dequeue()) {
# Do work on $item
return 1 if $filename eq '_DONE_';
print "[id=$ithread]\t$filename\n";
}
return 1;
}
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.