多个线程从sftp服务器下载相同文件

Question

I have a system that, when files of a certain type are found, I download, encode, and upload them in a separate thread.

while(true) {
    for(SftpClient c : clients) {
        try {
            filenames = c.list("*.wav", "_rdy_");
        } catch (SftpException e) {
            e.printStackTrace();
        }
        if(filenames.size() > 0) {
            //AudioThread run() method handles the download, encode, and upload
            AudioThread at = new AudioThread(filenames);
            at.setNode(c.getNode());
            Thread t = new Thread(at);
            t.start();
        }
    }
    try {
        Thread.sleep(3000);
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}

The run method from AudioThread

public void run() {
    System.out.println("Running...");
    this.buildAsteriskMapping();
    this.connectToSFTP();
    ac = new AudioConvert();
    this.connectToS3();

    String downloadDir = "_rough/" + getNode() + "/" + Time.getYYYYMMDDDate() + "/";
    String encodeDir = "_completed" + getNode() + "/" + Time.getYYYYMMDDDate() + "/";
    String uploadDir = getNode() + "/" + Time.getYYYYMMDDDate() + "/";

    System.out.println("Downloading...");
    try {
        sftp.get(filenames, downloadDir);
    } catch (SftpException e) {
        //download failed
        System.out.println("DL Failed...");
        e.printStackTrace();
    }

    System.out.println("Encoding...");
    try {
        ac.encodeWavToMP3(filenames, downloadDir, encodeDir);
    } catch (IllegalArgumentException | EncoderException e) {
        System.out.println("En Failed...");
        e.printStackTrace();
    }

    System.out.println("Uploading...");
    try {
        s3.upload(filenames, encodeDir, uploadDir);
    } catch (AmazonClientException e) {
        System.out.println("Up Failed...");
        e.printStackTrace();
    }

}

The download method:

public void get(ArrayList<String> src, String dest) throws SftpException {
    for(String file : src) {
        System.out.println(dest + file);
        channel.get(file, dest + file);
    }
}

The encode method:

public void encodeWavToMP3(ArrayList<String> filenames, String downloadDir, String encodeDir) throws IllegalArgumentException, EncoderException {
    for(String f : filenames) {
        File wav = new File(downloadDir + f);
        File mp3 = new File(encodeDir + wav.getName().replace(".wav", ".mp3"));
        encoder.encode(wav, mp3, attrs);
    }
}

The upload method:

public void upload(ArrayList<String> filenames, String encodeDir, String uploadDir)  throws AmazonClientException, AmazonServiceException {
    for(String f : filenames) {
        s3.putObject(new PutObjectRequest(bucketName, uploadDir, new File(encodeDir + f)));
    }
}

The issue is I keep downloading the same files (or about the same files) for every thread. I want to add a variable for each client that holds the files that are being downloaded but I don't know how to remove the lists/filenames from this variable. What would be a solution? My boss would also like to only allow x amount of threads to run.

Answer 1

It's kind of hard to see the problem, as the code that actually does the download is missing :P

However, I would use some kind of ExecutorService instead.

Basically, I would add each download request to the service (wrapped in a "DownloadTask" with a reference to the file to be downloaded and any other relevant information it might need to get the file) and let the service take care of the rest.

The download tasks could be coded to take into account existing files as you see fit.

Depending on your requirements, this could be a single thread or multi-threaded service. It could also allow you to place upload quests in it as well.

Check out the Executors trail for more info

The general idea is to use a kind of producer/consumer pattern. You would have (at least) a thread that would look up all the files to be downloaded and for each file, you would add it to the executor service. After the file has been downloaded, I would queue and upload request into the same service.

This way, you avoid all the mess with synchronization and thread management :D

You could use the same idea with the scan tasks, for each client, you could a task to a separate service

Answer 2

There is a problem in your code where you instantiate AudioThread in a while loop.

Note that after you create a thread and do a t.start(), all downloading, encoding and uploading happens asynchronously. Therefore, after you start the thread the loop continuous to do another call to c.list(...) while the first thread you created is still processing the first set of files. Most probably the same set of files is returned in the succeeding c.list() calls since you specified a file pattern in the call and there is no code which marks which files are currently being processed.

My suggestion:

• Use Executors.newFixedThreadPool(int nThreads) as mentioned in previous post. And specify the number of threads to the number of processors in your machine. Do this before your while loop.
• For each filename you retrieved from ftp s.list(), create a Callable class and call ExecutorService.invokeAll(Collection<Callable<T>> tasks). The code in the Callable you will create is your AudioThread code. Modify AudioThread code to only process one file at at time (if possible), this way you are doing downloads,uploads, encoding in parallel for each file.
• Add code which marks which files were already processed. I would suggest adding a code which renames the files you have processed to a different name to avoid getting returned in the next c.list() call.
• Call ExecutorService.shutdown(...) after your while loop block