Using Akka actors to download many files

Question

Background

I'm trying to use Akka actors to download many files. (Akka 2.5.1 & Java 8)

Each actor is assigned with a different URL it should download from.

A different actor is creating the downloaders actors and it should not wait for the downloaders to finish. Once they will finish, they will create another actor to handle the downloaded file.

The problem

When I run only one actor - it is able to download the file.

As I increas the number of actors it seems like none of them is able to finish its task. They are downloading portion of the files and stopping with no particular error / exception.

Actors creation code:

ActorRef downloaderActor = context().actorOf(Props.create(DownloaderActor.class)); downloaderActor.tell("URL to download", this.getSelf());

Inside the DownloaderActor class I have a download function where it seems like the problem occurs:

public void downloadFile(String fileURL, String saveDir) {
        try {
            URL url = new URL(fileURL);
            HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
            int responseCode = httpConn.getResponseCode();
            if (responseCode == HttpURLConnection.HTTP_OK) {
                InputStream inputStream = httpConn.getInputStream();
                String saveFilePath = saveDir + File.separator + fileName;
                FileOutputStream outputStream = new FileOutputStream(saveFilePath);

                int bytesRead = -1;
                byte[] buffer = new byte[4096];
                while ((bytesRead = inputStream.read(buffer)) != -1) {
                    outputStream.write(buffer, 0, bytesRead);
                }
                outputStream.close();
                inputStream.close();
                System.out.println("File downloaded: " + fileURL);
            } else {
                System.out.println("No file to download. Server replied HTTP code: " + responseCode + " . when accessing " + url);
            }
            httpConn.disconnect();
        }catch (MalformedURLException murl){
            murl.printStackTrace();
        }catch (IOException ioe){
            ioe.printStackTrace();
        }
}

And to be more specific - it seems like the problem is in the "while" loop, because if I add there logging, I can see that the loop is looping and than stops after a while.

Failed attempts

I also tried to set some http connection parameters:

httpConn.setRequestProperty("Range", "bytes=0-24"); httpConn.setConnectTimeout(10_000_000);

But it didn't seems to help.

I tried to put the download function as static function in a different Util class and it also didn't helped.

Will appriciate any help here.

Answer 1

You are using blocking I/O here to download the file, which means you need a thread for each concurrent download.

You're on the right track in the sense that the Actor model can help you model your problem without requiring one thread per concurrent download: actors are a good way to model concurrent asynchronous processes.

To actually take advantage of this, however, you still need to write the 'implementation' of the actor to be non-blocking. There are a number of non-blocking HTTP libraries available for Scala, for example you could use akka-http's future-based client API together with the 'ask' pattern.

Answer 2

I found the problem origin:

I run the code in a Junit contex. Seems like the Junit at some point cut the running threads and thus terminating the activiti of the actors.

Once I started to run the program in regular run mode, the problem (seems to be) gone.