Java - Error sending multiple files concurrently using threadpool

Question

I'm trying to send some files to a rest api concurrently. Tried to implement this answer

public class UploadThread implements Runnable {

    private File file;

    public UploadThread(File file) {
        this.file = file;
    }

    @Override
    public void run() {
        try {
            String url = // api url

            URL server = new URL(url);
            HttpURLConnection connection = (HttpURLConnection) server.openConnection();

            // setting request properties here

            connection.connect();

            OutputStream out = connection.getOutputStream();
            FileInputStream in = new FileInputStream(file);

            try {
              // write to outputstream

            } catch (Exception ex) {
                ex.printStackTrace();
            } finally {
                out.close();
                in.close();

                if (connection.getResponseCode() != HttpURLConnection.HTTP_OK) {
                    // error handling here
                }

                connection.disconnect();            
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

public class UploadTest {

    public static void main(String[] args) throws Exception {

        List<File> files = Files.walk(Paths.get("/home/random"))
                                .filter(Files::isRegularFile)
                                .map(Path::toFile)
                                .collect(Collectors.toList());

        ExecutorService executorService = Executors.newFixedThreadPool(5);

        for(File file : files) {
            Runnable worker = new UploadThread(file);

            executorService.execute(worker);
        }

        executorService.shutdown();

        while(!executorService.isTerminated()) {
            //
        }

        System.out.println("upload finished....");

    }
}

I have 10 files in the random directory.

When I execute this, the first file is sent exactly same number of times as the number of threads I have allocated in Executors.newFixedThreadPool() method.

If I allocate 5 threads, the first file (different in each execution) is sent 5 times, where as rest of the files are sent exactly once! For 10 threads, it is sent 10 times!

How is only one file is sent multiple times, but rest are not? What am I doing wrong in the code?

Edit : The api I'm sending the files to is using Jersey on Tomcat 8 . Here's how the files are handled.

On resource

@Path("file")
public class MyResource {

    private StorageManager storageManager;

    public MyResource() {
        this.storageManager = new StorageManager();
    }

    @GET
    @Path("upload")
    @Consumes(MediaType.APPLICATION_OCTET_STREAM)
    public Response upload(InputStream inputStream) {
        return this.storageManager.upload(inputStream);
    }
}

and the StorageManager class is

public class StorageManager {

    public Response upload(InputStream inputStream) {

        // handling file upload here
        // this method is called 5 or 10 times for one file

        // return some response
    }
}

For a single file, this works pretty fine. I have uploaded hundreds of files using REST clients and this works everytime. But for concurrent requests, this upload() method is called a number of times for any one file (first one).

Answer 1

Tried your code, seems to work fine. By the way, there is no need to map the Path to a File because with Files.copy you can easily copy the contents of a Path to an OutputStream .

public class UploadThread implements Runnable {

   private Path file;

   public UploadThread(Path file) {
    this.file = Objects.requireNonNull(file);
   }

 @Override
  public void run() {
    try {
        String url = // api url

        URL server = new URL(url);
        HttpURLConnection connection = (HttpURLConnection) server.openConnection();

        // setting request properties here

        connection.connect();

        try (OutputStream out = connection.getOutputStream()) {
            Files.copy(file, out);
          } finally {
            if (connection.getResponseCode() != HttpURLConnection.HTTP_OK) {
                // error handling here
            }

            connection.disconnect(); 
          }           
    } catch (Exception e) {
        e.printStackTrace();
    }
  }
}

You can also ditch the for and the 'while' loops:

public static void main(String[] args) throws Exception {
    ExecutorService executorService = Executors.newFixedThreadPool(5);
    Files.walk(Paths.get("/home/random"))
                            .filter(Files::isRegularFile)
                            .map(UploadThread::new)
                            .forEach(executorService::execute);

    executorService.shutdown();
    System.out.println("upload finished....");
}