简体   繁体   中英

Multi Thread (ExecutorService) Java on GCP

At the moment I'm working with a demo on GCP I read a huge File from a bucket do some process and then write on another bucket the result.

Once I read the file I send each line of the file to a Thread using ExecutorService, the thing is I've tested the process with 50, 25 and 10 Threads (Executors.newFixedThreadPool(10);) and locally work pretty well (with 10 and 25 threads), however, once I deploy the project on the GCP Kubernetes it dies at some point (no log no error the pod just die and it restarts).

Any idea why this could happen?

The config; 1 Cluster (1.14.10-gke.36) 3vCPUs, 7.5gb ram from which I have 3 nodes each one with 1 vCPU and 2.77gb RAM by extension the same to a single pod.

Part of the code;

  void someMethod (List<String> strings){
    ExecutorService executor = Executors.newFixedThreadPool(10);

    strings.forEach(string -> {
        executor.execute(()-> {
                //Some code...
                toProcess(string);
                //Some code...
        });
    });
    executor.shutdown();
    // Wait until all threads are finished
    while (!executor.isTerminated()) {}

    //Some more things to finish
  }

  public void toProcess(String string){
        Processable[] toDo = {someVisitors};
                for (Processable process : toDo) {
                    if(process.accept(address)) {
                        break;
                    }

                }
  }

This was due to lack of memory of the Pods. The process was making them run out of memory which caused them to restart, this was solved by assigning more memory to them.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM