简体   繁体   English

Java:如何使用多线程和异步有效地处理 Zipfile 读取和创建字节 []

[英]Java: How to efficiently process Zipfile reading and create byte[] using Multithreading and Async

I am currently developing a method in the Service layer implementation where the method receives a.zip file (file size could go up to 600~700MB) as a Multipart file.我目前正在服务层实现中开发一种方法,该方法接收 a.zip 文件(文件大小可以 go 高达 600~700MB)作为多部分文件。 Out of all the files zipped in that Multipart file, there are only 4-5 JSON files of interest to me which I am reading from the zip using ZipInputStream and storing them as String values for further usage.在该 Multipart 文件中压缩的所有文件中,只有 4-5 个我感兴趣的 JSON 文件,我使用 ZipInputStream 从 zip 读取这些文件并将它们存储为字符串值以供进一步使用。

Service class:服务 class:

@Async("taskExecutor")
public CompletableFuture<ResponseEntity<?>> methodname(MultipartFile file){

    ZipEntry entry = null;
    try(ZipInputStream zipFileStream = new ZipInputStream(file.getInputStream())){
        while((entry = zipFileStream.getNextEntry) != null){
            String entryName = entry.getName();
            
            if(entryName.contains("<file1name>")){
            BufferedReader br = new BufferedReader(new InputStreamReader(zipFileStream));
            String value1 = br.lines().collect(Collectors.joining("\n"));
            zipFileStream.closeEntry();
            }
            
            if(entryName.contains("<file2name>")){
            BufferedReader br = new BufferedReader(new InputStreamReader(zipFileStream));
            String value2 = br.lines().collect(Collectors.joining("\n"));
            zipFileStream.closeEntry();
            }
            
            if(entryName.contains("<file3name>")){
            BufferedReader br = new BufferedReader(new InputStreamReader(zipFileStream));
            String value3 = br.lines().collect(Collectors.joining("\n"));
            zipFileStream.closeEntry();
            }
        }
    }
    
    //String value1 & String value2 merged based on some condition to finally prepare String value1.
    //some logic to prepare a file
    
    if(fileExists){
        //create byte[] and Httpheaders with content disposition and mediatype and send CompletableFuture<ResponseEntity<?>>
    }
}

I have annotated the method @Async (as I have created an Executor bean in config class), still I have not been able to figure out how can I run different processes of this methods asynchronously or in multi-threaded way to make the processing faster.我已经注释了方法@Async(因为我在配置类中创建了一个Executor bean),但我仍然无法弄清楚如何异步或以多线程方式运行此方法的不同进程以使处理更快. The entire process still runs on single thread from that executor service pool.整个过程仍然在该执行程序服务池的单个线程上运行。

Can anyone please advise how can I introduce asynchronous or multi thread processing in my above method, so that concurrent processes like谁能告诉我如何在上述方法中引入异步或多线程处理,以便并发进程像

  • Reading the Zip file读取 Zip 文件
  • Creating the final byte[]创建最终字节[]

can be done a little bit faster to reduce the overall response time.可以做得更快一点,以减少整体响应时间。

store MultipartFile to temp file and try ZipFile (which supports streams ootb)将 MultipartFile 存储到临时文件并尝试ZipFile (支持流 ootb)

final ZipFile zipFile = new ZipFile("dummy.zip");
zipFile
  .stream()
  .parallel()
  .filter(entry -> entry.getName().matches("regexFile1")
    || entry.getName().matches("regexFile2")
    || entry.getName().matches("regexFile3")
  )
  .map(entry -> {
    try {
      return new EntryDto(entry.getName(), new String(zipFile.getInputStream(entry).readAllBytes(), StandardCharsets.UTF_8));
    } catch (IOException e) {
      throw new RuntimeException(e);
    }
  })
  .map(dto -> {
    // custom logic
    return ...;
  })
  .collect(Collectors.toList());

dto class dto class

class EntryDto {
    private String name;
    private String json;

    public EntryDto(String name, String json) {
        this.name = name;
        this.json = json;
    }

    public String getName() {
        return name;
    }

    public String getJson() {
        return json;
    }
}

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM