I want to download all the files from Azure blob storage, create a zip file out of these files and upload the zip file back to the blob storage. As the file size can be very large, I dont want to max out the memory. Also this operation needs to be very FAST.
JAVA SDK - azure-storage-blob 12.8.0
EDIT : Code written so far. Not sure how to proceed further with uploading pipedinputstream data parallely.
String zipFileName = formFileName(exportRequest, requestId);
final PipedOutputStream pipedOutputStream = new PipedOutputStream();
final PipedInputStream pipedInputStream = new PipedInputStream(pipedOutputStream);
AzureObjectStoreService objectStoreService =managedObjectStoreUtils.getObjectStoreService();
if (filesToZip.size() > 0) {
System.out.println("Files to zip "+ filesToZip.size());
CompletableFuture<Boolean> zipCreationFuture = CompletableFuture.runAsync(() -> {
LoggerHelper.logInfo(logger, "Inside createZIP file async function");
ZipOutputStream zipOutputStream = new ZipOutputStream(pipedOutputStream);
try {
for (String fileName : filesToZip) {
try {
BlobClient blobClient = objectStoreService.getBlobContainerClient().getBlobClient(fileName);
LoggerHelper.logInfo(logger, "Adding zipEntry for file : " + fileName);
final ZipEntry zipEntry = new ZipEntry(fileName);
zipOutputStream.putNextEntry(zipEntry);
byte[] buffer;
ByteArrayOutputStream output = new ByteArrayOutputStream();
buffer= output.toByteArray();
blobClient.getBlockBlobClient().download(output);
int len;
while ((len = buffer.length) > 0) {
zipOutputStream.write(buffer, 0, len);
}
zipOutputStream.closeEntry();
} catch (SdkClientException e) {
LoggerHelper.logExceptionWithMessage(logger, this.getClass().getName(), (Exception) e);
LoggerHelper.logError(logger, "Failed while getting s3 object");
}
}
zipOutputStream.finish();
} catch (IOException ex) {
LoggerHelper.logExceptionWithMessage(logger, this.getClass().getName(), (Exception) ex);
LoggerHelper.logError(logger, "Creating zip file failed");
} finally {
try {
zipOutputStream.close();
} catch (IOException e) {
LoggerHelper.logExceptionWithMessage(logger, this.getClass().getName(), (Exception) e);
LoggerHelper.logError(logger, "Failed to close the zip output stream");
}
}
LoggerHelper.logInfo(logger, "Completed createZIP file async function");
// return true;
}).handle((o, exception) -> {
LoggerHelper.logExceptionWithMessage(logger, this.getClass().getName(), (Exception) exception);
LoggerHelper.logError(logger, "Creating zip file failed");
return null;
});
Was able to do it this way. Please let me know if anyone has a better approach.
CompletableFuture.runAsync(() -> {
BlobClient blobClient = objectStoreService.getBlobContainerClient().getBlobClient(zipFileName);
BlobOutputStream blobOutputStream = blobClient.getBlockBlobClient().getBlobOutputStream();
try {
int nextData= pipedInputStream.read();
while (nextData!=-1) {
blobOutputStream.write(nextData);
nextData = pipedInputStream.read();
}blobOutputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.