Java：使用 apache 光束管道讀取存儲在存儲桶中的 excel 文件

Question

我知道以下方式，但我需要在 Apache 光束管道中請提供一個示例：

try (ReadableByteChannel chan = FileSystems.open(FileSystems.matchNewResource(
            "gs://bucketname/filename.xlsx", false ))) {
      InputStream inputStream = Channels.newInputStream(chan);

Answer 1

我已經實現了從本地文件系統讀取 .xlsx 文件，但同樣適用於您的 GCS 存儲桶路徑。 我在不同的管道中嘗試過相同的方法，效果很好。

下面代碼中的enrichedCollection可以被視為逐行讀取的.csv 文件。 我使用分號作為分隔符來分隔值。

package com.fooBar;

import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.io.FileIO;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
import org.apache.poi.ss.usermodel.Cell;
import org.apache.poi.ss.usermodel.Row;
import org.apache.poi.xssf.usermodel.XSSFSheet;
import org.apache.poi.xssf.usermodel.XSSFWorkbook;

import java.io.IOException;
import java.io.InputStream;
import java.nio.channels.Channels;
import java.util.Iterator;

public class SampleExcelInput {

    public static void main(String[] args) throws IOException {
        
        Pipeline pipeline = Pipeline.create();
        
        PCollection<FileIO.ReadableFile> inputCollection  = pipeline.apply(FileIO.match()
//                .filepattern("gs://bucket/file.xlsx"))
                        .filepattern("C:\\Workspace\\ApacheBeam\\src\\main\\resources\\Inputfiles\\SampleExcel.xlsx"))
                .apply(FileIO.readMatches());
        
        PCollection<String> enrichedCollection = inputCollection.apply(ParDo.of(new ReadXlsxDoFn()));
        //TODO: do further processing treating the lines of enrichedCollection pcollection as if they were read from csv
        pipeline.run().waitUntilFinish();
    }

    static class ReadXlsxDoFn extends DoFn<FileIO.ReadableFile, String>{
        final static String  DELIMITER  = ";";
        @ProcessElement
        public void process(ProcessContext c) throws IOException {
            FileIO.ReadableFile  fileName = c.element();
            System.out.println("FileName being read is :" + fileName);
            assert fileName != null;
            InputStream stream = Channels.newInputStream(fileName.openSeekable());
            XSSFWorkbook wb = new XSSFWorkbook(stream);
            XSSFSheet sheet = wb.getSheetAt(0);     //creating a Sheet object to retrieve object
            //iterating over Excel file
            for (Row row : sheet) {
                Iterator<Cell> cellIterator = row.cellIterator();   //iterating over each column
                StringBuilder sb  = new StringBuilder();
                while (cellIterator.hasNext()) {
                    Cell cell = cellIterator.next();
                    if(cell.getCellType() ==  Cell.CELL_TYPE_NUMERIC){
                        sb.append(cell.getNumericCellValue()).append(DELIMITER);
                    }
                    else{
                        sb.append(cell.getStringCellValue()).append(DELIMITER);
                    }
                }
                System.out.println(sb.substring(0, sb.length()-1));
            c.output(sb.substring(0, sb.length()-1));//removing the delimiter present @End of String 

            }
        }
    }
}

對於依賴項，我必須手動添加一些 jar 以使其工作，您可以從此處獲取該參考

除了上面的罐子，我還有以下作為我的 Maven 依賴項。

         <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-sdks-java-core</artifactId>
            <version>2.37.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-runners-direct-java</artifactId>
            <version>2.37.0</version>
        </dependency>
        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.11.0</version>
        </dependency>

示例 .xlsx 文件的鏈接： 這里

DoFn 的控制台輸出

FileName being read is :ReadableFile{metadata=Metadata{resourceId=C:\Users\USER\Desktop\Java 
   
   


    
    FileName being read is :ReadableFile{metadata=Metadata{resourceId=C:\Users\USER\Desktop\Java Masterclass\ApacheBeam\src\main\resources\Inputfiles\SampleExcel.xlsx, sizeBytes=7360, isReadSeekEfficient=true, checksum=null, lastModifiedMillis=0}, compression=UNCOMPRESSED}
0.0;First Name;Last Name;Gender;Country;Age;Date;Id
1.0;Dulce;Abril;Female;United States;32.0;15/10/2017;1562.0
2.0;Mara;Hashimoto;Female;Great Britain;25.0;16/08/2016;1582.0
3.0;Philip;Gent;Male;France;36.0;21/05/2015;2587.0
4.0;Kathleen;Hanner;Female;United States;25.0;15/10/2017;3549.0
5.0;Nereida;Magwood;Female;United States;58.0;16/08/2016;2468.0
6.0;Gaston;Brumm;Male;United States;24.0;21/05/2015;2554.0
7.0;Etta;Hurn;Female;Great Britain;56.0;15/10/2017;3598.0
8.0;Earlean;Melgar;Female;United States;27.0;16/08/2016;2456.0  
.
.
.
50.0;Rasheeda;Alkire;Female;United States;29.0;16/08/2016;6125.0

Process finished with exit code 0

注意：由於文件是在 Simple DoFn 中逐行解析的，這意味着每個文件只有一個線程。 如果你只有一個非常大的文件，比如~5GB，你會發現性能顯着下降。 一種解決方法是縮小輸入文件的大小並使用通配符文件模式。