使用 Java 讀取 parquet 文件，但它在本地機器上工作，在 docker 容器中不工作

Question

我需要在 Java 獨立應用程序中讀取鑲木地板文件並發布到 Kafka。 我有下面的代碼來讀取由 spark scala 應用程序生成的鑲木地板文件。

public void readTest(Path path) {
        try {
            ParquetMetadata readFooter = ParquetFileReader.readFooter(conf, path, ParquetMetadataConverter.NO_FILTER);
            List<BlockMetaData> blocks = readFooter.getBlocks();
            System.out.println("Blocks size: "+blocks.size());
            Map<String, String> keyValueMetaData = readFooter.getFileMetaData().getKeyValueMetaData();
            System.out.println("K-v metadata: "+keyValueMetaData);
            System.out.println("Created by: "+readFooter.getFileMetaData().getCreatedBy());
            
            MessageType schema = readFooter.getFileMetaData().getSchema();
            System.out.println("Schema: "+schema);
            ParquetFileReader r = new ParquetFileReader(conf, path, readFooter);

            PageReadStore pages = null;
            try {
                while (null != (pages = r.readNextRowGroup())) {
                    final long rows = pages.getRowCount();
                    System.out.println("Number of rows: " + rows);

                    final MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
                    
                    final RecordReader recordReader = columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
                    for (int i = 0; i < rows; i++) {
                        final Group g = (Group) recordReader.read();
                        System.out.println("group: "+g);
                        printGroup(g);
                    }
                }
            } finally {
                r.close();
            }
        } catch (Exception e) {
            System.out.println("Error reading parquet file.");
            e.printStackTrace();
        }
    }

java 應用程序是一個 Spring Boot 非 Web 應用程序。 此代碼在本地或 intelliJ IDE 中運行時工作正常，但是當相同的文件和代碼被 dockerized 時，我收到以下錯誤。

............Processing File name............: part-00000-2b69fe41-592a-485b-85e8-8971c5842155-c000.snappy.parquet
07:45:50.770 [main] WARN  o.a.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-08-28 07:45:50 DEBUG Tracer:106 - sampler.classes = ; loaded no samplers
2021-08-28 07:45:51 DEBUG Tracer:128 - span.receiver.classes = ; loaded no span receivers
Blocks size: 1
K-v metadata: {org.apache.spark.sql.parquet.row.metadata={"type":"struct","fields":[{"name":"first_name","type":"string","nullable":true,"metadata":{}},{"name":"last_name","type":"string","nullable":true,"metadata":{}},{"name":"email","type":"string","nullable":true,"metadata":{}},{"name":"gender","type":"string","nullable":true,"metadata":{}}]}}
Created by: parquet-mr version 1.8.2 (build c6522788629e590a53eb79874b95f6c3ff11f16c)
Schema: message spark_schema {
  optional binary first_name (STRING);
  optional binary last_name (STRING);
  optional binary email (STRING);
  optional binary gender (STRING);
}

07:45:52.073 [main] INFO  o.a.hadoop.io.compress.CodecPool - Got brand-new decompressor [.snappy]
Number of rows: 1000
Error reading parquet file.
java.lang.IllegalArgumentException
    at java.nio.Buffer.limit(Buffer.java:275)
    at org.xerial.snappy.Snappy.uncompress(Snappy.java:553)
    at org.apache.parquet.hadoop.codec.SnappyDecompressor.decompress(SnappyDecompressor.java:71)
    at org.apache.parquet.hadoop.codec.NonBlockedDecompressorStream.read(NonBlockedDecompressorStream.java:51)
    at java.io.DataInputStream.readFully(DataInputStream.java:195)
    at java.io.DataInputStream.readFully(DataInputStream.java:169)
    at org.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:286)
    at org.apache.parquet.bytes.BytesInput.toByteBuffer(BytesInput.java:237)
    at org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.<init>(PlainValuesDictionary.java:91)
    at org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.<init>(PlainValuesDictionary.java:74)
    at org.apache.parquet.column.Encoding$1.initDictionary(Encoding.java:90)
    at org.apache.parquet.column.Encoding$5.initDictionary(Encoding.java:163)
    at org.apache.parquet.column.impl.ColumnReaderBase.<init>(ColumnReaderBase.java:413)
    at org.apache.parquet.column.impl.ColumnReaderImpl.<init>(ColumnReaderImpl.java:46)
    at org.apache.parquet.column.impl.ColumnReadStoreImpl.getColumnReader(ColumnReadStoreImpl.java:82)
    at org.apache.parquet.io.RecordReaderImplementation.<init>(RecordReaderImplementation.java:271)
    at org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:147)
    at org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:109)
    at org.apache.parquet.filter2.compat.FilterCompat$NoOpFilter.accept(FilterCompat.java:177)
    at org.apache.parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:109)
    at org.apache.parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:80)
    at com.csp.cdp.CPRemediationIntegrationPub.readers.NFSStorageReader.readTest(NFSStorageReader.java:131)
    at com.csp.cdp.CPRemediationIntegrationPub.readers.NFSStorageReader.read(NFSStorageReader.java:59)
    at com.csp.cdp.CPRemediationIntegrationPub.readers.NFSStorageReader.read(NFSStorageReader.java:38)
    at com.csp.cdp.CPRemediationIntegrationPub.executors.AppExecutor.execute(AppExecutor.java:29)
    at com.csp.cdp.CPRemediationIntegrationPub.CpRemediationIntegrationPubApplication.main(CpRemediationIntegrationPubApplication.java:47)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) 

我的 Dockerfile 供參考。

FROM openjdk:8-jdk-alpine
RUN mkdir /app
RUN mkdir /app/files/
COPY testDir /app/files/
COPY normal /app/files/
WORKDIR /app
COPY target/appname-0.0.1-SNAPSHOT.jar /app/app.jar
ENTRYPOINT ["java","-jar","app.jar"]

請幫助解決這個問題。

Answer 1

我將基本圖像更改為

FROM openjdk:8

它就像一個魅力。 不確定早期的基本映像有什么問題。 我認為鑲木地板無法支持特殊的 unicode 字符。

Answer 2

這里有2個問題

1. snappy-java 沒有找到原生庫，結束回退到純 java 實現
2. java 實現被破壞，這是固定的，但尚未發布

解決方案：安裝一個原生庫並使用它

我寫了一篇關於它的小帖子： https ://thomasdecaux.medium.com/use-snappy-compression-native-when-running-spark-on-alpine-5fee0ce28ed7：

1. 安裝apk add java-snappy-native
2. 使用它spark.executor.extraJavaOptions: -Dorg.xerial.snappy.use.systemlib=true -Dorg.xerial.snappy.lib.path=/usr/lib/libsnappyjava.so