從主題讀取時，Kafka LZ4 解壓導致 lz4-java 中的 IllegalAccessError

Question

在我目前正在從事的項目中，我們有四個基於 Spring 的應用程序，它們正在編寫和讀取各種 Kafka 主題。 由於要發送大量數據，我們決定啟用 LZ4 壓縮。 這對於其中三個應用程序來說非常有效，但在第四個應用程序中卻引起了巨大的問題。

流程如下：App A 接收 HTTP 請求，將 payload 轉換為 Kafka 消息。 應用程序 B 從應用程序 A 讀取消息，對其進行一些處理，然后將它們進一步發送到兩個不同的主題到應用程序 C 和 D。我們通過將壓縮類型配置設置為 lz4 對 A 和 B 中的生產者啟用了壓縮。 然后我們進行了測試，它對於從 A 到 B 以及從 B 到 D 的消息都可以正常工作。但是，當應用程序 C 收到壓縮消息時，以下錯誤開始被垃圾郵件：

java.lang.IllegalAccessError: failed to access class net.jpountz.lz4.LZ4SafeUtils (class loader System@7452) from class net.jpountz.lz4.LZ4JavaSafeSafeDecompressor (class loader org.springframework.boot.loader.LaunchedURLClassLoader@11104)

我對類加載器不是很熟悉，但在我看來，雖然它們屬於同一個包，但它們似乎是由不同的類加載器加載的，導致訪問錯誤？

使用的spring-kafka的版本是2.7.3，lz4-java的版本是1.7.1，作為spring-kafka的依賴導入。

我不明白為什么這個問題只發生在其中一個應用程序上而不是其他應用程序上，因為所有配置都是相同的（通過所有四個應用程序用來創建所需 bean 的公共庫共享和加載）。 最初，在這個提出這個問題的應用程序中，我們導入了另一個使用修改版本的 lz4-java 的庫，所以我們認為這就是問題的來源。 但是，即使完全刪除它，也會收到上面發布的相同錯誤。

有沒有人遇到過類似的問題，或者可以指導我下一步該怎么做才能解決這個問題？

謝謝你。

編輯：應用程序正在雲中運行，這是在應用程序日志中看到的堆棧跟蹤。 接下來的塊會重復發送垃圾郵件，直到它被關閉或壓縮消息從正在收聽的主題中刪除。

[container-1-C-1] essageListenerContainer$ListenerConsumer : Consumer exception
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT org.springframework.kafka.KafkaException: Seek to current after exception; nested exception is org.apache.kafka.common.KafkaException: Received exception when fetching the next record from ETD-NewUserContextSystemData.ibsoetdcfsubacc4.eb419dde-795b-48a2-977a-8117ac15cb4e-2. If needed, please seek past the record to continue consumption.
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.springframework.kafka.listener.SeekToCurrentBatchErrorHandler.handle(SeekToCurrentBatchErrorHandler.java:79) ~[spring-kafka-2.7.3.jar:2.7.3]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.springframework.kafka.listener.RecoveringBatchErrorHandler.handle(RecoveringBatchErrorHandler.java:124) ~[spring-kafka-2.7.3.jar:2.7.3]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.handleConsumerException(KafkaMessageListenerContainer.java:1603) [spring-kafka-2.7.3.jar:2.7.3]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1210) [spring-kafka-2.7.3.jar:2.7.3]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_301]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_301]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at java.lang.Thread.run(Thread.java:836) [?:1.8.0_301]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT Caused by: org.apache.kafka.common.KafkaException: Received exception when fetching the next record from ETD-NewUserContextSystemData.ibsoetdcfsubacc4.eb419dde-795b-48a2-977a-8117ac15cb4e-2. If needed, please seek past the record to continue consumption.
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1611) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1700(Fetcher.java:1432) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:684) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:635) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1303) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1237) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doPoll(KafkaMessageListenerContainer.java:1410) ~[spring-kafka-2.7.3.jar:2.7.3]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1249) ~[spring-kafka-2.7.3.jar:2.7.3]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1161) ~[spring-kafka-2.7.3.jar:2.7.3]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     ... 3 more
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT Caused by: org.apache.kafka.common.KafkaException: java.lang.IllegalAccessError: failed to access class net.jpountz.lz4.LZ4SafeUtils (class loader System@7452) from class net.jpountz.lz4.LZ4JavaSafeSafeDecompressor (class loader org.springframework.boot.loader.LaunchedURLClassLoader@11104)
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.common.record.CompressionType$4.wrapForInput(CompressionType.java:113) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:261) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.common.record.DefaultRecordBatch.streamingIterator(DefaultRecordBatch.java:346) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.nextFetchedRecord(Fetcher.java:1554) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1591) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1700(Fetcher.java:1432) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:684) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:635) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1303) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1237) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doPoll(KafkaMessageListenerContainer.java:1410) ~[spring-kafka-2.7.3.jar:2.7.3]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1249) ~[spring-kafka-2.7.3.jar:2.7.3]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1161) ~[spring-kafka-2.7.3.jar:2.7.3]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     ... 3 more
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT Caused by: java.lang.IllegalAccessError: failed to access class net.jpountz.lz4.LZ4SafeUtils (class loader System@7452) from class net.jpountz.lz4.LZ4JavaSafeSafeDecompressor (class loader org.springframework.boot.loader.LaunchedURLClassLoader@11104)
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at net.jpountz.lz4.LZ4JavaSafeSafeDecompressor.decompress(LZ4JavaSafeSafeDecompressor.java:71) ~[lz4-java-1.7.1.jar:2.9.16-38a800d98fb18a51ecde14f570bbf28d81b66bbc]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at net.jpountz.lz4.LZ4SafeDecompressor.decompress(LZ4SafeDecompressor.java:74) ~[lz4-java-1.7.1.jar:2.9.16-38a800d98fb18a51ecde14f570bbf28d81b66bbc]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at net.jpountz.lz4.LZ4Factory.<init>(LZ4Factory.java:214) ~[lz4-java-1.7.1.jar:2.9.16-38a800d98fb18a51ecde14f570bbf28d81b66bbc]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at net.jpountz.lz4.LZ4Factory.instance(LZ4Factory.java:51) ~[lz4-java-1.7.1.jar:2.9.16-38a800d98fb18a51ecde14f570bbf28d81b66bbc]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at net.jpountz.lz4.LZ4Factory.safeInstance(LZ4Factory.java:105) ~[lz4-java-1.7.1.jar:2.9.16-38a800d98fb18a51ecde14f570bbf28d81b66bbc]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at net.jpountz.lz4.LZ4Factory.fastestJavaInstance(LZ4Factory.java:141) ~[lz4-java-1.7.1.jar:2.9.16-38a800d98fb18a51ecde14f570bbf28d81b66bbc]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at net.jpountz.lz4.LZ4Factory.fastestInstance(LZ4Factory.java:169) ~[lz4-java-1.7.1.jar:2.9.16-38a800d98fb18a51ecde14f570bbf28d81b66bbc]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.common.record.KafkaLZ4BlockInputStream.<clinit>(KafkaLZ4BlockInputStream.java:50) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.common.record.CompressionType$4.wrapForInput(CompressionType.java:110) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:261) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.common.record.DefaultRecordBatch.streamingIterator(DefaultRecordBatch.java:346) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.nextFetchedRecord(Fetcher.java:1554) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1591) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1700(Fetcher.java:1432) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:684) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:635) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1303) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1237) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) ~[kafka-clients-2.6.2.jar:?]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doPoll(KafkaMessageListenerContainer.java:1410) ~[spring-kafka-2.7.3.jar:2.7.3]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1249) ~[spring-kafka-2.7.3.jar:2.7.3]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1161) ~[spring-kafka-2.7.3.jar:2.7.3]
   2021-09-24T10:42:29.30+0300 [APP/PROC/WEB/1] OUT     ... 3 more

Answer 1

我和你一樣面臨同樣的問題。 我在 Linux 系統上運行我的程序時遇到了這個問題。 問題的根本原因可能是：您的 Linux 操作系統內核不支持從 .so 文件調用方法（不確定您使用的是 Linux 還是 IBM AIX）。 當 lz4 jar 文件加載到基於 OS 內核版本的類路徑時，.so 文件會自動加載。

您可以運行命令行“uname -a”或“uname -mrs”來檢查操作系統內核版本。

如果您打開 lz4 jar 文件，您將在“util”包下看到每個操作系統的 .so 文件。 我的想法是您的（和我的）操作系統內核未添加到庫的支持列表中。 我們可能需要檢查圖書館的 Jira 問題列表，看看他們是否確認這是一個問題並且他們會修復它。