com.simba.spark.jdbc42 驅動程序的 resultset.next() 問題

Question

我在我的 Unix 服務器中使用 JDK 1.8，並使用 spark.jdbc42 連接到 Azure Databricks。

它在resultset.next()的 Unix 服務器中給出以下錯誤消息：

 Exception in thread "main" java.lang.ExceptionInInitializerError
         at com.simba.spark.jdbc42.internal.apache.arrow.memory.BaseAllocator$Config.getAllocationManagerFactory(BaseAllocator.java:776)
         at com.simba.spark.jdbc42.internal.apache.arrow.memory.ImmutableConfig.access$801(ImmutableConfig.java:24)
         at com.simba.spark.jdbc42.internal.apache.arrow.memory.ImmutableConfig$InitShim.getAllocationManagerFactory(ImmutableConfig.java:83)
         at com.simba.spark.jdbc42.internal.apache.arrow.memory.ImmutableConfig.<init>(ImmutableConfig.java:47)
         at com.simba.spark.jdbc42.internal.apache.arrow.memory.ImmutableConfig.<init>(ImmutableConfig.java:24)
         at com.simba.spark.jdbc42.internal.apache.arrow.memory.ImmutableConfig$Builder.build(ImmutableConfig.java:485)
         at com.simba.spark.jdbc42.internal.apache.arrow.memory.BaseAllocator.<clinit>(BaseAllocator.java:51)
         at com.simba.spark.spark.arrow.ArrowBuffer.<init>(Unknown Source)
         at com.simba.spark.spark.jdbc.SparkJDBC42Driver.createArrowBuffer(Unknown
 Source)
         at com.simba.spark.hivecommon.api.HS2Client.makeRowsetBuffer(Unknown
 Source)
         at com.simba.spark.hivecommon.dataengine.BackgroundFetcher.<init>(Unknown
 Source)
         at com.simba.spark.hivecommon.dataengine.HiveJDBCResultSet.moveToNextRow(Unknown
 Source)
         at com.simba.spark.jdbc.common.BaseForwardResultSet.next(Unknown Source)
 Caused by: java.lang.IllegalStateException: Arrow only runs on LittleEndian systems.
         at com.simba.spark.jdbc42.internal.io.netty.buffer.UnsafeDirectLittleEndian.<init>(UnsafeDirectLittleEndian.java:64)
         at com.simba.spark.jdbc42.internal.io.netty.buffer.UnsafeDirectLittleEndian.<init>(UnsafeDirectLittleEndian.java:50)
         at com.simba.spark.jdbc42.internal.io.netty.buffer.PooledByteBufAllocatorL.<init>(PooledByteBufAllocatorL.java:49)
         at com.simba.spark.jdbc42.internal.apache.arrow.memory.NettyAllocationManager.<clinit>(NettyAllocationManager.java:33)
         ... 14 more

我正在使用從這里借來的源代碼： http://sedeks.blogspot.com/2019/05/how-to-connect-to-databricks-delta.html

相同的源代碼在我使用 JDK 11.0.15 的本地系統中運行良好。

Answer 1

看起來您正在使用不支持使用 Apache 箭頭編碼數據的非小端系統。 您可以嘗試添加EnableArrow=0; 在ssl=true; 部分...