com.simba.spark.jdbc42 驱动程序的 resultset.next() 问题

Question

我在我的 Unix 服务器中使用 JDK 1.8，并使用 spark.jdbc42 连接到 Azure Databricks。

它在resultset.next()的 Unix 服务器中给出以下错误消息：

 Exception in thread "main" java.lang.ExceptionInInitializerError
         at com.simba.spark.jdbc42.internal.apache.arrow.memory.BaseAllocator$Config.getAllocationManagerFactory(BaseAllocator.java:776)
         at com.simba.spark.jdbc42.internal.apache.arrow.memory.ImmutableConfig.access$801(ImmutableConfig.java:24)
         at com.simba.spark.jdbc42.internal.apache.arrow.memory.ImmutableConfig$InitShim.getAllocationManagerFactory(ImmutableConfig.java:83)
         at com.simba.spark.jdbc42.internal.apache.arrow.memory.ImmutableConfig.<init>(ImmutableConfig.java:47)
         at com.simba.spark.jdbc42.internal.apache.arrow.memory.ImmutableConfig.<init>(ImmutableConfig.java:24)
         at com.simba.spark.jdbc42.internal.apache.arrow.memory.ImmutableConfig$Builder.build(ImmutableConfig.java:485)
         at com.simba.spark.jdbc42.internal.apache.arrow.memory.BaseAllocator.<clinit>(BaseAllocator.java:51)
         at com.simba.spark.spark.arrow.ArrowBuffer.<init>(Unknown Source)
         at com.simba.spark.spark.jdbc.SparkJDBC42Driver.createArrowBuffer(Unknown
 Source)
         at com.simba.spark.hivecommon.api.HS2Client.makeRowsetBuffer(Unknown
 Source)
         at com.simba.spark.hivecommon.dataengine.BackgroundFetcher.<init>(Unknown
 Source)
         at com.simba.spark.hivecommon.dataengine.HiveJDBCResultSet.moveToNextRow(Unknown
 Source)
         at com.simba.spark.jdbc.common.BaseForwardResultSet.next(Unknown Source)
 Caused by: java.lang.IllegalStateException: Arrow only runs on LittleEndian systems.
         at com.simba.spark.jdbc42.internal.io.netty.buffer.UnsafeDirectLittleEndian.<init>(UnsafeDirectLittleEndian.java:64)
         at com.simba.spark.jdbc42.internal.io.netty.buffer.UnsafeDirectLittleEndian.<init>(UnsafeDirectLittleEndian.java:50)
         at com.simba.spark.jdbc42.internal.io.netty.buffer.PooledByteBufAllocatorL.<init>(PooledByteBufAllocatorL.java:49)
         at com.simba.spark.jdbc42.internal.apache.arrow.memory.NettyAllocationManager.<clinit>(NettyAllocationManager.java:33)
         ... 14 more

我正在使用从这里借来的源代码： http://sedeks.blogspot.com/2019/05/how-to-connect-to-databricks-delta.html

相同的源代码在我使用 JDK 11.0.15 的本地系统中运行良好。

Answer 1

看起来您正在使用不支持使用 Apache 箭头编码数据的非小端系统。 您可以尝试添加EnableArrow=0; 在ssl=true; 部分...