[英]How to collect a single row dataframe and use fields as constants
I'm trying to read a single row from a Hive table and use its fields as constants in the rest of my Spark application. 我试图从Hive表中读取一行,并将其字段用作我的Spark应用程序其余部分中的常量。
object IniConstHive extends Serializable {
val techTbl: DataFrame = spark.table("my_db.my_conf_table").orderBy(col("my_date").desc)
val firstrow: Row = techTbl.head
val my_firstfield: Double = firstrow.getAs[java.math.BigDecimal](0).doubleValue
val my_secondfield: Double = firstrow.getAs[java.math.BigDecimal](1).doubleValue
}
Unfortunately, I get this exception when I call IniConstHive.my_firstfield: 不幸的是,当我调用IniConstHive.my_firstfield时遇到了这个异常:
Caused by: java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:347)
at scala.None$.get(Option.scala:345)
at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask(BlockInfoManager.scala:343)
at org.apache.spark.storage.BlockManager.releaseAllLocksForTask(BlockManager.scala:676)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:329)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Is this the right approach to do this? 这是正确的方法吗? Is there a more efficient way to acheive the result?
有没有更有效的方法来达到结果?
It turned out the problem was not in the IniConstHive object but the place where I called it. 事实证明,问题不在IniConstHive对象中,而在我调用它的地方。 In fact, I was trying to call IniConstHive.my_firstfield inside a custom UDAF, which runs inside executors.
实际上,我试图在自定义UDAF中调用IniConstHive.my_firstfield,该自定义UDAF在执行程序内部运行。 For this reason my application tried to instantiate multiple SparkContext outside the driver.
因此,我的应用程序尝试在驱动程序之外实例化多个SparkContext。
I solved by calling IniConstHive.my_firstfield in the driver and then passing the result as a parameter in the UDAF constructor. 我通过在驱动程序中调用IniConstHive.my_firstfield并将结果作为参数传递给UDAF构造函数来解决。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.