轉換JavaObject`scala.collection.Map <Object,RDD<?> &gt;`到python字典

Question

在pyspark中，調用getPersistentRDDs getPersistentRDDs() Java sparkContext方法將返回scala.collection.Map<Object,RDD<?>> JavaObject實例。

from pyspark.sql import SparkSession
from pyspark import StorageLevel

spark = SparkSession.builder.master('yarn').getOrCreate()
sc = spark.sparkContext

df = spark.range(0, 25000000, 1)
df.persist(StorageLevel.MEMORY_ONLY)
df.limit(1).count()

sc._jsc.sc().getPersistentRDDs()

返回JavaObject id=o477

如何將scala.collection.Map<Object,RDD<?>>的scala.collection.Map<Object,RDD<?>>轉換為python字典？

Answer 1

from pyspark import RDD

scala_map = sc._jsc.sc().getPersistentRDDs()
py_dict = {e._1(): RDD(e._2().toJavaRDD(), sc) for e in [scala_map.toList().apply(i) for i in range(scala_map.size())]}