转换scala.collection.Seq的Java对象 <String> 到python列表

Question

In a pyspark sparkSession , with sc as my sparkContext , calling the getRDDStorageInfo() Java sparkContext, _jsc.sc() , method returns a JavaObject instance of RDDInfo[] that I can iterate over in python.

sc._jsc.sc().getRDDStorageInfo() returns JavaObject id=o473

So the following works:

for s in sc._jsc.sc().getRDDStorageInfo():
    print s.name()
    print s.memSize()
    print s.numPartitions()
    print s.numCachedPartitions()

However, the Java sparkContext method listFiles() returns a JavaObject of scala.collection.Seq<String> .

sc._jsc.sc().listFiles() returns JavaObject id=o475

and when I try and iterate over that I get TypeError: 'JavaObject' object is not iterable

How can I convert the JavaObject of scala.collection.Seq<String> into a python list (or other python iterable)?

Answer 1

You can call scala methods size and apply to do this

files = sc._jsc.sc().listFiles()
py_files = [files.apply(i) for i in range(files.size())]