两个 Spark 数据帧的并集

Question

I tried to do an union between two Spark DataFrame in Python, one of them sometime is empty, I did a test if, to return that full. This following a small code for example, it return an error:

>>> from pyspark.sql.types import *
>>> fulldataframe = [StructField("FIELDNAME_1",StringType(), True),StructField("FIELDNAME_2", StringType(), True),StructField("FIELDNAME_3", StringType(), True)]
>>> schema = StructType([])
>>>
>>> dataframeempty = sqlContext.createDataFrame(sc.emptyRDD(), schema)
>>> resultunion = sqlContext.createDataFrame(sc.emptyRDD(), schema)
>>> if (fulldataframe.isEmpty()):
...     resultunion = dataframeempty
... elif (dataframeempty.isEmpty()):
...     resultunion = fulldataframe
... else:
...     resultunion=fulldataframe.union(dataframeempty)
...


Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'list' object has no attribute 'isEmpty'
>>>

Someone can tell me where's the fault ?

Answer 1

Count can take long. In Scala:

dataframe.rdd.isEmpty()