Pyspark dataframe get all values of a column

Question

I want to get all values of a column in pyspark dataframe. I did some search, but I never find a efficient and short solution.

Assuming I want to get a values in the column called "name". I have a solution:

sum(dataframe.select("name").toPandas().values.tolist(),[])

It works, but it is not efficient since it converts to pandas then flatten the list... Is there a better and short solution?

Answer 1

Below Options will give better performance than sum .

Using collect_list

import pyspark.sql.functions as f
my_list = df.select(f.collect_list('name')).first()[0]

Using RDD:

my_list = df.select("name").rdd.flatMap(lambda x: x).collect()

I am not certain but in my couple of stress test, collect_list gives better performance. Will be great if someone can confirm.

Answer 2

To get all values of a column in a list we can use collect() -

column_value_list = [row['<column_name>'] for row in df.select('<column_name>').collect()]