嗨，我正在尝试在不使用 spark_df.collect() 的情况下迭代 pyspark 数据帧

Question

HI I am trying to iterate over pyspark data frame without using spark_df.collect() and I am trying foreach and map method is there any other way to iterate?

df.foreach(lambda x: print(x)) and

def func1(x):
    firstname=x.firstname
    lastName=x.lastName
    name=firstName+","+lastName
    gender=x.gender.lower()
    salary=x.salary*2
    return (name,gender,salary)```

```rdd2=df.rdd.map(lambda x: func1(x))```

is there any other way to iterate over data frame

Answer 1

First of all, Spark is not made to do such kind of operations, like printing each records but more on distributed processing. Tune your process to work in a distributed fashion, like in terms of joins - that will unleash the power of spark.

If you want to process each record, UDFs ( User defined functions ) are a good way to do that. UDFs will be applied to each record once.

Answer 2

we can use this method to iterate over rows

pandasDF = df.toPandas()
for index, row in pandasDF.iterrows():
    print(row['itm_mtl_no'], row['itm_src_sys_cd']) ```