如何遍历 pyspark 中未知数据帧的列的行

Question

I am new to Data Science and I am working on a simple self project using Google Colab. I took a data from a something.csv file and the file's columns are encrypted with #### , so I don't know the names of the columns. I took

Here is my attempt to solve it using pyspark

df = spark.read.csv('something.csv', header=True)
col = df[df.columns[len(df.columns)-1]] #Taking last column of data-frame

Now I want to iterate through rows of 'col' column and print the rows that has a number less than 100. I searched other stackoverflow posts but didn't understand how to iterate through the column with no name.

Answer 1

In pyspark use .filter method on dataframe to filter records < 100.

#sample data po column is int
df.show()
#+---+----+---+
#| id|name| po|
#+---+----+---+
#|  1|   2|300|
#|  2|   1| 50|
#+---+----+---+

last_col = df[df.columns[len(df.columns)-1]]

from pyspark.sql.functions import *

df.filter(last_col < 100).show()
#+---+----+---+
#| id|name| po|
#+---+----+---+
#|  2|   1| 50|
#+---+----+---+

---

UPDATE:

#getting rows into list
lst=df.filter(last_col < 100).select(last_col).rdd.flatMap(lambda x:x)
lst.collect()
#[50]

to get all rows into list
lst=df.filter(last_col < 100).rdd.flatMap(lambda x:x)
lst.collect()
#[u'2', u'1', 50]