python spark change dataframe column data type to int error

Question

I want to cast the column type to int and get the first 3 rows

    df.withColumn("rn", rowNumber().over(windowSpec).cast('int')).where("rn"<=3).drop("rn").show()

but I this error

TypeError: unorderable types: str() <= int()

Answer 1

The error is here:

.where("rn"<=3)

And here's how you can figure that out if you ever encounter a similar problem in the future. Following

TypeError: unorderable types: str() <= int()

is a Python exception and there is no Py4JError . This typically means you can dismiss JVM issues and focus on core Python. The only part of your code where you explicitly compare things is:

"rn" <= 3

If you want it to be a SQL literal you should pass a string:

.where("rn <= 3")

If you want rn to be resolved as a column use col function:

from pyspark.sql.functions import col

.where(col("rn") <= 3)

Also rowNumber function has been removed in the latest release. You should use row_number for a forward compatibility.