列不可迭代-Apache Spark DataFrame-Python

Question

I have a column int_rate of type string in my spark dataframe and all its value are like 9.5%, 7.0%, etc .

Here is an image of how the column looks

Now I know that there is a way in which I can convert type string to float in python, but it is only applicable when the value would have be 9.5 without % symbol. I tried the following method:

df.int_rate = [x.strip('%') for x in df.int_rate]

given on this link to remove the % symbol, but it throws an error saying:

Column is not iterable

I also tried the other methods listed on the link , but nothing seems to work. Can someone please help me to get rid of the % symbol and convert my column to type float?

Answer 1

One possible solution:

from pyspark.sql.functions import expr

df = spark.createDataFrame(["9.5%", "7.0%"], "string").toDF("int_rate")

df.withColumn("int_rate", expr("rtrim('%', int_rate)").cast("float")).show()

and another

from pyspark.sql.functions import regexp_replace

df.withColumn("int_rate", regexp_replace("int_rate", "%$", "").cast("float"))