PySpark 如何遍历 Dataframe 列并更改数据类型？

Question

What is the best way to iterate over Spark Dataframe (using Pyspark) and once find data type of Decimal(38,10) -> change it to Bigint (and resave all to the same dataframe)?

I have a part for changing data types - eg:

df = df.withColumn("COLUMN_X", df["COLUMN_X"].cast(IntegerType()))

but trying to find and integrate with iteration..

Thanks.

Answer 1

You can loop through df.dtypes and cast to bigint when type is equal to decimal(38,10) :

from pyspark.sql.funtions import col

select_expr = [
    col(c).cast("bigint") if t == "decimal(38,10)" else col(c) for c, t in df.dtypes
]

df = df.select(*select_expr)

Answer 2

I found this post https://stackoverflow.com/a/54399474/11268096 , where you can loop trough all columns and cast them to your desired data type.

from pyspark.sql import functions as F

for col in df.columns:
  df = df.withColumn(
    col,
    F.col(col).cast("double")
  )