Azure Databricks - 写入 Az SQL 表

Question

Am reading a. parquet (sourced from MySql) file into Databricks as a DataFrame and wanted to convert few column datatypes to SQL data types.

Example :

In this case want to cast columns active and is_agent to SQL DataType bit and write back to new data frame.

I want to loop through all columns in the data frame and apply above cast where source column data type is Byte.

How can this be achieved using Python.

Answer 1

Ok got this one done as

from pyspark.sql.functions import col 
from pyspark.sql.types import BooleanType

df = sourceDF.withColumn("active", col("active").cast(BooleanType())).withColumn("is_agent", col("active").cast(BooleanType()))

What I want is a function where I will pass DataFrame and loop through all column and and cast column of DataType Byte to BooleanType.