数据类型转换Spark数据框架列-pyspark

Question

我有一个要投射的spark DataFrame中的列['col1','col2','col3']的列表。 我尝试了以下操作，但似乎无法正常工作。 我确实尝试过将数据从DECIMAL（11,3）转换为DECIMAL（3,2）和INT

create table database_nm.test_kou
(
YEAR_MNTH_NBR        INT,
PARTN_ID             TINYINT,
EMP_NAME             STRING,
FST_AMT              DECIMAL(11,3),
SND_AMT              DECIMAL(11,3),
FST_PCT              DECIMAL(11,3),
SND_PCT              DECIMAL(11,3),
TRD_AMT              DECIMAL(11,3),
TRD_PCT              DECIMAL(11,3),
FTH_AMT              DECIMAL(11,3)
);

INSERT INTO database_nm.test_kou VALUES 
    (201812,1,'Kou',11.11,12.12,13.13,14.14,15.15,16.16,17.17);

INSERT INTO database_nm.test_kou VALUES
    (201812,1,'Cha',21.21,22.22,23.23,24.24,25.25,26.26,27.27);

INSERT INTO database_nm.test_kou VALUES 
    (201812,1,'Cha',21.21,22.22,23.23,24.24,25.25,100.00,27.27);

我的pyspark代码：

col_unfmt=sql("select * from database_nm.test_kou")
colWithAmt = [column for column in col_unfmt.columns if '_amt' in column]
colWithPct = [column for column in col_unfmt.columns if '_pct' in column]
colWithRemainings = [
    column for column in col_unfmt.columns if column not in colWithPct+colWithAmt
]

selectAmtColsExpr = [lit(y).cast("int").alias(y) for y in colWithAmt]
selectPctColsExpr = [(lit(z)/lit(100)).cast("decimal(3,2)").alias(z) for z in colWithPct]
selectRemainingColsExpr = colWithRemainings

col_unfmt.select(selectRemainingColsExpr+selectPctColsExpr+selectAmtColsExpr).show()
+-------------+--------+--------+-------+-------+-------+-------+-------+-------+-------+
|year_mnth_nbr|partn_id|emp_name|fst_pct|snd_pct|trd_pct|fst_amt|snd_amt|trd_amt|fth_amt|
+-------------+--------+--------+-------+-------+-------+-------+-------+-------+-------+
|       201812|       1|     Kou|   null|   null|   null|   null|   null|   null|   null|
|       201812|       1|     Cha|   null|   null|   null|   null|   null|   null|   null|
+-------------+--------+--------+-------+-------+-------+-------+-------+-------+-------+

Answer 1

正如Pault上面建议的那样，使用col而不是lit看起来可以解决问题。

>>> selectAmtColsExpr = [col(y).cast("int") for y in colWithAmt]
>>> selectPctColsExpr = [(col(z)/100).cast("decimal(3,2)").alias(z) for z in colWithPct]
>>> selectRemainingColsExpr = colWithRemainings
col_unfmt.select(selectRemainingColsExpr+selectPctColsExpr+selectAmtColsExpr).show(20,False)
+-------------+--------+--------+-------+-------+-------+-------+-------+-------+-------+
|year_mnth_nbr|partn_id|emp_name|fst_pct|snd_pct|trd_pct|fst_amt|snd_amt|trd_amt|fth_amt|
+-------------+--------+--------+-------+-------+-------+-------+-------+-------+-------+
|201812       |1       |Kou     |0.13   |0.14   |0.16   |11     |12     |15     |17     |
|201812       |1       |Cha     |0.23   |0.24   |0.26   |21     |22     |25     |27     |
|201812       |1       |Cha     |0.23   |0.24   |1.00   |21     |22     |25     |27     |
+-------------+--------+--------+-------+-------+-------+-------+-------+-------+-------+