How to obtain row percentages of crosstab from a spark dataframe using python?

Question

I used python code:

df.stat.crosstab("age", "y").orderBy("age_y").show()

to create a crosstab from a spark dataframe as follows:

However, I cannot find a code to obtain the row percentages. For example, age 18 row percentages should be 5/12 = 41.7% for 'no' and 7/12 = 58.3% for 'yes'. The sum of 2 percentages is 100%.

May someone advise me in this case? Many thanks in advance.

Answer 1

Simply add 2 columns using using withColumn and your formula to calculate the percentages:

from pyspark.sql import functions as F

df1 = df.stat.crosstab("age", "y").orderBy("age_y")

result = df1.withColumn(
    "no_rp",
    F.round(F.col("no") / (F.col("no") + F.col("yes")) * 100, 2)
).withColumn(
    "yes_rp",
    F.round(F.col("yes") / (F.col("no") + F.col("yes")) * 100, 2)
)

result.show()

#+-----+---+---+-----+------+
#|age_y| no|yes|no_rp|yes_rp|
#+-----+---+---+-----+------+
#|   18|  5|  7|41.67| 58.33|
#|   19| 24| 11|68.57| 31.43|
#|   20| 35| 15| 70.0|  30.0|
#+-----+---+---+-----+------+