Create new column from existing Dataframe

Question

I am having a Dataframe and trying to create a new column from existing columns based on following condition.

Group data by column named Only filter those rows where column has value and call it . 组数据只能过滤，其中列具有价值的行并将其命名为 Values for new column are / /

Here is input Dataframe

+-----+-------------+----------+--------------+------+
|   id|   event_type|  location|fault_severity|source|
+-----+-------------+----------+--------------+------+
| 6597|event_type 11|location 1|            -1|  test|
| 8011|event_type 15|location 1|             0| train|
| 2597|event_type 15|location 1|            -1|  test|
| 5022|event_type 15|location 1|            -1|  test|
| 5022|event_type 11|location 1|            -1|  test|
| 6852|event_type 11|location 1|            -1|  test|
| 6852|event_type 15|location 1|            -1|  test|
| 5611|event_type 15|location 1|            -1|  test|
|14838|event_type 15|location 1|            -1|  test|
|14838|event_type 11|location 1|            -1|  test|
| 2588|event_type 15|location 1|             0| train|
| 2588|event_type 11|location 1|             0| train|
+-----+-------------+----------+--------------+------+

and i want following output.

 +--------------+------------+-----------+
 |              | event_type | PercTrain |
 +--------------+------------+-----------+
 |event_type 11 |   7888     | 0.388945  |
 |event_type 35 |   6615     | 0.407105  |
 |event_type 34 |   5927     | 0.406783  |
 |event_type 15 |   4395     | 0.392264  |
 |event_type 20 |   1458     | 0.382030  |
 +--------------+------------+-----------+

I have tried this code but this throws error

    EventSet.withColumn("z" , when($"source" === "train" , sum($"source") / length($"source"))).groupBy("fault_severity").count().show()

Here EventSet is input dataframe

Python code that gives desired output is

event_type_unq['PercTrain'] = event_type.pivot_table(values='source',index='event_type',aggfunc=lambda x: sum(x=='train')/float(len(x))) 

Answer 1

I guess you want to obtain the percentage of train values. So, here is my code,

val df2 = df.select($"event_type", $"source").groupBy($"event_type").pivot($"source").agg(count($"source")).withColumn("PercTrain", $"train" / ($"train" + $"test")).show

and gives the result as follows:

+-------------+----+-----+------------------+
|   event_type|test|train|         PercTrain|
+-------------+----+-----+------------------+
|event_type 11|   4|    1|               0.2|
|event_type 15|   5|    2|0.2857142857142857|
+-------------+----+-----+------------------+

Hope to be helpful.