从一个 PySpark 数据框中获取 ArrayType 列并在另一个数据框中获取相应的值

Question

I have two dataframes, one called itemsets and another called rules from running FPGrowth. They are formatted like so:

ITEMSETS DATAFRAME
+------------------------+-----+
|cart                    |freq |
+------------------------+-----+
|[7084781116]            |10492|
|[7084781116, 2606500532]|362  |
|[7084781116, 0]         |327  |
|[7084781116, 2001]      |393  |
|[7084781116, 4011]      |489  |
|[4460030705]            |448  |
|[3800035800]            |539  |
|[3022304060]            |1188 |
|[2800021730]            |901  |
|[1380018805]            |437  |
+------------------------+-----+

RULES DATAFRAME
+--------------------+------------+
|          antecedent|  consequent|
+--------------------+------------+
|        [1686, 4068]|[2640000010]|
|[1675, 4432, 3680...|      [1673]|
|[1676, 1677, 3680...|      [1678]|
|[1676, 1677, 3680...|[3680080816]|
|[1676, 1677, 3680...|[3680044476]|
|[1676, 1677, 3680...|      [1675]|
|[7283040006, 7110...|      [1683]|
|[7283040006, 7110...|      [1682]|
|[1674, 4300000953...|      [1673]|
|[1674, 4300000953...|[3680044476]|
+--------------------+------------+

I need to add a few new columns to perform a calculation (to reverse the antecedent and consequent relationship, if you're curious.)

In the first, I need to add a new column to the rules dataframe that has the corresponding frequencies from the itemsets dataframe. So for example, where the consequent is [7084781116], the new column will have the frequency of that array from the itemsets dataframe (10492, via the first row.)

Next, I need to append the value of the consequent to the antecedent, and then do the same thing. So for example, looking at the second row in the rules column, I need to add 1673 to [1675, 4432, 3680...] and then get the frequency of THAT array from the itemsets table, and store it in another column.

Can anyone help me out with this? I'm pretty new to PySpark and in over my head.I tried implementing several UDFs, for example with something like the following, with the hopes of converting the arrays to strings to possibly make them easier to work with:

In [6]: conv_to_str([1,3,2,6,5])
Out[6]: '1|2|3|5|6|'

In [7]: def conv_to_str(listname):
        listname.sort()
        rv = ""
        for val in listname:
            rv += str(val) + "|"
        return rv[:-1]

In [8]: conv_to_str([1,3,2,6,5])
Out[8]: '1|2|3|5|6' 

Thanks!

Answer 1

I suggest these three steps:

1. For adding the frequency column use left-join on the consequent & cart columns.
2. For adding the consequent value to the antecendent array use concat function (supported for arrays since Spark 2.4).
3. Do the left-join again on the enriched antecendent column to get the frequency of this concatenated array.

So in PySpark the query could look like this:

(
  rules_df
  .withColumn('antecedent_enriched', concat('antecendent', 'consequent'[0]))
  .alias('a')
  .join(itemsets_df.alias('b'), col('a.consequent') == col('b.cart'), 'left')
  .join(itemsets_df.alias('c'), col('a.antecedent_enriched') == col('c.cart'), 'left'))
  .select(
    'antecedent',
    'consequent',
    'b.freq',
    'antecedent_enriched',
    col('c.freq').alias('freq_enriched')
  )
)

Also be careful when using the concat function since if the consequent column contains Null values the result of the concatenation will be also Null.