如何验证一个数组是否包含另一个数组

Question

I have a pyspark dataframe that contain 4 columns.

Example Dataframe:

id                       |  name                          | age |  job
    -------------------------------------------------------------------
     ["98475", "748574"] |  ["98475",748574]              |
    -------------------------------------------------------------------
      ["75473","98456"]  |   ["98456"]                    |
    -------------------------------------------------------------------
      ["23456","28596"]  |   ["84758","56849","86954"]      
    -------------------------------------------------------------------

I want to compare 2 columns (array<string> types) :

Example:

Array_A (id)  | Array_B(name)
------------------------------

if all the values in the Array_B are matches are the values in the Array_A ==> ok

if all the values in the Array_B are in the array_A ==> medium

if the values of the Array_B are not exist in the array_A ==> not found

I did an UDF:

def contains(x,y):
        z = len(set(x) - set(y))
        if ((z == 0) & (set(x) == set(y))):
            return "ok"
        elif (set(y).isin(set(x))) & (z != 0):
            return "medium"
        else set(y) != set(x):
            return "not found in raw"


contains_udf = udf(contains)

Then:

new_df= df.withColumn(
    "new_column",
    F.when(
        (df.id.isNotNull() & df.name.isNotNull()),
        contains_udf(df.id,df.name)
    ).otherwise(
        F.lit(None)
    )

)

I got this error:

else set(y) != set(x):
           ^
SyntaxError: invalid syntax

How can I resolve it using udf or another solution like array_contains perhaps? Thank you

Answer 1

else set(y) != set(x):
           ^
SyntaxError: invalid syntax

This is because an else statement doesn't require a condition. It contains code to be executed only if none of the previous conditions are met. Use instead:

elif set(y) != set(x):
    #code

OR

else :
    #code

Answer 2

As @Buckeye14Guy and @Sid pointed out the main problems in your code, you might also need to clean out some of the logic:

from pyspark.sql.functions import udf

def contains(x,y): 
  try:
    sx, sy = set(x), set(y) 
    if len(sy) == 0: 
        return 'list is empty'
    elif sx == sy: 
        return "ok"    
    elif sy.issubset(sx): 
        return "medium"  
    # below none of sy is in sx
    elif sx - sy == sx: 
        return "none found in raw"  # including empty x
    else: 
        return "some missing in raw"
  # in exception, for example `x` or `y` is None (not a list)
  except:
    return "not an iterable or other errors"

udf_contains = udf(contains, 'string')

df.withColumn('new_column', udf_contains('id', 'name')).show(truncate=False)
+---------------+---------------------+-----------------+
|id             |name                 |new_column       |
+---------------+---------------------+-----------------+
|[98475, 748574]|[98475, 748574]      |ok               |
|[75473, 98456] |[98456]              |medium           |
|[23456, 28596] |[84758, 56849, 86954]|none found in raw|
+---------------+---------------------+-----------------+