AWS Glue SelectFields 和 Filter 不采用动态值

Question

I have an AWS Glue script written which does field selection and Filtering using SelectFields() and Filter() methods. I have tested these with static values and works fine, however, they do not work when dynamic values are passed in the same format. Any idea why the dynamic values are not taken? I tested out by passing one of the dynamic values and for this case, both methods work.

Pls also note that the passed key (filterkey) works if its static or dynamic

wordstoFilter = ['USA', 'France']
columnstoSelect = ['cust_id', 'custname', 'state']


#join and return all list values in single quote along with comma
fltr_string =', '.join(["'{}'".format(value) for value in wordstoFilter])
select_string =', '.join(["'{}'".format(value) for value in columnstoSelect ])


filterkey = "country"
#below statement works with static value
#country_filter_dyf = Filter.apply(frame=custData, f=(lambda x: x["country"] in ["USA"]))
country_filter_dyf = Filter.apply(frame=custData, f=(lambda x: x[filterkey] in [fltr_string]))

##Select case
#below statement works with static value
#selected_fields_dyf = SelectFields.apply(frame = custData, paths = ['cust_id', 'cust_name', 'state', 'country'])

#Below one doesn't work
selected_dyf = SelectFields.apply(frame = custData, paths = [select_string ])

Answer 1

As I see, the paths argument expects you to give a list but you give a str object:

>>> type(['cust_id', 'cust_name', 'state', 'country'])
<class 'list'>
>>> type(select_string)
<class 'str'>

Have you tried to give the list directly?

>>> type(columnstoSelect)
<class 'list'>

columnstoSelect = ['cust_id', 'custname', 'state']
selected_dyf = SelectFields.apply(frame = custData, paths = columnstoSelect )