转换火花dataframe柱

Question

I have a spark dataframe with two columns; Stars (numeric value), and categories (string of tags, eg.: "Restaurant, Italien, High-end"). I wish to recreate the dataframe so that categories is instead a count of tags. in the above example categories would instead become 3.

I've tried treating the dataframe as a pandas dataframe, but it does not seem to work. I am new to Spark so perhaps it is because i don't really grasp the idea of RDD's.

Answer 1

Please paste the code for better understanding of problem. Based on the description of your problem, you can try something like this:

df['CategoryCount'] = df['categories'].str.split(',').str.len()

df is your original dataframe and CategoryCount is new column which contains the count of tags. You can drop your categories column also if you want.