使用元组的RDD更新PySpark数据框列

Question

I am working with data where user info is string. I would like to assign unique integer values to those strings.

I was somewhat following this stack overflow post here . I am using the expression below to have an RDD of tuples:

user = data.map(lambda x:x[0]).distinct().zipWithUniqueId()

After that, I did

data = data.map(lambda x: Rating(int(user.lookup(x[0])), int(x[1]), float(x[2]))) 

What I ultimately want to do is run an ALS model on it, but so far I have been getting this error message

Exception: It appears that you are attempting to broadcast an RDD or reference an RDD from an action or transformation.

I think the data type is somehow wrong, but I am not sure how to fix this.

Answer 1

lookup approach suggested in the linked answer is simply invalid. Spark doesn't support nested action nor transformations so you cannot call RDD.lookup inside a map . If data is to large to be handled using a standard Python dict for lookups you can simply join and reshape:

from operator import itemgetter
from pyspark.mllib.recommendation import Rating

data = sc.parallelize([("foo", 1, 2.0), ("bar", 2, 3.0)])

user = itemgetter(0)

def to_rating(record):
    """
    >>> to_rating((("foobar", 99, 5.0), 1000))
    Rating(user=1000, product=99, rating=5.0)
    """
    (_, item, rating), user = record
    return Rating(user, item, rating)

user_lookup = data.map(user).distinct().zipWithIndex()

ratings = (data
    .keyBy(user)  # Add user string as a key
    .join(user_lookup)  # Join with lookup
    .values()  # Drop keys
    .map(to_rating))  # Create Ratings

ratings.first()
## Rating(user=1, product=1, rating=2.0)