[英]What's the best way to do matrix multiplication between 2 lists of sparseVectors with DataFrame-based API in pyspark?
I have 2 DataFrame
s with the same structure: DataFrame[id: bigint, tfidf_features: vector]
我有 2
DataFrame
具有相同结构的DataFrame[id: bigint, tfidf_features: vector]
: DataFrame[id: bigint, tfidf_features: vector]
I need to multiple rows in dataframe1
with rows in dataframe2
.我需要在多行
dataframe1
与行dataframe2
。 I can use a loop and do things like: dataframe1.collect()[i]['tfidf_features'].dot(dataframe2.collect()[j]['tfidf_features'])
.我可以使用循环并执行以下操作:
dataframe1.collect()[i]['tfidf_features'].dot(dataframe2.collect()[j]['tfidf_features'])
。
However, I would like to use matrix multiplication, something equivalent to: np.matmul(dataframe1_tfidf_features, dataframe2_tfidf_features.T)
.但是,我想使用矩阵乘法,相当于:
np.matmul(dataframe1_tfidf_features, dataframe2_tfidf_features.T)
。
You have two choices你有两个选择
1. mllib.linalg.distributed.BlockMatrix
convert both dataframes to block matrices and use mulitply 1.
mllib.linalg.distributed.BlockMatrix
将两个数据帧转换为块矩阵并使用mulitply
bm1 = IndexedRowMatrix(df1.rdd.map(lambda x: IndexedRow(x[0], x[1]))).toBlockMatrix()
bm2 = IndexedRowMatrix(df2.rdd.map(lambda x: IndexedRow(x[0], x[1]))).toBlockMatrix()
bm_result = bm1.multiply(bm2)
2. pyspark.sql.dataframe.crossJoin
crossjoin both dataframes and calculate individual element of resultant matrix and then use collect_list & sort 2.
pyspark.sql.dataframe.crossJoin
交叉连接两个数据帧并计算结果矩阵的单个元素,然后使用 collect_list & sort
arr = np.array
df =df1.crossJoin(df2.select(col("id").alias("id2"),
col("features").alias("features2"))
udf_mult = udf(lambda x,y = float(arr(x).dot(arr(y).T).sum()), DoubleType())
df = df.withColumn("val", udf_mult("features","features2")).
drop("features","features2")
st = struct(["id2","val"]).alias("map")
df = df.select("id", st).groupBy("id").agg(collect_list("map").alias("list"))
def sort(x):
x = sorted(x, key=lambda x:x[0])
y = list(map(lambda a:a[1], x))
return(y)
udf_sort = udf(sort, ArrayType(DoubleType()))
df = df.withColumn("list", udf_sort("list"))
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.