使 Spark 代码更高效、更简洁

Question

我有以下代码来清理文档语料库（ pipelineClean(corpus) ），它返回一个包含两列的数据框：

• “id”：长
• “令牌”：数组[字符串]。

之后，代码生成一个包含以下列的数据框：

• “术语”：字符串
• "postingList": List[Array[Long, Long]] (第一个 long 是文档中的另一个词频)

pipelineClean(corpus)
        .select($"id" as "documentId", explode($"tokens") as "term") // explode creates a new row for each element in the given array column
        .groupBy("term", "documentId").count //group by and then count number of rows per group, returning a df with groupings and the counting
        .where($"term" =!= "") // seems like there are some tokens that are empty, even though Tokenizer should remove them
        .withColumn("posting", struct($"documentId", $"count")) // merge columns as a single {docId, termFreq}
        .select("term", "posting")
        .groupBy("term").agg(collect_list($"posting") as "postingList") // we do another grouping in order to collect the postings into a list
        .orderBy("term")
        .persist(StorageLevel.MEMORY_ONLY_SER)

我的问题是：是否有可能使这段代码更短和/或更高效？ 例如，是否可以在单个groupBy内进行分组？

Answer 1

除了跳过withColumn调用并使用直接选择之外，您似乎无法做更多的事情：

.select(col("term"), struct(col("documentId"), col("count")) as "posting")

代替

.withColumn("posting", struct($"documentId", $"count")) // merge columns as a single {docId, termFreq}
        .select("term", "posting")