使用帶有 Python 的 Spark 結構化流的字數

Question

我對 Spark 很陌生。 此示例摘自 Spark 的結構化流式編程指南：

from pyspark.sql import SparkSession
from pyspark.sql.functions import explode
from pyspark.sql.functions import split

spark = SparkSession \
            .builder \
            .appName("StructuredNetworkWordCount") \
            .getOrCreate()

# Create DataFrame representing the stream of input lines from connection to localhost:9999
       lines = spark \
         .readStream \
         .format("socket") \
         .option("host", "localhost") \
         .option("port", 9999) \
         .load()

# Split the lines into words
      words = lines.select(
        explode(
   split(lines.value, " ")
   ).alias("word")
   )

 # Generate running word count
     wordCounts = words.groupBy("word").count()

 # Start running the query that prints the running counts to the console
    query = wordCounts \
          .writeStream \
          .outputMode("complete") \
          .format("console") \
          .start()

query.awaitTermination()

我需要修改此代碼以計算以字母“B”開頭且計數超過 6 個的單詞。 我該怎么做？

Answer 1

解決方案是：

wordCountsDF = wordsDF.groupBy('word').count().where('word.startsWith("B")' and 'count > 6')