java.lang.ClassCastException: java.lang.Integer cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow

Question

我是 Spark Structured Streaming 編程的新手。 使用此F.approx_count_distinct后出現此錯誤，這是我的代碼。 我的問題是我想獲得一個檢測欺詐的 dataframe，但首先我需要檢查是否有人擁有相同的card_number 。 誰能幫我？ 提前致謝。

from pyspark.sql import SparkSession
from pyspark import SparkContext, SparkConf
from pyspark.sql.functions import from_json, col
from pyspark.sql import functions as F
from pyspark.sql.functions import when
from pyspark.sql.types import *
conf = SparkConf().setAppName("Pruebas").setMaster("local")
sc = SparkContext(conf=conf)
sc.setLogLevel("ERROR")
sparkSQL = SparkSession \
.builder \
.appName("SparkSQL") \
.master("local") \
.getOrCreate()
broker="localhost:9092"
topic = "transacts"

# Construir el dataframe de streaming
df = sparkSQL \
.readStream \
.format("kafka") \
.option("kafka.bootstrap.servers", broker) \
.option("failOnDataLoss", "false") \
.option("subscribe", topic) \
.option("startingOffsets", "latest") \
.option("includeTImestamp", "true") \
.load()

# Definir el esquema que utilizaremos en el json
schema = StructType([ StructField("card_owner", StringType(), True),
StructField("card_number", StringType(), True),
StructField("geography", StringType(), True),
StructField("target", StringType(), True),
StructField("amount",  StringType(), True),
StructField("currency", StringType(), True)])


# decodificar el json
# al decodificar el json nos genera una serie de subcolumnas dentro del campo value
df = df.withColumn("value", from_json(df["value"].cast("string"), schema))
df.printSchema()

# seleccionamos el timestamp del mensaje y las columnas del json
df = df.select("timestamp","value.*")
df1 = df.groupBy(df.card_number).agg(F.approx_count_distinct(df.card_owner).alias('titulares')).filter((F.col('titulares')>1))

df1 = df1.selectExpr("'a' as key", "to_json(struct(*)) as value")

query= df1.writeStream\
.outputMode("complete")\
.format("kafka")\
.option("topic","aux_topic1")\
.option("kafka.bootstrap.servers", "localhost:9092")\
.option("checkpointLocation","hdfs://localhost:9000/checkpoints")\
.start()
#query.awaitTermination(200)

# Paso de json a df

topic1= "aux_topic1"
df1 = sparkSQL \
.readStream \
.format("kafka") \
.option("kafka.bootstrap.servers", broker) \
.option("failOnDataLoss", "false") \
.option("subscribe", topic1) \
.option("startingOffsets", "latest") \
.option("includeTImestamp", "true") \
.load()

# Definir el esquema que utilizaremos en el json
schema = StructType([ StructField("card_number", StringType(), True),
StructField("titulares", StringType(), True)])

# decodificar el json
df1 = df1.withColumn("value", from_json(df1["value"].cast("string"), schema))
df1.printSchema()

df1 = df1.select("timestamp","value.*")
df2=df.join(df1, on="card_number")

#Mostrar por pantalla
query1= df2.writeStream\
.outputMode("append")\
.format("console")\
.queryName("test")\
.start()
query1.awaitTermination()

Answer 1

問題似乎是這一行：

df1 = df \
    .groupBy(df.card_number) \
    .agg(F.approx_count_distinct(df.card_owner).alias('titulares')) \
    .filter((F.col('titulares')>1))

更准確地說，您的過濾器.filter((F.col('titulares')>1))

---

如果您想獲得所有出現多次的卡號，則可以使用以下方法：

這是你的 dataframe

df.show()
+-----------+-------------+                                                     
|card_number|   card_owner|
+-----------+-------------+
|      12345| Andrew Smith|
|      98765|   John Brown|
|      12345| Andrew Smith|
|      98765|   John Brown|
|      33445|Maria Johnson|
+-----------+-------------+

現在獲取每個卡號的所有計數（過濾掉那些沒有重復的）：

>>> df \
...     .groupBy('card_number') \
...     .count() \
...     .filter('count>1') \
...     .show()
+-----------+-----+
|card_number|count|
+-----------+-----+
|      12345|    2|
|      98765|    2|
+-----------+-----+

現在，如果您也想要card_owner ，那么：

>>> df \
...     .groupBy(['card_number', 'card_owner']) \
...     .count() \
...     .filter('count>1') \
...     .show()
+-----------+------------+-----+
|card_number|  card_owner|count|
+-----------+------------+-----+
|      12345|Andrew Smith|    2|
|      98765|  John Brown|    2|
+-----------+------------+-----+