[英]Join two dataframes using the closest timestamp pyspark
So I am very new to pyspark but I am still unable to correctly create my own query.所以我对 pyspark 很陌生,但我仍然无法正确创建自己的查询。 I try googling my problems but I just don't understand how most of this works.我尝试用谷歌搜索我的问题,但我只是不明白其中大部分是如何工作的。 I'm not sure what I'm missing.我不确定我错过了什么。
But anyway I have the following two dataframes, spark_p:但无论如何我有以下两个数据帧,spark_p:
+--------------------+-----+--------------------+
|__record_timestamp__|cycle| profiles|
+--------------------+-----+--------------------+
| 1651737406300000000| 0|[0, 1, 1, 1, 3, 1...|
| 1651737406300000000| 16|[0, 0, 1, 0, 0, 0...|
| 1651737406300000000| 17|[1, 1, 1, 1, 0, 0...|
| 1651737406300000000| 18|[0, 0, 0, 0, 0, 1...|
| 1651737406300000000| 19|[1, 1, 1, 0, 0, 0...|
+--------------------+-----+--------------------+
and spark_m:和 spark_m:
+-------------+--------------------+
| current|__record_timestamp__|
+-------------+--------------------+
| 0.007181627| 1651730407500000000|
| 8.3004625E-4| 1651730464000000000|
| 0.41976404| 1651730507000000000|
|-0.0017322368| 1651732761000000000|
|-2.5260705E-4| 1651732822500000000|
| 2.3460487E-4| 1651732824500000000|
+-------------+--------------------+
And I need to add a column to spark_p that contains the current at that specific timestamp.我需要向 spark_p 添加一列,其中包含该特定时间戳处的电流。
So the result would look something like:所以结果看起来像:
+--------------------+-----+--------------------+---------+
|__record_timestamp__|cycle| profiles| current|
+--------------------+-----+--------------------+---------+
| 1651737406300000000| 0|[0, 1, 1, 1, 3, 1...| 0.07|
| 1651737406300000000| 16|[0, 0, 1, 0, 0, 0...| 12|
| 1651737406300000000| 17|[1, 1, 1, 1, 0, 0...| 0.0|
| 1651737406300000000| 18|[0, 0, 0, 0, 0, 1...| 5.235654|
| 1651737406300000000| 19|[1, 1, 1, 0, 0, 0...| 125|
+--------------------+-----+--------------------+---------+
Now the time stamps won't exactly match up but I just need the closest timestamp, or to use the value of the previously recorded current, either is fine.现在时间戳不会完全匹配,但我只需要最接近的时间戳,或者使用先前记录的电流值,两者都可以。 I have no idea how...我不知道怎么...
When I try:当我尝试时:
spark_p.join(spark_m, spark_p.__record_timestamp__ == spark_m.__record_timestamp__, "inner").show()
I just get:我只是得到:
+--------------------+-----+--------+-----+--------------------+
|__record_timestamp__|cycle|profiles|value|__record_timestamp__|
+--------------------+-----+--------+-----+--------------------+
+--------------------+-----+--------+-----+--------------------+
So I'm guessing none of them match exactly, but how would I just grab the nearest value?所以我猜它们都不完全匹配,但我怎么才能获取最接近的值呢? TIA TIA
This solution contains the answer:此解决方案包含答案:
SPLIT_COUNT = 90
SPLIT_SIZE = 1024
spark_p = data.select("profiles", '__record_timestamp__')
spark_p = spark_p.withColumn("profiles", F.col("profiles").getField("elements") )
slices = [F.slice(F.col('profiles'), i * SPLIT_SIZE + 1, SPLIT_SIZE) for i in range(SPLIT_COUNT)]
spark_p = spark_p.select(F.posexplode(F.array(*slices)), F.col('__record_timestamp__'))
spark_p = spark_p.withColumn("cycle", F.col("pos") )
spark_p = spark_p.withColumn("profiles", F.col("col") )
spark_p = spark_p.drop('pos').drop('col')
spark_m = magnetData.select("value", '__record_timestamp__', )
spark_p = spark_p.withColumn('value', F.lit(None))
spark_m = spark_m.withColumn('profiles', F.lit(None))
spark_m = spark_m.withColumn('cycle', F.lit(None))
final_df = spark_p.unionByName(spark_m)
w = Window.orderBy('__record_timestamp__').rowsBetween(Window.unboundedPreceding, -1)
final_df = final_df.withColumn('value', F.last('value', True).over(w)).filter(~F.isnull('profiles'))
You must create a window with unboundedPreceding parameter.您必须使用 unboundedPreceding 参数创建一个 window。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.