I am woking on pyspark
in python3
and I have pyspark.sql.DataFrame
like the folliwing:
df.show()
+--------------------+----+----+---------+----------+---------+----------+---------+
| ID|Code|bool| lat| lon| v1| timestamp| v3|
+--------------------+----+----+---------+----------+---------+----------+---------+
|5ac52674ffff34c98...|IDFA| 1|42.377167| -71.06994|17.422535|1525319638|36.853622|
|5ac52674ffff34c98...|IDFA| 1| 42.37747|-71.069824|17.683573|1525319639|36.853622|
|5ac52674ffff34c98...|IDFA| 1| 42.37757| -71.06942|22.287935|1525319640|36.853622|
|5ac52674ffff34c98...|IDFA| 1| 42.37761| -71.06943|19.110023|1525319641|36.853622|
|5ac52674ffff34c98...|IDFA| 1|42.377243| -71.06952|18.904774|1525319642|36.853622|
|5ac52674ffff34c98...|IDFA| 1|42.378254| -71.06948|20.772903|1525319643|36.853622|
|5ac52674ffff34c98...|IDFA| 1| 42.37801| -71.06983|18.084948|1525319644|36.853622|
|5ac52674ffff34c98...|IDFA| 1|42.378693| -71.07033| 15.64326|1525319645|36.853622|
|5ac52674ffff34c98...|IDFA| 1|42.378723|-71.070335|21.093477|1525319646|36.853622|
|5ac52674ffff34c98...|IDFA| 1| 42.37868| -71.07034|21.851894|1525319647|36.853622|
|5ac52674ffff34c98...|IDFA| 1|42.378716| -71.07029|20.583202|1525319648|36.853622|
|5ac52674ffff34c98...|IDFA| 1| 42.37872| -71.07067|19.738768|1525319649|36.853622|
|5ac52674ffff34c98...|IDFA| 1|42.379112| -71.07097|20.480911|1525319650|36.853622|
|5ac52674ffff34c98...|IDFA| 1| 42.37952| -71.0708|20.526752|1525319651| 44.93808|
|5ac52674ffff34c98...|IDFA| 1| 42.37902| -71.07056|20.534052|1525319652| 44.93808|
|5ac52674ffff34c98...|IDFA| 1|42.380203| -71.0709|19.921381|1525319653| 44.93808|
|5ac52674ffff34c98...|IDFA| 1| 42.37968|-71.071144| 20.12599|1525319654| 44.93808|
|5ac52674ffff34c98...|IDFA| 1|42.379696| -71.07114|18.760069|1525319655| 36.77853|
|5ac52674ffff34c98...|IDFA| 1| 42.38011| -71.07123|19.155525|1525319656| 36.77853|
|5ac52674ffff34c98...|IDFA| 1| 42.38022| -71.0712|16.978994|1525319657| 36.77853|
+--------------------+----+----+---------+----------+---------+----------+---------+
only showing top 20 rows
I register it as a table
sqlContext.registerDataFrameAsTable(df, "myTable")
I would like to keep only the IDs that have at least one record everyday. I used the following query to count the DISTINCT ID
that I have every day.
query = """ SELECT DATE(FROM_UNIXTIME(timestamp)) AS ForDate,
COUNT(DISTINCT ID) AS NumPosts
FROM myTable
GROUP BY DATE(FROM_UNIXTIME(timestamp))
ORDER BY ForDate """
countIDDay = spark.sql(query)
I am trying this query
query = """ SELECT ID from myTable
GROUP BY ID
having count(DISTINCT DATE(FROM_UNIXTIME(timestamp))) = (SELECT COUNT(DISTINCT DATE(FROM_UNIXTIME(timestamp))) FROM myTable) """
You can aggregate and filter in the having
clause:
select id
from mytable
group by id
having count(distinct date(from_unixtime(timestamp))) =
(select count(distinct date(from_unixtime(timestamp)))
from mytable
);
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.