如何使用 Spark SQL 选择计数而不会出错？

Question

I am trying to do a very simple select statement to count the amount of iPod data that is null in my table in spark. My table looks like this

-----+------+------+------+----+-----+
| Time|Period|iPhone|  iPad|iPod|  Mac|
+-----+------+------+------+----+-----+
|Q4/98|     1|  null|  null|null|0.944|
...

The command:

apl_df.select("count(iPod) from apl_tbl where iPod is null")

Gives: org. apache.spark.sql.AnalysisException: cannot resolve '`count(iPod) from apl_tbl where iPod is null`' given input columns: [iPhone, iPod, Mac, Period, iPad, Time];;

And

apl_df.selectExpr("count(iPod) from apl_tbl where iPod is null")

Gives: org.apache.spark.sql.catalyst.parser.ParseException:

Please help me fix this issue and understand the meaning of these errors.

Answer 1

Try this:

apl_df.select("iPod").filter("iPod is null").count()

Or, if you want to use more familiar sql syntax, you can try

apl_df.createOrReplaceTempView("apl_tbl")

spark.sql("select count(iPod) from apl_tbl where iPod is Null")

See reference: https://spark.apache.org/docs/1.6.1/api/java/org/apache/spark/sql/DataFrame.html#selectExpr(scala.collection.Seq)