按列名及其数值过滤带有PySpark的spark RDD

Question

I am translating Scala / Spark model into Python / Spark. The problem is I have RDD with about 1 million observations and about 33 columns. I am splitting the RDD based on numerical threshold ('Time'). The time variable is in numerical format (double) (not posix).

Here is Scala source code:

// get the time to split the data.
val splitTime = data.stat.approxQuantile("Time", Array(0.7), 0.001).head

val trainingData = data.filter(s"Time<$splitTime").cache()
val validData = data.filter(s"Time>=$splitTime").cache()

and here is my PySpark failed interpretation:

splitTime = data.approxQuantile("Time", [0.7], 0.001)
trainingData = data.filter(data["Time"] < splitTime)
validData = data.filter(data["Time"] >= splitTime)

The first line works fine. The problem starts when I try to use the threshold on the RDD. I also couldn't decode the Scala format s" >=$ " around the condition and its importance in the condition. Internet source on the meaning of s" >=$ " are vague.

Answer 1

approxQuantile returns either List[float] (single column case like here) or List[List[float]] (multi columns case) so you have to extract the values:

splitTime = data.approxQuantile("Time", [0.7], 0.001)
data.filter(data["Time"] < splitTime[0])

or

(litTime, ) = data.approxQuantile("a", [0.7], 0.001)
trainingData = data.filter(data["Time"] < splitTime)