Apache Spark isEmpty为false，但集合为空

Question

I am experiencing problems with Apache Spark when I call isEmpty on JavaRDD collection it returns false even though collection is empty.

Here's sample code (modified as it's from my final year project and I'm not allowed to publish any code):

sampleRdd = inputRdd.filter(someFilterFunction)
if(sampleRdd.isEmpty()) {
       return inputRdd.first();
} else {
        return sampleRdd.first(); // JVM points error on this line
}

Problem is sometimes condition is false so sampleRdd.isEmpty() returns false meaning it's not empty therefore execution proceeds to return statement where it's trying to retrieve first() element of that collection but it throws exception:

Exception in thread "main" java.lang.UnsupportedOperationException: empty collection
at org.apache.spark.rdd.RDD$$anonfun$first$1.apply(RDD.scala:1314)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
at org.apache.spark.rdd.RDD.first(RDD.scala:1311)
at org.apache.spark.api.java.JavaRDDLike$class.first(JavaRDDLike.scala:510)
at org.apache.spark.api.java.AbstractJavaRDDLike.first(JavaRDDLike.scala:47)
.
.
.

Is there a bit I am missing? I'm currently running it on local machine as it's still not fully developed.

Thanks

EDIT: To add more info JVM points to line sampleRdd.first() when I get this error so initial inputRdd is not empty

EDIT2: I wrote some extra lines that print size of inputRDD before filter and sampleRDD after filter like that:

System.out.println(inputRdd.count());  // Returns nonzero possitive int eg.100
sampleRdd = inputRdd.filter(someFilterFunction)
System.out.println(sampleRdd.count()); // Returns int eg. 1 
System.out.println(sampleRdd.count()); // Sometimes returns different int than first call eg.3
if(sampleRdd.isEmpty()) {
       return inputRdd.first();
} else {
        return sampleRdd.first(); // JVM points error on this line
}

And I observed very interesting behaviour and that is that sometimes inputRdd.count() returns 100 but first sampleRdd.count() returns 1 and second sampleRdd.count() returns 3 or basically different number from the first call. So basically looks like size of sampleRdd changes between two calls and therefore I assume sometimes it might change to after passing condition and trying to call first() returns error.

Any idea what might be causing that?

Answer 1

What if inputRdd is originally empty?

In that case, sampleRdd is also empty. Therefore samplerdd.isEmpty evaluates to true and inputRdd.first() throws the Exception.