如何创建数组或数据集集合<Row>火花数据帧类型？

Question

I am working in filtration of an avro file in spark using java. I am getting different Dataframes for different type of filtration conditions like (equalto,greater than, less than) as below:

df1 = sourceDf.filter(sourceDf.col(fieldName).equalTo(value)),
df2 = sourceDf.filter(sourceDf.col(fieldName).gt(value)),
df3 = sourceDf.filter(sourceDf.col(fieldName).lt(value)) and so on....

Now, i want to collect all dataframes(df1,df2,df3,...)` in one collection or array not individual ones as above. please let me know how can i achieve this as i am new in java and apache-spark.

i tried Dataset[] RecordCollection = new Dataset[3]; but it is not allowed.

The exception is: "can't create a generic array of dataset"

Answer 1

It's not clear what you're trying to accomplish since the examples you post filters nothing if combined. but still you can do:

union from the API:

Dataset<Row> df = df1.union(df2).union(df3)

or from the start filter using or :

Column c1 = sourceDf.col(fieldName).equalTo(value);
Column c2 = sourceDf.col(fieldName).gt(value);
Column c3 = sourceDf.col(fieldName).lt(value);
df1 = sourceDf.filter(c1.or(c2).or(c3))