Spark Java分区

Question

Java spark2
Is there some difference in the two statements-

Dataset<Row> Data; 


Data.repartition(new Column("key" )) ; 

and 

Data.repartition(Data.col("key" ) ;

Answer 1

Doing

Data.repartition(new Column("key"));

is equivalent to

import static org.apache.spark.sql.functions.col;
Data.repartition(col("key"));

in these cases the column is not directly associated with a Dataset and Spark need to resolve it during the Analysis phase .

If you use instead

Data.repartition(Data.col("key");

you are giving an instruction to Spark on which is the Dataset to which the column belongs. This method is mainly useful in joins where you can have, for example, two dataset with a common column name.

In your example the result is equivalent, you can use both forms.