从 DataFrame (Java/Spark) 中选择指定的列

Question

I have the following problem:

I've a schema(1) and a DataFrame with an other schema(2). The DataFrames schema has only one difference to the other schema, it has one more column. Now, I want to select columns from the DataFrame, which were specified in schema(1).

Example:

StructType schema; //specified in constructor
DataFrame df_old; //given as parameter
DataFrame df_new = df_old.select(schema.fieldNames());

This won't work, because select() needs two parameter and only one is given. So, my idea was:

StructType schema; //specified in constructor
DataFrame df_old; //given as parameter

String[] columns = schema.fieldNames(); //get column names as string array
String   first_col = columns[0]; // get first element of string array
columns = Arrays.copyOfRange(columns, 1, columns.length); //remove first element

DataFrame df_new = df_old.select(first_col,columns);

I think, that this is not the best way, because copyOfRange() would cost a lot of time. Especially, if there are a lot of columns and I need to run this multiply times.

Does somebody has a better idea?

Thanks for your answers. :)

Answer 1

//try this 
StructType schema; //specified in constructor
DataFrame df_old; //given as parameter

String[] columns = schema.fieldNames(); 
Column[] colList = new Column[columns.length];
for(int i=0; i<columns.length; i++){
    colList[I]=columns[i];`enter code here`
}

DataFrame df_new = df_old.select(colList);