Spark Java：将可变数量的参数传递给函数

Question

Please see section " Programmatically Specifying the Schema ". Java section.

The example works. But I have a question about this particular code snippet.

JavaRDD<Row> rowRDD = people.map(
new Function<String, Row>() {
public Row call(String record) throws Exception {
String[] fields = record.split(",");
  return Row.create(fields[0], fields[1].trim());
}

The Row create method is being called with a static number of objects determined at compile time.

However, in my code, I need to call the Row.create method for a dynamic number of arguments.

I will only know the number of fields at run time

For example, it may be one of:

return Row.create(fields[0], fields[1].trim(), fields[2]);

or

return Row.create(fields[0]);

or

return Row.create(fields[0],fields[1].trim(), fields[2], fields[3],fields[4]);

How do I do it?

Answer 1

Here is how you can do it. Worked for me.

JavaRDD<Row> rowRDD = people.map(
  new Function<String, Row>() {
   public Row call(String record) throws Exception {
     String[] fields = record.split(",");         
    //return Row.create(fields[0], fields[1].trim());
      Object[] fields_converted = fields;
      return Row.create(fields_converted);
      }
      });

Answer 2

Try using elipsis in your implemented method as below.

public static void create(String ...arg) { ... }

Elipsis accept n number of arguments.

Answer 3

You can specify a method to take multiple arguments by using three dots after the argument, for example:

public static <return_type> create(String...args){
    // Yoo can now use the String[] args
}

Replace with your desired return type. Please change the signature of your call method as you have not specified a return type for it!

Answer 4

Here is what I did in the same situation

new Function<String, Row>(String s) {
    public Row call(String s){
        int n = /* width of actual schema */
        Object rec[] = new Object[n];
        for( int i = 0; i < n; ++i )
            rec[i] = /* Something that aligns with the type of #i field */
        return Row.create( rec );
    }
}

There might be dragons here. My version compiles, looks good, not tested yet.