Flink：从行字段中提取数组

Question

I am using Flink, and I am using a custom function in a map. This custom function use the Flink Row type as input and output a Map of (String, Object) that contains each field and values of my row.

In basic case this function work well, but now I need to do some processing on a specific field, which is an array of integer. In this case, I have some trouble extracting the data in my Row to a Java Collection object (list or array or whatever).

Here is the code of my CustomMap :

public class CustomMap implements MapFunction<Row, Map<String, Object>> {

    private final String arrayField = "ArrayField";
    private final String[] fields = {"genTimestampMs", "some_field", "timestampMs", "some_field_2", "ArrayField"};

    public CustomMap(){}

    @Override
    public Map<String, Object> map(Row myRow) throws Exception {
        LOGGER.debug("Mapping the row "+myRow.toString());

        final Map<String, Object> m = new HashMap<>();

        for (int i = 0; i < myRow.getArity(); i++) {
            LOGGER.debug("  Field "+i);
            if (arrayField.equals(fields[i])) {
                LOGGER.debug("Is the field  "+arrayField);
                Integer wCount = 0;

                LOGGER.debug("  row0 : "+myRow.getField(i));
                Row test = Row.of(myRow.getField(i));
                LOGGER.debug("  row : "+test);
                LOGGER.debug("  getArity: "+test.getArity());

                List<Integer> myList = (List<Integer>)myRow.getField(i); // <--- Error here

                String value = // Do something with my list

                m.put(fields[i], value);

            } else {
                LOGGER.debug("  Put field in map : ("+fields[i]+" -> "+myRow.getField(i)+")");
                m.put(fields[i], myRow.getField(i));
            }
        }
        return m;
    }
}

Here is an example of a json I use as input data (it is parsed with Flink JsonRowDeserializationSchema ) :

{"genTimestampMs": 1561130625000, "some_field": "some_value", "timestampMs": 1561130625000, "some_field_2":"some_value", "ArrayField": [1,2,3,4,5]}

And here are the logs of my code execution on this data :

2019-06-27 13:40:02.854 [Source: Custom Source -> Map -> Map -> Sink: Unnamed (5/12)] DEBUG CustomMap  - Mapping the row 1561130625000,some_value,1561130625000,some_value,[1, 2, 3, 4, 5]
2019-06-27 13:40:02.854 [Source: Custom Source -> Map -> Map -> Sink: Unnamed (5/12)] DEBUG CustomMap  -   Field 0
2019-06-27 13:40:02.858 [Source: Custom Source -> Map -> Map -> Sink: Unnamed (5/12)] DEBUG CustomMap  -   Put field in map : (genTimestampMs -> 1561130625000)
2019-06-27 13:40:02.858 [Source: Custom Source -> Map -> Map -> Sink: Unnamed (5/12)] DEBUG CustomMap  -   Field 1
2019-06-27 13:40:02.858 [Source: Custom Source -> Map -> Map -> Sink: Unnamed (5/12)] DEBUG CustomMap  -   Put field in map : (some_field -> some_value)
2019-06-27 13:40:02.858 [Source: Custom Source -> Map -> Map -> Sink: Unnamed (5/12)] DEBUG CustomMap  -   Field 2
2019-06-27 13:40:02.858 [Source: Custom Source -> Map -> Map -> Sink: Unnamed (5/12)] DEBUG CustomMap  -   Put field in map : (timestampMs -> 1561130625000)
2019-06-27 13:40:02.858 [Source: Custom Source -> Map -> Map -> Sink: Unnamed (5/12)] DEBUG CustomMap  -   Field 3
2019-06-27 13:40:02.858 [Source: Custom Source -> Map -> Map -> Sink: Unnamed (5/12)] DEBUG CustomMap  -   Put field in map : (some_field_2 -> some_value)
2019-06-27 13:40:02.858 [Source: Custom Source -> Map -> Map -> Sink: Unnamed (5/12)] DEBUG CustomMap  -   Field 4
2019-06-27 13:40:02.858 [Source: Custom Source -> Map -> Map -> Sink: Unnamed (5/12)] DEBUG CustomMap  -   Is the field  ArrayField
2019-06-27 13:40:02.858 [Source: Custom Source -> Map -> Map -> Sink: Unnamed (5/12)] DEBUG CustomMap  -   row0 : [Ljava.lang.Integer;@68374747
2019-06-27 13:40:02.859 [Source: Custom Source -> Map -> Map -> Sink: Unnamed (5/12)] DEBUG CustomMap  -   row : [1, 2, 3, 4, 5]
2019-06-27 13:40:02.859 [Source: Custom Source -> Map -> Map -> Sink: Unnamed (5/12)] DEBUG CustomMap  -   getArity: 1 
java.lang.ClassCastException: class [Ljava.lang.Integer; cannot be cast to class java.util.List ([Ljava.lang.Integer; and java.util.List are in module java.base of loader 'bootstrap')

Note :

• Trying to parse the variable test does not work either : java.lang.ClassCastException: class org.apache.flink.types.Row cannot be cast to class java.util.List (org.apache.flink.types.Row is in unnamed module of loader 'app'; java.util.List is in module java.base of loader 'bootstrap')

Answer 1

The following cast is wrong:

List<Integer> myList = (List<Integer>)myRow.getField(i); // <--- Error here

This is a cast of myRow to List<Integer> , then an attempt to call getField() on that List .

Try:

List<Integer> myList = (List<Integer>)(myRow.getField(i));

which is to say, perform getField() on myRow (which will return Object ), then cast that Object to List<Integer> .

EDIT: I think the issue is that the object is an array of Integers, not a List. Try the following:

List<Integer> myList = Arrays.asList((Integer[])(myRow.getField(i)));