How to: write aerospike udf filter without map OR return full record in map

Question

I have a udf stream with filter and map in Aerospike.

If i map, as per all examples i have seen, i can pick fields from the record and return a new map with filtered and selected fields. However, I don't want to do that. I want to take whatever stream, with whatever columns/bins, apply a filter and return the full record. One approach might be to use something like stream : fiter(my_filter) and not use a map. Intuitively (to me at least) this would just filter and relay the stream. This doesnt seem to work, to my frustration. The next thing i try is to use a map but simply relay the full record. This doesnt work either. In both cases, when i say it doesnt work, i get an empty list as my result.

Can someone kindly explain how this is supposed to work. It is driving me absolutely nuts. Given that this is among the most basic things in the world one would like to do with udfs, i realise i am missing something obvious. I should point out, i have done lots of more complicated things with udfs but for some reason this is a problem for me.

Answer 1

The thing you are missing is that you cannot return record or stream types in a UDF return value. I believe all return types are mapped by aerospike system lua module to client specific types; it cannot map back a record "type".

If you absolutely want to get the record, store your key in a bin, return that bin in a map type or string type or integer type - whatever is most suitable type to your application. You may also be able to return the record digest from the record metadata in a map type. I have not tested retrieving and returning a records digest via a UDF but worth a try.

Once you have namespace, set and your key, or namespace & record digest, you can access the record from the client API. Record digest is RIPEMD160 hash computed from a combination of set name and your key.

Answer 2

A record in Aerospike is the tuple ( key , metadata , bins ). An Aerospike UDF written in Lua , whether it's a record UDF or a stream UDF , can only return one of the supported types - string, integer, double, list, map, bytes (See: Known Limitations ).

In a stream UDF, if you only have a filter you still need to cast the bin-name / bin-value pairs of the record to a map, and return that:

local function bins_match_filter(bin1, bin2)
  return function(rec)
    if rec[bin1] and rec[bin2] and
       (type(rec[bin1]) == type(rec[bin2])) and
       rec[bin1] == rec[bin2] then
      return true
    end
    return false
  end
end

local function record_to_map(rec)
  local ret = map()
  for i, bin_name in ipairs(record.bin_names(rec)) do
    ret[bin_name] = rec[bin_name]
  end
  return ret
end

function check_bins_match(stream, bin1, bin2)
  return stream : filter(bins_match_filter(bin1, bin2)) : map(record_to_map)
end

You may be able to translate certain stream UDF based filters into a predicate filter expression. It wouldn't work for the example above, because there's no way to compare the values of two bins. But for most cases the predicate expression operations are adequate (see the PredExp class of the Java client). You wouldn't need to invoke a UDF at all, which would run much faster, scale better, and you wouldn't need to cast the record into a map of bin name/value pairs.