优化PIG中的过滤器功能

Question

I have written a filter function for my pig script but my job is taking too much time. Amount of Data being processed is 15 GB on a 5 node cluster.

Can anybody suggest how to optimize my code:

     package org.apache.pig.builtin;
    import java.util.*;
    import java.io.IOException;
    import java.util.Map;
    import org.apache.pig.FilterFunc;
    import org.apache.pig.backend.executionengine.ExecException;
    import org.apache.pig.data.DataBag;
    import org.apache.pig.data.Tuple;
    import org.apache.pig.data.DataType;
    import org.apache.pig.impl.util.WrappedIOException;

   public class filterIP extends FilterFunc {
     ArrayList<String> Ar1=new ArrayList<String>(){
    {
add("151.193.220.28");
....
//Around 2000 IP's to be filtered
add("129.22.63.207");
    }
    };
       public Boolean exec(Tuple input) throws IOException {
           if (input == null || input.size() == 0)
               return true;
           try {
               Object values = input.get(0);
               if (values instanceof DataBag)
                   return ((DataBag)values).size() == 0;
               else if (values instanceof Map)
                   return ((Map)values).size() == 0;
           else if (values instanceof String){

            for(String s:Ar1){
            if(((String)values).matches(".*"+s+".*"))
            return false;
            }
            return true;
            //return !((String)values).matches(".*"+Ar1.get(1)+".*");
            }
               else{
            return false;
                  // throw new IOException("Cannot test a " + DataType.findTypeName(values) + " for required match.");
               }
           } catch (ExecException ee) {
               throw WrappedIOException.wrap("Caught exception processing input row ", ee);
           }
       }
   }*

Answer 1

Use a HashSet rather than an ArrayList - the for (String s:Ar1) is your bottleneck.

Are you trying to filter a field by a set of IP addresses?, Can you replace the matches() call with set.contains(values)? This will only work however if your value only contains the IP address, and no pre / post fix characters - but i'm sure you can write a regex to find the ip address string first, extract it and then check for membership against the hash set.