Large data retrieval from IBM Domino

Question

I am trying to collect account data and group data from IBM Domino v9.0.

The code that I wrote for collecting data used the lotus.domino.View API. The code works fine for a small data set, I ran the code for 40 users and it was fine. When I ran the code to extract the data for 10k users , it takes roughly around 3-4 hrs to extract the data, which is not acceptable.

Can you please suggest me some other API with which we can retrieve data at much faster rate?

The code that i am currently using is as follows:

private static void testUserDataRetrieval(String host, String userName, String password) throws Exception{
    Session s = NotesFactory.createSession(host, userName, password);
    Database db = s.getDatabase(s.getServerName(), "names");
    File outFile = new File("C:\\Users\\vishva\\Desktop\\dominoExtract.txt");
    FileWriter writer = new FileWriter(outFile);
    View view = db.getView("($Users)");
    ViewEntryCollection entryCollection = view.getAllEntries();
    writer.write("\n--------------------------------- Printing data for view:"+view.getName()+"("+view.getEntryCount()+")--------------------------------------------");
    Vector<Object> columnNames = view.getColumnNames();
    for(int i = 0; i< entryCollection.getCount(); i++){
        ViewEntry entry = entryCollection.getNextEntry();
        if(entry == null) continue;
        Vector<Object> colValues = entry.getColumnValues();
        for(int j = 0; j < columnNames.size(); j++)
            writer.write("\n"+columnNames.get(j)+":"+colValues.get(j));
        writer.write("\n*****************************************");
    }
    writer.flush();
    writer.close();
}

Please let me know which other API should I use to increase the speed for data retrieval?

Answer 1

First of all: You read the count of the collection in every run of your for- loop. That makes it very slow. The for- loop is not necessary at all. Second: You never recycle the objects. That is MANDATORY when working with domino objects, as otherwise your code will eat up the memory faster than you can think:

private static void testUserDataRetrieval(String host, String userName, String password) throws Exception{
  Session s = NotesFactory.createSession(host, userName, password);
  Database db = s.getDatabase(s.getServerName(), "names");
  File outFile = new File("C:\\Users\\vishva\\Desktop\\dominoExtract.txt");
  FileWriter writer = new FileWriter(outFile);
  View view = db.getView("($Users)");
  ViewEntryCollection entryCollection = view.getAllEntries();
  writer.write("\n--------------------------------- Printing data for view:"+view.getName()+"("+view.getEntryCount()+")--------------------------------------------");
  Vector<Object> columnNames = view.getColumnNames();
  ViewEntry entry = entryCollection.getFirstEntry();
  ViewEntry entryNext;
  while(entry != null) {
    Vector<Object> colValues = entry.getColumnValues();
    for(int j = 0; j < columnNames.size(); j++)
        writer.write("\n"+columnNames.get(j)+":"+colValues.get(j));
    writer.write("\n*****************************************");
    entryNext = entryCollection.getNextEntry();
    entry.recycle();
    entry = entryNext ;
  }

writer.flush();
writer.close();

}

Answer 2

By default, your View object has the isAutoUpdate property set to true. That means it is synchronizing with any changes that are occurring on the back end instead of just giving you a one-time snapshot. Add this:

view.setAutoUpdate(false);

Also, you should probably investigate using the ViewNavigator class. See here for an article detailing performance improvements in this class as of Domino 8.5.2. It's written by the lead developer for the classes. Also see this blog post for a lot of interesting detail.

Answer 3

Thanks Everyone for their input. For retrieving the users i tried out the LDAP API later on, and it worked like charm. I was able to retrieve 10k accounts in 23 seconds , much acceptable. The code that i used was as follows

    private void extractDominoDataViaLdap(){
    String [] args = new String[]{"192.168.21.156","Administrator","password","","C:\\Users\\vishva\\Desktop"};
    String server = args[0];
    String userName = args[1];
    String password = args[2];
    String baseDN = args[3];
    String fileLocation = args[4];
    // Set up environment for creating initial context
    Hashtable<String, Object> env = new Hashtable<String, Object>(11);
    env.put(Context.INITIAL_CONTEXT_FACTORY,"com.sun.jndi.ldap.LdapCtxFactory");
    env.put(Context.PROVIDER_URL, "ldap://"+server+":389");

    long time = System.currentTimeMillis();
    // Authenticate as S. User and password "mysecret"
    env.put(Context.SECURITY_AUTHENTICATION, "simple");
    env.put(Context.SECURITY_PRINCIPAL, userName);
    env.put(Context.SECURITY_CREDENTIALS, password);
    FileWriter writer = null;
    BufferedWriter out = null;
    try {
        DirContext ctx = new InitialDirContext(env);
        //fetching user data
        // Create the search controls
        SearchControls searchCtls = new SearchControls();
        // Specify the attributes to return
        String returnedAtts[] = {"FullName","dn","displayname","givenName","sn","location","mail","mailfile","mailserver"};
        searchCtls.setReturningAttributes(returnedAtts);
        // Specify the search scope
        searchCtls.setSearchScope(SearchControls.SUBTREE_SCOPE);
        searchCtls.setCountLimit(0);
        // specify the LDAP search filter
        String searchFilter = "objectClass=inetOrgPerson";

        // Specify the Base for the search
        String searchBase = baseDN;

        // Search for objects using the filter
        NamingEnumeration<SearchResult> answer = ctx.search(searchBase,searchFilter, searchCtls);

        writer = new FileWriter(fileLocation+"\\users.csv");
        out = new BufferedWriter(writer);
        for(String attr : returnedAtts)
            out.write(attr+",");
        out.write("\n");
        int count = 0;
        // Loop through the search results
        while (answer.hasMoreElements()) {
            ++count;
            SearchResult sr = (SearchResult) answer.next();
            Attributes attrs = sr.getAttributes();
            StringBuilder sb = new StringBuilder();
            for(String attr : returnedAtts)
                sb.append("\""+((attrs.get(attr) != null)?attrs.get(attr).get():"")+"\"").append(",");
            out.write(sb.toString()+"\n");
            out.flush();
        }
        System.out.println("# of users returned: "+count);
        out.close();
        writer.close();
    }
}