How do I determine the term frequency of the terms in each document?

Question

I'm building an inverted index, but I can't seem to get the correct frequencies when I check the database. I read everywhere that you should use a HashMap, but I'm not quite sure if this is the correct method of doing so. Any ideas?

public class Tokenize {

public static void createIndex() throws Exception{

    ArrayList<Dokument> dok = new QueryHandler().getDokuments();
    ArrayList<String> queries = new ArrayList<String>();
    ArrayList<String> queries2 = new ArrayList<String>();
    HashMap<String, Integer> frek = new HashMap<String, Integer>();

    for(int d = 0; d < dok.size(); d++){
        String token = "";
        int frekvens = 0;


        try{

            Dokument document = dok.get(d);
            StringTokenizer st = new StringTokenizer(document.dokument());
            while (st.hasMoreTokens()) {


                token = st.nextToken();
                token.replaceAll("[']", "");
                token.replaceAll("[,]", "");
                token.replaceAll("[)]", "");
                token.replaceAll("[(]", "");
                token.replaceAll("[.]", "");
                frekvens ++;
                frek.put(token, frekvens);


                    queries.add("INSERT IGNORE INTO termindeks (docID, term) values ("+document.docID()+", '"+token+"')");
                    queries2.add("INSERT IGNORE INTO invertedindeks (term, docID, termfrekvens) values ('"+token+"', "+document.docID()+", "+ frekvens+")");


            }
        }


        catch (Exception e) {
        e.printStackTrace();
        System.out.println(token);
        }
    }

    String[] ffs = new String[queries.size()];
    ffs = queries.toArray(ffs);
    getDB().runQueriesIgnoreException(queries.toArray(ffs));

    String[] ffs2 = new String[queries2.size()];
    ffs2 = queries2.toArray(ffs2);
    getDB().runQueriesIgnoreException(queries2.toArray(ffs2));

}

}

Answer 1

You should get the value for the token first, increment it and put it again.

Like this in your loop:

Integer frekvens = frek.get(token); //remove the other frekvens as it's not needed - or find a better name for this one ;)
if( frekvens == null ) { frekvens = 0 };
frekvens++;
frek.put(token, frekvens);

Answer 2

The idea is correct but as far as I see, you don't make correct use of the HashMap . You have to get the value associated with the key, ie

Integer i = map.get(token);
i += 1;
map.put(token, i);

EDIT

Another option would be to use an AtomicInteger instead of an Integer because AtomicInteger s are mutable.

Map<String, AtomicInteger> map = new HashMap<String, AtomicInteger>();    
map.get(token).getAndIncrement();