Google App Engine object exchange between instances

Question

I am running a GAE java application and using backend instance for some computation. After computation, backend instance creates a map which should be used by frontend while serving http requests.

Initially I created the static map and I updated the map values using cron job running on backend instance. But when I try to retrieve the value by sending http request, I still get old value. This is my code

public class ServerServlet extends HttpServlet {

public static Map<String,String> highQualityMap;

protected void doGet( HttpServletRequest request,
        HttpServletResponse response)
                throws ServletException, IOException {
    try{
        String uri = request.getRequestURI();
        PrintWriter out = response.getWriter();

        if(uri.equalsIgnoreCase("/getstatus")){
            String id = request.getParameter("id");
            out.write(highQualityMap.get(id));
        }
        else if(uri.equalsIgnoreCase("/recacheAll")){
            System.out.println("recache all");
            buildData();
        }
        if(out!=null)
            out.close();
    }
    catch(Exception e){
        e.printStackTrace();
    }
}

private void buildData(){
    // here after some processing, data is populated in highQualityMap
}

}

This is my cron.xml

<?xml version="1.0" encoding="UTF-8"?>
<cronentries>
  <cron>
    <url>/recacheAll</url>
    <description>Repopulate the cache every 3 hours</description>
    <schedule>every 3 hours</schedule>
    <target>backend</target>
  </cron>
</cronentries>

And this is my backends.xml

<?xml version="1.0" encoding="UTF-8"?>
<backends>
    <backend name="backend">
        <class>B1</class>
        <options>
            <dynamic>true</dynamic>
            <public>false</public>
        </options>
    </backend>
</backends>

I believe both frontend and backend instances run separately and do not share objects. When I run the cron job on frontend, it picks up the correct data. So I am looking for a way so that I can use the same computed map by backend instance in frontend.

Update1: After computing the hashmap on backend, I tried saving it in datastore, but I got an error stating "java.util.HashMap is not a supported property type". So instead of creating a map, I created list and tried storing it in datastore. But due to restriction on entity size (which is 1mb), I wasn't able to store the arraylist also in the datastore.

Update2: I converted my map into list, splitter it up into multiple smaller lists and stored them into different entities of datastore. After storing, I fired up a task which should get executed on frontend. This task reads the smaller lists from datastore, creates a big list out of them and finally creates a map which I am keeping in memory.

Answer 1

You have at least five options:

1. POST (I am not sure what the limit on App Engine is, but I would guess that up to 10MB should be fine).
2. Datastore (no limit).
3. Google Cloud Storage (no limit).
4. Dedicated memcache (up to 1MB).
5. Regular memcache backed by a datastore (up to 1MB).

The last option is probably the most cost-effective assuming your object fits in Memcache. Your backend instance creates a map, puts it in a datastore and puts it into the Memcache. Your front-end instance tries to load it from Memcache. If it's not available there, it loads it from the datastore and puts it into the Memcache in case it needs it again.

UPDATE:

While your map is large, you can still go with the last, most effective, option. On the backend:

ArrayList<Entity> batch = new ArrayList<Entity>(500);

for (String key : map.keySet()) {

    memcache.put(key, map.get(key));

    Entity entity = new Entity("Entry", key);
    entity.setUnindexedProperty("value", map.getKey());
    /* or, if a value can be longer than 500 characters
     * entity.setUnindexedProperty("value", new Text(map.getKey()));
     */

    batch.add(entity);
    if (batch.size() == 500) {
         datastore.put(batch);
         batch.clear();
    }
}
datastore.put(batch);

This will work if you don't need to know all keys on the front-end. If you do, then check if your keySet() is smaller than 1MB so you can put it in Memcache as an object. If it's larger, then skip the memcache part and simply use the datastore:

Query q = new Query("Entry");
// etc.