How to estimate if the JVM has enough free memory for a particular data structure?

Question

I have the following situation: there are a couple of machines forming a cluster. Clients can load data-sets and we need to select the node on which the dataset will be loaded and refuse to load / avoid an OOM error if there is no one machine which could fit the dataset.

What we do currently: we now the entry count in the dataset and estimate the memory to be used as entry count * empirical factor (determined manually). Then check if this is lower than free memory (got by Runtime.freeMemory() ) and if so, load it (otherwise redo the process on other nodes / report that there is no free capacity).

The problems with this approach are:

• the empirical factor needs to be revisited and updated manually
• freeMemory sometimes may underreport because of some non-cleaned-up garbage (which could be avoided by running System.gc before each such call, however that would slow down the sever and also potentially lead to premature promotion)
• an alternative would be to "just try to load the dataset" (and back out if an OOM is thrown) however once an OOM is thrown, you potentially corrupted other threads running in the same JVM and there is no graceful way of recovering from it.

Are there better solutions to this problem?

Answer 1

The empirical factor can be calculated as build step and placed in a properties file.

While freeMemory() is almost always less than the amount which would be free after a GC, you can check it to see if it is available and call a System.gc() if the maxMemory() indicates there might be plenty.

NOTE: Using System.gc() in production only makes in very rare situations and in general it often incorrectly used resulting in a reduction in performance and obscuring the real problem.

I would avoid triggering an OOME unless you are running is a JVM you can restart as required.

Answer 2

My solution:

1. Set the Xmx as 90%-95% of RAM of physical machine if no other process is running except your program. For 32 GB RAM machine, set Xmx as 27MB - 28MB .

2. Use one of good gc algorithms - CMS or G1GC and fine tune relevant parameters. I prefer G1GC if you need more than 4 GB RAM for your application . Refer to this question if you chose G1GC:

   Agressive garbage collector strategy

   Reducing JVM pause time > 1 second using UseConcMarkSweepGC

3. Calculate Cap on memory usage by yourself instead of checking free memory. Add used memory and memory to be allocated. Subtract it from your own cap like 90% of Xmx . If you still have available memory, grant memory allocation request.

Answer 3

An alternative approach is to isolate each data-load in its own JVM. You just predefine each JVM's max-heap-size and so on, and set the number of JVMs per host in such a way that each JVM can take up its full max-heap-size. This will use a bit more resources — it means you can't make use of every last byte of memory by cramming in more low-memory data-loads — but it massively simplifies the problem (and reduces the risk of getting it wrong), it makes it feasible to tell when/whether you need to add new hosts, and most importantly, it reduces the impact that any one client can have on all other clients.

With this approach, a given JVM is either "busy" or "available".

After any given data-load completes, the relevant JVM can either declare itself available for a new data-load, or it can just close. (Either way, you'll want to have a separate process to monitor the JVMs and make sure that the right number are always running.)

Answer 4

an alternative would be to "just try to load the dataset" (and back out if an OOM is thrown) however once an OOM is thrown, you potentially corrupted other threads running in the same JVM and there is no graceful way of recovering from it.

There isn't good ways to handle and recover from OOME in JVM but there is way to react before OOM happens. Java has java.lang.ref.SoftReference which is guaranteed to have been cleared before the virtual machine throws an OutOfMemoryError . This fact can be used for early prediction of OOM. For example data load can be aborted if prediction triggered.

    ReferenceQueue<Object> q = new ReferenceQueue<>();
    SoftReference<Object> reference = new SoftReference<>(new Object(), q);
    q.remove();
    // reference removed - stop data load immediately

Sensitivity can be tuned with -XX:SoftRefLRUPolicyMSPerMB flag (for Oracle JVM). Solution not ideal, it effectiveness depends on various factors - do other soft references used in code, how GC tuned, JVM version, weather on Mars... But it can help if you lucky.

Answer 5

As you rightly noted, using freeMemory will not tell you the amount of memory that can be freed by Java Garbage Collection. You could run load tests and understand the JVM heap usage pattern and memory allocation, de-allocation pattern using tools like JConsole, VisualVM, jstat and printGCStats option to JVM. This will give an idea about calculating the empirical factor more accurately, basically understand what is the load pattern your java application can handle. Next would be to do choose the right GC and tune basic GC settings for better efficiency. This is not a quick solution, but maybe in the long-term a better solution.

The other way to be kill your JVM with -XX:OnOutOfMemoryError="kill -9 %p" JVM setting, once OOM happens, and then write, resue a simple process monitoring script to bring up your JVM if it is down.

Answer 6

Clients can load data-sets and we need to select the node on which the dataset will be loaded and refuse to load / avoid an OOM error if there is no one machine which could fit the dataset.

This is a job scheduling problem ie I'll get the issue near the end. 问题。

We have one of the main factors ie RAM but solutions to scheduling problems are dependent on many factors ie...

1. Are the jobs small or large ie are there hundreds/thousands of these running on a node or two or three. Think Linux scheduler.

2. Do they need to complete in a particular time frame? Realtime Scheduler.

Given everything we know at the start of a job can we predict when a job will end with within some time frame? If we can predict that on Node X we free up 100MB every 15 - 20 seconds we have a way to schedule a 200Mb job on that node ie I'm confident that in 40 seconds I'll have completed 200Mb of space on that node and the 40 seconds is an acceptable limit for the person or machine submitting the job.

Lets assume that we have a function as follows.

predicted_time predict(long bytes[, factors]); 

The factors are the other things we would need to take into consideration that I mentioned above and for every application there will be things you can add to suit your scenario.

The factors would be given weights when calculating predicted_time .

predicted_time is the number of milliseconds (can be any TimeUnit) that this node believes from now that it can service this Task, the node giving you the smallest number is the node the job should be scheduled on. You could then use this function as follows where we have a queue of tasks ie, in the following code this.nodes[i] represents a JVM instance.

private void scheduleTask() {
  while(WorkEvent()) {
        while(!this.queue.isEmpty()) {
            Task t = this.queue.poll();
            for (int i = 0; i < this.maxNodes; i++) {
                long predicted_time = this.nodes[i].predict(t);
                if (predicted_time < 0) {
                    boolean b = this.queue.offer(t);
                    assert(b);
                    break;
                }
                if (predicted_time <= USER_EXPERIENCE_DELAY) {
                    this.nodes[i].addTask(t);
                    break;
                }
                alert_user(boolean b = this.queue.offer(t);
                assert(b);
            }
        }
    }
}

If predicted_time < 0 we have an error, we reschedule the job, in reality we'd like to know why but that's not difficult to add. If the predicted_time <= USER_EXPERIENCE_DELAY the job can be scheduled.

We can gather any statistics we want from our scheduler ie how many jobs of size X where scheduled correctly, the aim would be to reduce the errors and to make it more reliable over time ie reduce the amount of times we tell a customer that their job cannot be serviced. What we've done is reduce the problem to something we can statistically improve towards an optimal solution.

Answer 7

Clients can load data-sets and we need to select the node on which the dataset will be loaded and refuse to load / avoid an OOM error if there is no one machine which could fit the dataset.

This is a job scheduling problem ie I'll get the issue near the end. 问题。

We have one of the main factors ie RAM but solutions to scheduling problems are dependent on many factors ie...

1. Are the jobs small or large ie are there hundreds/thousands of these running on a node or two or three. Think Linux scheduler.

2. Do they need to complete in a particular time frame? Realtime Scheduler.

Given everything we know at the start of a job can we predict when a job will end within some time frame? If we can predict that on Node X we free up 100MB every 15 - 20 seconds we have a way to schedule a 200Mb job on that node ie I'm confident that in 40 seconds I'll have completed 200Mb of space on that node and the 40 seconds is an acceptable limit for the person or machine submitting the job.

Lets assume that we have a function as follows.

predicted_time predict(long bytes[, factors]); 

The factors are the other things we would need to take into consideration that I mentioned above and for every application there will be things you can add to suit your scenario ie how many factors is up to you.

The factors would be given weights when calculating predicted_time .

predicted_time is the number of milliseconds (can be any TimeUnit) that this node believes from now that it can service this Task, the node giving you the smallest number is the node the job should be scheduled on. You could then use this function as follows where we have a queue of tasks ie, in the following code this.nodes[i] represents a JVM instance.

private void scheduleTask() {
  while(WorkEvent()) {
        while(!this.queue.isEmpty()) {
            Task t = this.queue.poll();
            for (int i = 0; i < this.maxNodes; i++) {
                long predicted_time = this.nodes[i].predict(t);
                if (predicted_time < 0) {
                    boolean b = this.queue.offer(t);
                    assert(b);
                    break;
                }
                if (predicted_time <= USER_EXPERIENCE_DELAY) {
                    this.nodes[i].addTask(t);
                    break;
                }
                alert_user(boolean b = this.queue.offer(t);
                assert(b);
            }
        }
    }
}

If predicted_time < 0 we have an error, we reschedule the job, in reality we'd like to know why but that's not difficult to add. If the predicted_time <= USER_EXPERIENCE_DELAY the job can be scheduled.

We can gather any statistics we want from our scheduler ie how many jobs of size X where scheduled correctly, the aim would be to reduce the errors and to make it more reliable over time ie reduce the amount of times we tell a customer that their job cannot be serviced or it failed. What we've done or at least are trying to attempt is to reduce the problem to something we can statistically improve upon towards an optimal solution.