Unexpected output of InfluxDB batch write

Question

I am using batch processing to write into InfluxDB and below is my code for doing that.

    String dbName = "test";
    influxDB.query(new Query("CREATE DATABASE " + dbName, dbName));
    Stopwatch watch = Stopwatch.createStarted();
    influxDB.enableBatch(2000, 100, TimeUnit.MILLISECONDS);


        for (int j = 0; j < 100000; j++) {
            Point point = Point.measurement("cpu")
                    .addField("idle", (double) j)
                     .addField("system", 3.0 * j).build();
            influxDB.write(dbName, "autogen", point);
        }
       influxDB.disableBatch();
       System.out.println("Write for " + 100000 + " Points took:" + watch);
   }

Here i am writing 100000 points and which is taking very reasonable time to write, however only few records are written into DB instead of expected 100000 records.

select count(idle) from cpu gives me only "89" i am expecting it to be "100000"

While select * from cpu gives me following:

cpu
time                        idle    system
2016-10-06T23:57:41.184Z    8       24
2016-10-06T23:57:41.185Z    196     588
2016-10-06T23:57:41.186Z    436     1308
2016-10-06T23:57:41.187Z    660     1980
2016-10-06T23:57:41.188Z    916     2748
2016-10-06T23:57:41.189Z    1278    3834
2016-10-06T23:57:41.19Z     1405    4215
2016-10-06T23:57:41.191Z    1409    4227
2016-10-06T23:57:41.192Z    1802    5406
2016-10-06T23:57:41.193Z    1999    5997
2016-10-06T23:57:41.456Z    3757    11271
2016-10-06T23:57:41.457Z    3999    11997
2016-10-06T23:57:41.858Z    4826    14478 and so on.....

Here my question is why the values of idle are missing, for example, after 8 it should 9, 10, 11, and so on but these values were not persisted and comes directly 196 and then missing in between and then 436. Any idea how to persist all value of loop variable "j" in this situation?

Answer 1

This line

influxDB.enableBatch(2000, 100, TimeUnit.MILLISECONDS);

says that it will flush input data if there are more than 2000 samples per 100 ms period. Since you are trying to write 100k samples then logically most of them get flushed.

Instead, write less samples in a single batch. My recommendation would be to write 5000 samples in a single batch, and make multiple batches until all your data is in the db.

// Batch 1
influxDB.enableBatch(5000, 100, TimeUnit.MILLISECONDS);
for (int j = 0; j < 5000; j++) {
   Point point = Point.measurement("cpu")
                      .addField("idle", (double) j)
                      .addField("system", 3.0 * j).build();
   influxDB.write(dbName, "autogen", point);
}
influxDB.disableBatch();

// Batch 2
// ...