Couchbase - Bulk insert via Java

Question

Introduction

I am currently working on a Java project using Couchbase as a database. As part of it, I came to create a massive data insert via a json file and a csv file.

Question

And I was wondering, what is the best way to achieve this one? Is it possible to have a sample code?

Answer 1

CSV conversion:

Benchmark:

You have several libraries to perform the conversion between CSV <-> Java object, here is a benchmark of the different libraries:

Library	Read (rec/sec)	Write (rec/sec)	Dependencies	Size (KiB)
Commons CSV	1,128,102	3,354,703	no	50
FastCSV	4,738,726	5,034,953	no	31
Jackson CSV	3,770,602	3,995,294	yes	2,040
Java CSV	1,922,189	2,732,843	no	13
Opencsv	1,085,935	1,808,982	yes	2,625
Sfm+ASM	5,164,967	1,901,154	yes	1.498
Sfm-ASM	4,652,517	1,901,154	yes	1,498
Super CSV	1,406,090	1,730,984	no	96
Univocity	3,594,900	4,050,255	no	437

For more information on this benchmark you can visit this site: https://github.com/osiegmar/JavaCsvBenchmarkSuite#results

FastCSV:

Iterative reading of some CSV data with a header

NamedCsvReader.builder().build("header 1,header 2\nfield 1,field 2")
    .forEach(row -> row.getField("header 2"));

For more information about FastCSV you can visit this site: https://github.com/osiegmar/FastCSV

---

JSON conversion:

Benchmark:

You have several libraries to perform the conversion between Json <-> Java object, here is a benchmark of the different libraries:

For more information on this benchmark you can visit this site: https://github.com/ngs-doo/dsl-jsons

jackson-databind:

Conversion from json string to java object:

String json = "{\"name\":\"Hassan\",\"age\":23}";
Person person = new ObjectMapper().readValue(json, Person.class);

For more information about jackson-databind you can visit this site: https://github.com/FasterXML/jackson-databind

---

CouchBase insertion:

Example of Bulk Insert in Java:

protected void doWork() {
    final String key = "javaDevguideExampleBulkInsert";
    // Create a JSON document content
    final JsonObject content = JsonObject.create().put("item", "A bulk insert test value");
    // Describe what we want to do asynchronously using RxJava Observables:
    ReactiveCollection reactiveCollection = collection.reactive();
    Flux<MutationResult> resultFlux = Flux.range(0, 10)
            .map(index -> { return key + "_" + index; })
            .flatMap(k -> reactiveCollection.upsert(k, content));
    resultFlux.subscribe(System.out::println);
}

This code is from official docs-sdk-java

Answer 2

We've recently published a guide to importing which may also be useful: https://docs.couchbase.com/server/current/guides/import.html#importing-using-an-sdk