使用kafka从oracle迁移到mongodb数据

Question

i am trying to migrate data from oracle to mongodb using kafka. I took a sample record set of 10 million with column length of 90 each row is of 5Kb

i am dividing the data into 10 threads but one of the thread is not running every time.... when i check the data i see 1 million records are missing in mongodb.

main class:

    int totalRec = countNoOfRecordsToBeProcessed;
    int minRownum =0;
    int maxRownum =0;
    int recInThread=totalRec/10;
    System.out.println("oracle  "+new Date());
    for(int i=minRownum;i<=totalRec;i=i+recInThread+1){ 
          KafkaThread kth = new KafkaThread(i, i+recInThread, conn);
        Thread th = new Thread(kth);
        th.start();
    }
    System.out.println("oracle done+  "+new Date());    

kafka producer thread class:

JSONObject obj = new JSONObject();
while(rs.next()){

        int total_rows = rs.getMetaData().getColumnCount();
        for (int i = 0; i < total_rows; i++) {
            obj.put(rs.getMetaData().getColumnLabel(i + 1)
                    .toLowerCase(), rs.getObject(i + 1));
     }
            //System.out.println("object->"+serializedObject);
            producer.send(new ProducerRecord<String, String>("oracle_1",obj.toString()));
            obj= new JSONObject();
        //System.out.println(counter++);
}

consumer class:

        KafkaConsumer consumer = new KafkaConsumer<>(props);
        //subscribe to topic
        consumer.subscribe(Arrays.asList(topicName));

        MongoClientURI clientURI = new MongoClientURI(mongoURI);
        MongoClient mongoClient = new MongoClient(clientURI);
        MongoDatabase database = mongoClient.getDatabase(clientURI.getDatabase());
        final MongoCollection<Document> collection = database.getCollection(clientURI.getCollection());
        while (true) {
            final ConsumerRecords<Long, String> consumerRecords =
                    consumer.poll(10000);

            if (consumerRecords.count()!=0) {
                List<InsertOneModel> list1 = new ArrayList<>();
                consumerRecords.forEach(record -> {
//                    System.out.printf("Consumer Record:(%d, %s, %d, %d)\n",
//                            record.key(), record.value(),
//                            record.partition(), record.offset());'

                    String row =null;
                     row = record.value();
                     Document doc=Document.parse(row);
                     InsertOneModel t = new InsertOneModel<>(doc);
                     list1.add(t);
                });
                collection.bulkWrite((List<? extends WriteModel<? extends Document>>) (list1), new BulkWriteOptions().ordered(false));
                consumer.commitAsync();
                list1.clear();
            }
            }
    }

Answer 1

My advice: use Kafka Connect JDBC connector to pull the data in, and a Kafka Connect MongoDB sink to push the data out. Otherwise you are just reinventing the wheel. Kafka Connect is part of Apache Kafka.

Getting started with Kafka Connect:

• https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-1/
• https://www.confluent.io/blog/blogthe-simplest-useful-kafka-connect-data-pipeline-in-the-world-or-thereabouts-part-2/
• https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-3/