Automate mongodb and elasticsearch sync

Question

I am currently working on a project where our main database is mongodb and for searching we use elasticsearch. We have inserted data in to mongodb by a java application. And we used river plugin to sync data. Up to now we have done syncing data between mongodb and elasticsearch manually by executing shellscript files mentioned below. (setup.sh && bash.sh)

//setup.sh
curl -XPOST http://localhost:9200/classdata -d @setup.json

//setup.json
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"classdata": {
  "properties": {
    "className": {
      "type": "string"
    },
"jarID": {
      "index": "not_analyzed",
      "type": "string"
    },
    "jarFileName": {
      "index": "not_analyzed",
      "type": "string"
    },
    "dependencies": {
      "properties": {
        "methodSignature": {
          "type": "string"
        },
        "dependedntClass": {
          "type": "string"
        }
      }
    }
  }
}
}
}

//bash.sh
curl -XPUT "localhost:9200/_river/classdata/_meta" -d '
{
"type": "mongodb",
"mongodb": {
"servers": [
  { "host": "127.0.0.1", "port": 27017 }
],
"options": { "secondary_read_preference": true },
"db": "E",
"collection": "ClassData"
},
"index": {
"name": "classdata",
"type": "classdata"
}
}'

But now our requirement has changed. Now we need to automate the process, like after inserting data in to mongodb we have to automatically sync data between elasticsearch and mongodb. I have no idea how to do that. If some one know how to automate this process please help me.

Answer 1

I strongly recommend you monstache . It runs in the background and automately sync data from Mongodb to Elasticsearch. And you can configure to specify which db and what kind of operation(insert, update, delete...) you want to sync, the configuration options listed in here

Answer 2

MongoConnector Plugin supports data sync between MongoDB and Elastic Search.

1) Install Mongo Connector in your server.

`pip install mongo-connector`

2) Install Doc Manager based on target system. There are various implementations for Doc Manager based on the Target system. Install the one that supports Elastic Search and in particular the version that you have. Eg)

 pip install 'mongo-connector[elastic5]'

3) Start Mongo Connector with configurations of the source(mongodb) and target systems. Eg)

mongo-connector -m <mongodb server hostname>:<replica set port> -t <replication endpoint URL, e.g. http://localhost:8983/solr> -d <name of doc manager>

Now data will be automatically synced up between the two systems.

For more information, use the following links,

https://www.mongodb.com/blog/post/introducing-mongo-connector

https://github.com/mongodb-labs/mongo-connector

https://github.com/mongodb-labs/mongo-connector/wiki/Usage%20with%20ElasticSearch