deploy fiware-orion in specific mongodb setup

Question

I use a mongodb sharded and replicated with docker. ( 3 virtual machines hosting 15 containers). My Mongodb cluster is fully functionnal.

Now I try to connect Orion CB on it. Normaly I have to connect it to Mongo routers, but it doesn't works.

Here is my stack file for MongoDB

version: "3.4"
services:

  db1-1:
    deploy:
      placement:
        constraints: [node.hostname == docker-vm1]
    image: mongo:3.6.2
    hostname: db1-1
    environment:
      - MONGODB_REPLICA_SET_MODE=primary
    ports:
      - "27011:27011"
    volumes:
      - /var/lib/docker/config/STC-mongo1/data/db1-1:/data/db
      - /var/lib/docker/config/STC-mongo1/data/certificats:/etc/ssl/iot-certs
      - /var/lib/docker/config/STC-mongo1/config:/etc/config
    command: mongod --replSet iot-rs1 --shardsvr --dbpath /data/db --port 27011
    networks:
      STC-nw-mongo1:
        aliases:
          - db1-1

  db1-2:
    deploy:
      placement:
        constraints: [node.hostname == docker-vm2]
    image: mongo:3.6.2
    hostname: db1-2
    environment:
      - MONGODB_REPLICA_SET_MODE=secondary

    ports:
      - "27012:27012"
    volumes:
      - /var/lib/docker/config/STC-mongo1/data/db1-2:/data/db
      - /var/lib/docker/config/STC-mongo1/data/certificats:/etc/ssl/iot-certs
      - /var/lib/docker/config/STC-mongo1/config:/etc/config
    command: mongod --replSet iot-rs1 --shardsvr --dbpath /data/db --port 27012
    networks:
      STC-nw-mongo1:
        aliases:
          - db1-2

  db1-3:
    deploy:
      placement:
        constraints: [node.hostname == docker-vm3]
    image: mongo:3.6.2
    hostname: db1-3
    environment:
      - MONGODB_REPLICA_SET_MODE=secondary
    ports:
      - "27013:27013"
    volumes:
      - /var/lib/docker/config/STC-mongo1/data/db1-3:/data/db
      - /var/lib/docker/config/STC-mongo1/data/certificats:/etc/ssl/iot-certs
      - /var/lib/docker/config/STC-mongo1/config:/etc/config
    command: mongod --replSet iot-rs1 --shardsvr --dbpath /data/db --port 27013
    networks:
      STC-nw-mongo1:
        aliases:
          - db1-3

  db2-1:
    deploy:
      placement:
        constraints: [node.hostname == docker-vm1]
    image: mongo:3.6.2
    hostname: db2-1
    environment:
      - MONGODB_REPLICA_SET_MODE=primary
    ports:
      - "27021:27021"
    volumes:
      - /var/lib/docker/config/STC-mongo1/data/db2-1:/data/db
      - /var/lib/docker/config/STC-mongo1/data/certificats:/etc/ssl/iot-certs
      - /var/lib/docker/config/STC-mongo1/config:/etc/config
    command: mongod --replSet iot-rs2 --shardsvr --dbpath /data/db --port 27021
    networks:
      STC-nw-mongo1:
        aliases:
          - db2-1

  db2-2:
    deploy:
      placement:
        constraints: [node.hostname == docker-vm2]
    image: mongo:3.6.2
    hostname: db2-2
    environment:
      - MONGODB_REPLICA_SET_MODE=secondary
    ports:
      - "27022:27022"
    volumes:
      - /var/lib/docker/config/STC-mongo1/data/db2-2:/data/db
      - /var/lib/docker/config/STC-mongo1/data/certificats:/etc/ssl/iot-certs
      - /var/lib/docker/config/STC-mongo1/config:/etc/config
    command: mongod --replSet iot-rs2 --shardsvr --dbpath /data/db --port 27022
    networks:
      STC-nw-mongo1:
        aliases:
          - db2-2

  db2-3:
    deploy:
      placement:
        constraints: [node.hostname == docker-vm3]
    image: mongo:3.6.2
    hostname: db2-3
    environment:
      - MONGODB_REPLICA_SET_MODE=secondary
    ports:
      - "27023:27023"
    volumes:
      - /var/lib/docker/config/STC-mongo1/data/db2-3:/data/db
      - /var/lib/docker/config/STC-mongo1/data/certificats:/etc/ssl/iot-certs
      - /var/lib/docker/config/STC-mongo1/config:/etc/config
    command: mongod --replSet iot-rs2 --shardsvr --dbpath /data/db --port 27023
    networks:
      STC-nw-mongo1:
        aliases:
          - db2-3

  db3-1:
    deploy:
      placement:
        constraints: [node.hostname == docker-vm1]
    image: mongo:3.6.2
    hostname: db3-1
    environment:
      - MONGODB_REPLICA_SET_MODE=secondary
    ports:
      - "27031:27031"
    volumes:
      - /var/lib/docker/config/STC-mongo1/data/db3-1:/data/db
      - /var/lib/docker/config/STC-mongo1/data/certificats:/etc/ssl/iot-certs
      - /var/lib/docker/config/STC-mongo1/config:/etc/config
    command: mongod --replSet iot-rs3 --shardsvr --dbpath /data/db --port 27031
    networks:
      STC-nw-mongo1:
        aliases:
          - db3-1

  db3-2:
    deploy:
      placement:
        constraints: [node.hostname == docker-vm2]
    image: mongo:3.6.2
    hostname: db3-2
    environment:
      - MONGODB_REPLICA_SET_MODE=secondary
    ports:
      - "27032:27032"
    volumes:
      - /var/lib/docker/config/STC-mongo1/data/db3-2:/data/db
      - /var/lib/docker/config/STC-mongo1/data/certificats:/etc/ssl/iot-certs
      - /var/lib/docker/config/STC-mongo1/config:/etc/config
    command: mongod --replSet iot-rs3 --shardsvr --dbpath /data/db --port 27032
    networks:
      STC-nw-mongo1:
        aliases:
          - db3-2

  db3-3:
    deploy:
      placement:
        constraints: [node.hostname == docker-vm3]
    image: mongo:3.6.2
    hostname: db3-3
    environment:
      - MONGODB_REPLICA_SET_MODE=secondary
    ports:
      - "27033:27033"
    volumes:
      - /var/lib/docker/config/STC-mongo1/data/db3-3:/data/db
      - /var/lib/docker/config/STC-mongo1/data/certificats:/etc/ssl/iot-certs
      - /var/lib/docker/config/STC-mongo1/config:/etc/config
    command: mongod --replSet iot-rs3 --shardsvr --dbpath /data/db --port 27033
    networks:
      STC-nw-mongo1:
        aliases:
          - db3-3

  dbconfig1:
    deploy:
      placement:
        constraints: [node.hostname == docker-vm1]
    image: mongo:3.6.2
    hostname: dbconfig1
    environment:
      - MONGODB_REPLICA_SET_MODE=primary
    ports:
      - "37001:37001"
    volumes:
      - /var/lib/docker/config/STC-mongo1/data/dbconfig1:/data/configdb
      - /var/lib/docker/config/STC-mongo1/data/certificats:/etc/ssl/iot-certs
      - /var/lib/docker/config/STC-mongo1/config:/etc/config
    command: mongod --replSet conf-serv --configsvr --port 37001
    networks:
      STC-nw-mongo1:
        aliases:
          - dbconfig1

  dbconfig2:
    deploy:
      placement:
        constraints: [node.hostname == docker-vm2]
    image: mongo:3.6.2
    hostname: dbcoonfig2
    environment:
      - MONGODB_REPLICA_SET_MODE=secondary
    ports:
      - "37002:37002"
    volumes:
      - /var/lib/docker/config/STC-mongo1/data/dbconfig2:/data/configdb
      - /var/lib/docker/config/STC-mongo1/data/certificats:/etc/ssl/iot-certs
      - /var/lib/docker/config/STC-mongo1/config:/etc/config
    command: mongod --replSet conf-serv --configsvr --port 37002
    networks:
      STC-nw-mongo1:
        aliases:
          - dbconfig2

  dbconfig3:
    deploy:
      placement:
        constraints: [node.hostname == docker-vm3]
    image: mongo:3.6.2
    hostname: dbconfig3
    environment:
      - MONGODB_REPLICA_SET_MODE=secondary
    ports:
      - "37003:37003"
    volumes:
      - /var/lib/docker/config/STC-mongo1/data/dbconfig3:/data/configdb
      - /var/lib/docker/config/STC-mongo1/config:/etc/config
    command: mongod --replSet conf-serv --configsvr --port 37003
    networks:
      STC-nw-mongo1:
        aliases:
          - dbconfig3

  dbrouter1:
    deploy:
      placement:
        constraints: [node.hostname == docker-vm1]
    image: mongo:3.6.2
    hostname: dbrouter1
    ports:
      - "27997:27997"
    volumes:
      - /var/lib/docker/config/STC-mongo1/data/dbrouter1:/data/db
      - /var/lib/docker/config/STC-mongo1/data/certificats:/etc/ssl/iot-certs
      - /var/lib/docker/config/STC-mongo1/config:/etc/config
    command: mongos --configdb conf-serv/dbconfig1:37001,dbconfig2:37002,dbconfig3:37003 --port 27997 --bind_ip_all
    networks:
      STC-nw-mongo1:
        aliases:
          - dbrouter1

  dbrouter2:
    deploy:
      placement:
        constraints: [node.hostname == docker-vm2]
    image: mongo:3.6.2
    hostname: dbrouter2
    ports:
      - "27998:27998"
    volumes:
      - /var/lib/docker/config/STC-mongo1/data/dbrouter2:/data/db
      - /var/lib/docker/config/STC-mongo1/data/certificats:/etc/ssl/iot-certs
      - /var/lib/docker/config/STC-mongo1/config:/etc/config
    command: mongos --configdb conf-serv/dbconfig1:37001,dbconfig2:37002,dbconfig3:37003 --port 27998 --bind_ip_all
    networks:
      STC-nw-mongo1:
        aliases:
          - dbrouter2

  dbrouter3:
    deploy:
      placement:
        constraints: [node.hostname == docker-vm3]
    image: mongo:3.6.2
    hostname: dbrouter3
    ports:
      - "27999:27999"
    volumes:
      - /var/lib/docker/config/STC-mongo1/data/dbrouter3:/data/db
      - /var/lib/docker/config/STC-mongo1/data/certificats:/etc/ssl/iot-certs
      - /var/lib/docker/config/STC-mongo1/config:/etc/config
    command: mongos --configdb conf-serv/dbconfig1:37001,dbconfig2:37002,dbconfig3:37003 --port 27999 --bind_ip_all
    networks:
      STC-nw-mongo1:
        aliases:
          - dbrouter3

networks:
  STC-nw-mongo1:

and here is the docker-compose of my orion

version: '3.4'
services:

  orion1:
    deploy:
        placement:
          constraints: [node.hostname == docker-vm1]
    image: fiware/orion:latest
    hostname: orion1
    ports:
      - "1026:1026"
    volumes:
      - /var/lib/docker/config/STC-orion/config:/etc/sysconfig
    command: -dbhost "dbrouter1:27997,dbrouter2:27998,dbrouter3:27999" -rplSet "iot-rs1,iot-rs2,iot-rs3" -corsOrigin __ALL
    networks: 
      STC-nw-orion:
        aliases:
          - orion1

networks:
  STC-nw-orion:
    external:
      name: STC-mongo1_STC-nw-mongo1

My orion container is running, but when I look at logs I only have that information:

docker logs -f STC-orion_orion1.1.iiv0xf6qe4pco4fk679b1s3i9 time=Wednesday 28 Feb 14:07:47 2018.732Z | lvl=ERROR | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=mongoConnectionPool.cpp[195]:mongoConnect | msg=Database Startup Error (cannot connect to mongo - doing 100 retries with a 1000 microsecond interval)

Thanks for all your help

Answer 1

I am not an expert on sharding and I never tested it with Orion, but technically speaking, @fgalan is right.

When you use mongo sharding the access goes through mongos that acts as routers, ie they have the knowledge through the configuration server to route write and reads to the correct shard cluster, which also means that they handle the connection to the sharded replica set for the client accessing the routers. so you don't need to specify any replicaset, you need only to list routers to which you want to connect (that will be probably selected by the driver with a round robin approach, but I am not sure, you should check C++ legacy driver implementation to verify).

Btw, the router is a headless service so (assuming you are using stack deploy) i would have a single service that you scale up and down, so that you won't need to place a load balancer in front (to be tested honestly)

To have a more configurable / scalable docker service, you can probably tweak the work we have been doing here:

• https://github.com/smartsdk/smartsdk-recipes/tree/master/recipes/data-management/context-broker/ha
• https://github.com/smartsdk/mongo-rs-controller-swarm

Of course, feel free to do pull request if you feel like.

Answer 2

That message in the log basically means that Orion is not able to connect to MongoDB. Probably the -rplSet "iot-rs1,iot-rs2,iot-rs3" is not working, as it expects only one replica set name.

I don't remember pretty well configuration aspects of MongoDB sharded setups in which each shard is implemented by a replica set, but I'd say that replica set name is an internal aspect in each shard. In other words, the process accesing mongos (Context Broker in this case) shouldn't be aware of the replica set name used internally in each shard.

I'd recomment to start with a simpler setup, using:

command: -dbhost dbrouter1:27997 -corsOrigin __ALL

Once you get it work, try to evolve with with a 2nd and a 3rd mongos at the same time. However, not sure if the Orion driver will support it. Note that the typical multi-mongos deploy assume several instances of the application (ie 3 nodes with CB) runing mongos locally in each node (ie one mongos in each one of the 3 nodes with CB, each CB connected with its local mongos).

The following link could be useful (althogh I'm not sure if that models the same case you are trying to achive).

Answer 3

Thanks for all your advices. Here is my Orion stack file.

version: '3.4'
services:

  orion1:
    deploy:
        placement:
          constraints: [node.hostname == docker-vm1]
    image: fiware/orion:latest
    hostname: orion1
    ports:
      - "1026:1026"
    command: -dbhost dbrouter1:27997 -corsOrigin __ALL
    networks: 
      STC-nw-orion:
        aliases:
          - orion1

  orion2:
    deploy:
        placement:
          constraints: [node.hostname == docker-vm2]
    image: fiware/orion:latest
    hostname: orion2
    ports:
      - "1027:1026"
    command: -dbhost dbrouter2:27998 -corsOrigin __ALL
    networks: 
      STC-nw-orion:
        aliases:
          - orion2

  orion3:
    deploy:
        placement:
          constraints: [node.hostname == docker-vm3]
    image: fiware/orion:latest
    hostname: orion3
    ports:
      - "1028:1026"
    command: -dbhost dbrouter3:27999 -corsOrigin __ALL
    networks: 
      STC-nw-orion:
        aliases:
          - orion3

networks:
  STC-nw-orion:
    external:
      name: STC-mongo1_STC-nw-mongo1

I configure now my HAProxy in front for load balancing. Next step add authentication to my special MongoDB cluster.