Fiware Orion + Cosmos: Subscription failed - Timeout was reached trying to notify a PC that belongs to the same network

Question

I have a scenario on a Citrix Xenserver server with 3 virtual machines (VM) Centos7 on the same network (10.0.1.0/24). Each VM are responsible to give a prediction done with Apache Spark on Scala (Logistic Regression). I use Orion Context broker (CB) on Docker to create subscriptions that will be triggered to ask predictions. The CB it is located only in 10.0.1.4 VM and I made some ports available to access from the other machines.

My docker-compose.yml:

 mongo:
   image: mongo:4.4
   command: --nojournal
 orion:
   image: fiware/orion
   links:
     - mongo
   ports:
     - "1026:1026"
     - "1027:1027"
     - "1028:1028"
     - "9090:9090"
     
   command: -dbhost mongo -corsOrigin __ALL

For example, to access CB from 10.0.1.2 VM I use 10.0.1.4:1028/..... and so on.

This is the subscription I'm facing problems (and maybe the other related to 10.0.1.3 VM must have the same problem too)

 curl -v localhost:1026/v2/subscriptions -s -S -H 'Content-Type: application/json' -d @- <<EOF
{
  "description": "Suscripcion de anemia para monitorear al Paciente",
  "subject": {
    "entities": [
      {
        "id": "Paciente1",
        "type": "Paciente"
      }
    ],
    "condition": {
      "attrs": ["calculateAnaemia"]
    },
    "expression":{
        "q":"calculateAnaemia:1"
    }
  },
  "notification": {
    "http": {
      "url": "http://10.0.1.2:9002/notify"
    },
    
    "attrs": ["gender","age","hemoglobin","mch","mchc","mcv"]
    
  },
  "expires": "2040-01-01T14:00:00.00Z",
  "throttling": 10
  
}
EOF

I have a code on the 10.0.1.2 VM that is listening about changes related to this subscription on port 9002 with Fiware Cosmos:

For eventStream variable on 10.0.1.4 VM got port 9004 and for 10.0.1.3 VM it is 9003 port

For conf variable I set "spark.driver.host" on 10.0.1.4 VM with 10.0.1.4 IP and for 10.0.1.3 VM to 10.0.1.3 IP

import esqMensajeria.ActorSysMensajeria.ActoresEsquema
import org.apache.spark._
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.fiware.cosmos.orion.spark.connector._

object Main extends App{
    val conf = new SparkConf().setMaster("local[*]").setAppName("AnaemiaPrediction").set("spark.driver.host", "10.0.1.2")
    val ssc = new StreamingContext(conf, Seconds(10))
    // Create Orion Source. Receive notifications on port 9002
    val eventStream = ssc.receiverStream(new OrionReceiver(9002))
    // Esquema de mensajeria
    val actorSysMsj = new ActoresEsquema ()
    println("Esperando cambios para obtener información...")
    // Process event stream
    val processedDataStream = eventStream
      .flatMap(event => event.entities)
      .map(entity => {
        val gender: Int = entity.attrs("gender").value.asInstanceOf[Number].intValue()
        val age: Int = entity.attrs("age").value.asInstanceOf[Number].intValue()
        val hemoglobin: Double = entity.attrs("hemoglobin").value.asInstanceOf[Double].doubleValue()
        val mch: Double = entity.attrs("mch").value.asInstanceOf[Double].doubleValue()
        val mchc: Double = entity.attrs("mchc").value.asInstanceOf[Double].doubleValue()
        val mcv: Double = entity.attrs("mcv").value.asInstanceOf[Double].doubleValue()
        actorSysMsj.start((entity.id, gender,age,hemoglobin,mch,mchc,mcv),conf)
        (entity.id, gender,age,hemoglobin,mch,mchc,mcv)
      })
    processedDataStream.print
    ssc.start()
    ssc.awaitTermination()
}

But when I trigger, the subscription fails showing the next (not only on 10.0.1.2 but 10.0.1.3 VM too):

{"id":"61cb8569a1e87a254e16066d",
"description":"Suscripcion de anemia para monitorear al Paciente",
"expires":"2040-01-01T14:00:00.000Z",
"status":"failed",
"subject":{"entities":[{"id":"Paciente1","type":"Paciente"}],
"condition":{"attrs":["calculateAnaemia"]}},
"notification":
{"timesSent":3,
"lastNotification":"2021-12-29T00:03:49.000Z",
"attrs":"gender","age","hemoglobin","mch","mchc","mcv"],"
onlyChangedAttrs":false,
"attrsFormat":"normalized",
http":{"url":"http://10.0.1.2:9002/notify"},
"lastFailure":"2021-12-29T00:03:54.000Z",
"lastFailureReason":"Timeout was reached"},
"throttling":10}]

The curious thing is when I worked with the subscription related to the 10.0.1.4 VM that has the CB container, the subscription remains active and i get my expected result.

This is the subscription:

 curl -v localhost:1026/v2/subscriptions -s -S -H 'Content-Type: application/json' -d @- <<EOF
{
  "description": "Suscripcion de deceso para monitorear al Paciente",
  "subject": {
    "entities": [
      {
        "id": "Paciente1",
        "type": "Paciente"
      }
    ],
    "condition": {
      "attrs": [
        "calculateDeceased"
      ]
    },
    "expression":{
        "q":"calculateDeceased:1"
    }
  },
  "notification": {
    "http": {
      "url": "http://10.0.1.4:9004/notify"
    },
    "attrs": [
      "gender","age","hasAnaemia","creatinePP","hasDiabetes","ejecFrac","highBloodP","platelets","serumCreatinine","serumSodium","smoking","time"
    ]
  },
  "expires": "2040-01-01T14:00:00.00Z"
}
EOF

This is the answer when it is triggered and process perfectly:

{"id":"61caab07a1e87a254e160665",
"description":"Suscripcion de deceso para monitorear al Paciente",
"expires":"2040-01-01T14:00:00.000Z",
"status":"active",
"subject":{"entities":[{"id":"Paciente1","type":"Paciente"}],
"condition":{"attrs":["calculateDeceased"]}},
"notification":{"timesSent":1,"lastNotification":"2021-12-28T06:15:41.000Z",
"attrs":["gender","age","hasAnaemia","creatinePP","hasDiabetes","ejecFrac","highBloodP","platelets","serumCreatinine","serumSodium","smoking","time"],
"onlyChangedAttrs":false,
"attrsFormat":"normalized",
"http":{"url":"http://10.0.1.4:9004/notify"},
"lastSuccess":"2021-12-28T06:15:43.000Z",
"lastSuccessCode":200}}

I have to say I'm new to Spark, Scala and even Fiware. But projects are projects and maybe I'm missing something I did not see in all I read to set up this project. Also, I stopped all firewalls (firewalld) because I was facing a "Couldn't connect to server" error on subscriptions related to 10.0.1.2 and 10.0.1.3 VMs. I did an sudo yum update too, I pinged all the VMs between each other and I got nice response. One thing that I do not know if it is important: I have internet on all my VMs but I can't ping for example... www.google.com or 8.8.8.8. So, any suggestions are welcome. I apologized for my english. Thanks in advance~

Answer 1

Well, after 3 days of keep looking and trying i just discovered that I need to turn off the 1.2 and 1.3 firewalls and keep it on 1.4.