Delaying module start till edgeHub fully initiates

Question

I am trying to implement a python module (say, my-module ) on arm32v7 Raspberry Pi-3 model. The module is based on resin/rpi-raspbian:stretch (azure-iothub-device-client==1.4.0b0). I have the iothub_client python library then working fine inside the module. However when I am trying to deploy and do the iotedgectl start in the Pi, I get the following error in docker logs -f <my-module> :

Python 2.7.13 (default, Nov 24 2017, 17:33:09) 
[GCC 6.3.0 20170516]

IoT Hub Client for Python
Adding TrustedCerts from: /mnt/edgemodule/edge-device-ca.cert.pem
set_option TrustedCerts successful

Converted audio file  2643035.wav in 5.67014288902 seconds
Payload: {"translatedutctime": "2018-06-23T14:56:59.972036", "filename": "2643035.wav"}
Sending Payload for 2643035.wav ...
Error: Time:Sat Jun 23 14:56:59 2018 File:/usr/sdk/src/c/c-utility/adapters/socketio_berkeley.c Func:socketio_open Line:701 Failure: connect failure 111.
Error: Time:Sat Jun 23 14:56:59 2018 File:/usr/sdk/src/c/c-utility/src/tlsio_openssl.c Func:on_underlying_io_open_complete Line:751 Invalid tlsio_state. Expected state is TLSIO_STATE_OPENING_UNDERLYING_IO.
Error: Time:Sat Jun 23 14:56:59 2018 File:/usr/sdk/src/c/c-utility/src/tlsio_openssl.c Func:tlsio_openssl_open Line:1251 Failed opening the underlying I/O.
Error: Time:Sat Jun 23 14:56:59 2018 File:/usr/sdk/src/c/umqtt/src/mqtt_client.c Func:mqtt_client_connect Line:980 Error: io_open failed
Error: Time:Sat Jun 23 14:56:59 2018 File:/usr/sdk/src/c/iothub_client/src/iothubtransport_mqtt_common.c Func:SendMqttConnectMsg Line:1932 failure connecting to address elderberrypi.
Error: Time:Sat Jun 23 14:57:00 2018 File:/usr/sdk/src/c/c-utility/adapters/socketio_berkeley.c Func:socketio_open Line:701 Failure: connect failure 111.
Error: Time:Sat Jun 23 14:57:00 2018 File:/usr/sdk/src/c/c-utility/src/tlsio_openssl.c Func:on_underlying_io_open_complete Line:751 Invalid tlsio_state. Expected state is TLSIO_STATE_OPENING_UNDERLYING_IO.
Error: Time:Sat Jun 23 14:57:00 2018 File:/usr/sdk/src/c/c-utility/src/tlsio_openssl.c Func:tlsio_openssl_open Line:1251 Failed opening the underlying I/O.
Error: Time:Sat Jun 23 14:57:00 2018 File:/usr/sdk/src/c/umqtt/src/mqtt_client.c Func:mqtt_client_connect Line:980 Error: io_open failed
Error: Time:Sat Jun 23 14:57:00 2018 File:/usr/sdk/src/c/iothub_client/src/iothubtransport_mqtt_common.c Func:SendMqttConnectMsg Line:1932 failure connecting to address elderberrypi.
Error: Time:Sat Jun 23 14:57:02 2018 File:/usr/sdk/src/c/c-utility/adapters/socketio_berkeley.c Func:socketio_open Line:701 Failure: connect failure 111.
Error: Time:Sat Jun 23 14:57:02 2018 File:/usr/sdk/src/c/c-utility/src/tlsio_openssl.c Func:on_underlying_io_open_complete Line:751 Invalid tlsio_state. Expected state is TLSIO_STATE_OPENING_UNDERLYING_IO.
Error: Time:Sat Jun 23 14:57:02 2018 File:/usr/sdk/src/c/c-utility/src/tlsio_openssl.c Func:tlsio_openssl_open Line:1251 Failed opening the underlying I/O.
Error: Time:Sat Jun 23 14:57:02 2018 File:/usr/sdk/src/c/umqtt/src/mqtt_client.c Func:mqtt_client_connect Line:980 Error: io_open failed
Error: Time:Sat Jun 23 14:57:02 2018 File:/usr/sdk/src/c/iothub_client/src/iothubtransport_mqtt_common.c Func:SendMqttConnectMsg Line:1932 failure connecting to address elderberrypi.

Converted audio file  4417734.wav in 4.16613197327 seconds    
Payload: {"translatedutctime": "2018-06-23T14:57:04.059473", "filename": "4417734.wav"}
Sending Payload for 4417734.wav ...    
Error: Time:Sat Jun 23 14:57:06 2018 File:/usr/sdk/src/c/c-utility/adapters/socketio_berkeley.c Func:socketio_open Line:701 Failure: connect failure 111.
Error: Time:Sat Jun 23 14:57:06 2018 File:/usr/sdk/src/c/c-utility/src/tlsio_openssl.c Func:on_underlying_io_open_complete Line:751 Invalid tlsio_state. Expected state is TLSIO_STATE_OPENING_UNDERLYING_IO.
Error: Time:Sat Jun 23 14:57:06 2018 File:/usr/sdk/src/c/c-utility/src/tlsio_openssl.c Func:tlsio_openssl_open Line:1251 Failed opening the underlying I/O.
Error: Time:Sat Jun 23 14:57:06 2018 File:/usr/sdk/src/c/umqtt/src/mqtt_client.c Func:mqtt_client_connect Line:980 Error: io_open failed
Error: Time:Sat Jun 23 14:57:06 2018 File:/usr/sdk/src/c/iothub_client/src/iothubtransport_mqtt_common.c Func:SendMqttConnectMsg Line:1932 failure connecting to address elderberrypi.

Converted audio file  4486452.wav in 4.94396114349 seconds
Payload: {"translatedutctime": "2018-06-23T15:32:28.491401", "filename": "4486452.wav"}
Sending Payload for 4486452.wav ...
Confirmation[0] received for message with result = MESSAGE_TIMEOUT
    Properties: {}
    Total calls confirmed: 1

Now this goes on for several messages, and only after the start (never in between, once it stabilizes). After several such message timeouts and errors, the scenario returns back to normal, with callback confirmation, and I am also, able to receive the messages in IoTHub. As a result I always miss the first 5-10 messages and am unable to see those messages in IotHub.

After much trouble shooting, I figured there is a consistent problem. The edgeHub module always seems to fully initialize several seconds after my-module already start sending messages . Since edgeHub is not fully initialized and TLS authentication and MQTT paths are not set, the first few messages of my-module are lost. Maybe this is because the raspberry pi is a slow device?

I have tried starting my-module manually, after the edgeHub is fully initialized, all messages are delivered without any issues in that case.

Question: Is there a way to wait for the edgeHub to fully initialize and then start sending messages? More importantly, is this the actual problem or something else altogether is the cause of the issue?

Please let me know if more information is needed to get the problem context. I have tried to keep my query concise :)

Answer 1

What you pointed out is an accurate description of current behavior! As of current implementation, all modules are started at the same time. Some of them are initialized faster than edgeHub, which caused the behavior you'd seen. This is a known problem and we are implementing a fix!