Gitlab CI 中的 Kafka 和 Zookeeper

Question

I'm trying to run a simple test if my application is running properly without any issues. My issue is that faust needs a connection to kafka on initialization - so I'm trying to run kafka with zookeeper as services but I'm not able to connect them properly.

Error:

2021-12-16T13:53:51.385341793Z [2021-12-16 13:53:51,385] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
2021-12-16T13:53:51.391012666Z [2021-12-16 13:53:51,390] INFO zookeeper.request.timeout value is 0. feature enabled=false (org.apache.zookeeper.ClientCnxn)
2021-12-16T13:53:51.395158219Z [2021-12-16 13:53:51,395] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
2021-12-16T13:53:51.399485772Z [2021-12-16 13:53:51,397] ERROR Unable to resolve address: zookeeper:2181 (org.apache.zookeeper.client.StaticHostProvider)
2021-12-16T13:53:51.399499707Z java.net.UnknownHostException: zookeeper: Name or service not known
2021-12-16T13:53:51.399503169Z  at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
2021-12-16T13:53:51.399506400Z  at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
2021-12-16T13:53:51.399509510Z  at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1519)
2021-12-16T13:53:51.399512353Z  at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
2021-12-16T13:53:51.399531020Z  at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
2021-12-16T13:53:51.399534098Z  at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
2021-12-16T13:53:51.399537044Z  at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
2021-12-16T13:53:51.399540881Z  at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2021-12-16T13:53:51.399544771Z  at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2021-12-16T13:53:51.399548877Z  at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2021-12-16T13:53:51.399553025Z  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1207)
2021-12-16T13:53:51.406655054Z [2021-12-16 13:53:51,406] WARN Session 0x0 for sever zookeeper:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
2021-12-16T13:53:51.406696302Z java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
2021-12-16T13:53:51.406703099Z  at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:78)
2021-12-16T13:53:51.406707676Z  at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:41)
2021-12-16T13:53:51.406711700Z  at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1161)
2021-12-16T13:53:51.406715631Z  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1210)
2021-12-16T13:53:52.508636206Z [2021-12-16 13:53:52,508] ERROR Unable to resolve address: zookeeper:2181 (org.apache.zookeeper.client.StaticHostProvider)
2021-12-16T13:53:52.508665462Z java.net.UnknownHostException: zookeeper

.gitlab-ci.yml:

.zoo_service: &zoo_service
  name: zookeeper:latest
  alias: zookeeper

.kafka_service: &kafka_service
  name: bitnami/kafka:latest
  alias: kafka


faust:
  variables:
    ALLOW_ANONYMOUS_LOGIN: "yes"
    KAFKA_BROKER_ID: 1
    KAFKA_CFG_LISTENERS: "PLAINTEXT://:9092"
    KAFKA_CFG_ADVERTISED_LISTENERS: "PLAINTEXT://127.0.0.1:9092"
    KAFKA_CFG_ZOOKEEPER_CONNECT: "zookeeper:2181"
    ALLOW_PLAINTEXT_LISTENER: "yes"
  stage: test
  <<: *python_image
  services:
    - *zoo_service
    - *kafka_service
  before_script:
    - *setup_venv_script
  script:
    - faust -A runner worker -l info & sleep 15; kill -HUP $!
  <<: *load_env
  except:
    - schedules

I was hoping I'm doing it the right way - sadly there is not many resources I can read about this issue. I understand the issue is between kafka and zookeeper, but I'm not sure how to fix it (Thought this is the correct way). Can even 2 services communicate to each other in CI?

Thanks!

Answer 1

Glancing over the GitLab CI docs about connecting to different services, it mentions a feature flag to allow cross-service communication, so try

faust:
  variables:
    FF_NETWORK_PER_BUILD: 1
    ...
  services:
    ...

Also, for Kafka communication, it need to advertise its alias rather than localhost, so change

KAFKA_CFG_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:9092"