Redhat clair could not send notification via notifier

Question

I'm currently looking that I can use clair to scan quayrepos. Here some basic Informations:

• Docker Version: 19.03.13
• Docker API Version: 1.40
• GO Version: go1.13.15
• OS: redhat 7.9
• Container Version (Redis,Postgres,Clair,Quay): latest
• Storage: RadisGWStorage
• Quay DB: Mariadb (external Server)
• Clair DB: Postgres (running on the same server like quay)
• Redis, Postgres, Clair and Quay are running on the same server but in different Containers.

My Problem:

{"Event":"could not send notification via notifier","Level":"error","Location":"notifier.go:173","Time":"2020-10-15 08:04:40.730379","error":"Post https://domain/secscan/notify: proxyconnect tcp: dial tcp IP:6063: connect: connection refused","notification name":"09c0498e-c30d-4f1b-9bb2-d07588351618","sender name":"webhook"}


{"Event":"giving up on sending notification : max attempts exceeded","Level":"info","Location":"notifier.go:157","Time":"2020-10-15 08:04:40.730431","max attempts":3,"notification name":"09c0498e-c30d-4f1b-9bb2-d07588351618","sender name":"webhook"}

My Config for Clair:

    clair:
  database:
    type: pgsql
    options:
      # A PostgreSQL Connection string pointing to the Clair Postgres database.
      # Documentation on the format can be found at http//www.postgresql.org/docs/9.4/static/libpq-connect.html
      source:  postgresql://username:password@domain:5432/clairtest?sslmode=disable
      cachesize: 16384
  api:
    # The port at which Clair will report its health status. For example, if Clair is running at
    # https://clair.mycompany.com, the health will be reported at
    # http://clair.mycompany.com:6061/health.
    healthport: 6061

    port: 6062
    timeout: 900s

    # paginationkey can be any random set of characters. *Must be the same across all Clair instances*.
    paginationkey: "key"

  updater:
    # interval defines how often Clair will check for updates from its upstream vulnerability databases.
    interval: 6h
  notifier:
    attempts: 3
    renotifyinterval: 1h
    http:
      # QUAY_ENDPOINT defines the endpoint at which Quay is running.
      # For example: http://myregistry.mycompany.com
      endpoint: https://domain/secscan/notify
      proxy: https://domain:6063

jwtproxy:
  signer_proxy:
    enabled: true
    listen_addr: :6063
    ca_key_file: /certificates/mitm.key # Generated internally, do not change.
    ca_crt_file: /certificates/mitm.crt # Generated internally, do not change.
    insecure_skip_verify: true
    signer:
      issuer: security_scanner
      expiration_time: 5m
      max_skew: 1m
      nonce_length: 32
      private_key:
        type: preshared
        options:
          key_id: key
          private_key_path: /clair/config/security_scanner.pem

  verifier_proxies:
  - enabled: true
    # The port at which Clair will listen.
    listen_addr: :6060

    # If Clair is to be served via TLS, uncomment these lines. See the "Running Clair under TLS"
    # section below for more information.
    # key_file: /clair/config/clair.key
    # crt_file: /clair/config/clair.crt

    verifier:
      # CLAIR_ENDPOINT is the endpoint at which this Clair will be accessible. Note that the port
      # specified here must match the listen_addr port a few lines above this.
      # Example: https://myclair.mycompany.com:6060
      audience: https://domain:6060

      upstream: https://domain:6062
      key_server:
        type: keyregistry
        options:
          # QUAY_ENDPOINT defines the endpoint at which Quay is running.
          # Example: https://myregistry.mycompany.com
          registry: https://domain/keys/
      claims_verifiers:
      - type: static
        options:
          iss: jwtproxy

So do you know how to solve this problem, or do you know how I can debug it better. Btw I have tried to debug it with tcpdump and strace and wireshark.

Thanks for your help!

Answer 1

I have solved it a couple hours before. First off all I changed my IP to 127.0.0.1:6063. And after that we found out that quay and clair can't create the chain of trust to the rootca if you don't give him the intermediate ca. And then we found out that clair had an expired key and couldn't create a new one. So we deleted all keys and after a few restart, it was working fine.

LG VallingSki