无法使用 NodePort 服务连接到 kubernetes 中的 redis pod

Question

I'm fairly new to kubernetes and I'm trying to orchestrate my rails app using minikube on my MacBook. My app includes MySQL, Redis and Sidekiq. I'm running webapp, sidekiq, redis and database in isolated pods. Sidekiq pod is not connecting to redis pod.

kubectl logs of sidekiq pod says this:

2020-09-15T14:01:16.978Z 1 TID-gnaz4yzs0 INFO: Booting Sidekiq 4.2.10 with redis options {:url=>"redis://redis:6379/0"}
2020-09-15T14:01:18.475Z 1 TID-gnaz4yzs0 INFO: Running in ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
2020-09-15T14:01:18.475Z 1 TID-gnaz4yzs0 INFO: See LICENSE and the LGPL-3.0 for licensing details.
2020-09-15T14:01:18.475Z 1 TID-gnaz4yzs0 INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
/usr/local/bundle/gems/redis-3.3.1/lib/redis/client.rb:459: warning: constant ::Fixnum is deprecated
Error connecting to Redis on redis:6379 (Errno::ECONNREFUSED)
/usr/local/bundle/gems/redis-3.3.1/lib/redis/client.rb:345:in `rescue in establish_connection'
/usr/local/bundle/gems/redis-3.3.1/lib/redis/client.rb:330:in `establish_connection'
/usr/local/bundle/gems/redis-3.3.1/lib/redis/client.rb:101:in `block in connect'
/usr/local/bundle/gems/redis-3.3.1/lib/redis/client.rb:293:in `with_reconnect'
/usr/local/bundle/gems/redis-3.3.1/lib/redis/client.rb:100:in `connect'
/usr/local/bundle/gems/redis-3.3.1/lib/redis/client.rb:364:in `ensure_connected'
/usr/local/bundle/gems/redis-3.3.1/lib/redis/client.rb:221:in `block in process'
/usr/local/bundle/gems/redis-3.3.1/lib/redis/client.rb:306:in `logging'
/usr/local/bundle/gems/redis-3.3.1/lib/redis/client.rb:220:in `process'
/usr/local/bundle/gems/redis-3.3.1/lib/redis/client.rb:120:in `call'
/usr/local/bundle/gems/redis-3.3.1/lib/redis.rb:251:in `block in info'
/usr/local/bundle/gems/redis-3.3.1/lib/redis.rb:58:in `block in synchronize'
/usr/local/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
/usr/local/bundle/gems/redis-3.3.1/lib/redis.rb:58:in `synchronize'
/usr/local/bundle/gems/redis-3.3.1/lib/redis.rb:250:in `info'
/usr/local/bundle/gems/sidekiq-4.2.10/lib/sidekiq.rb:113:in `block in redis_info'
/usr/local/bundle/gems/sidekiq-4.2.10/lib/sidekiq.rb:95:in `block in redis'
/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb:63:in `block (2 levels) in with'
/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `handle_interrupt'
/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `block in with'
/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `handle_interrupt'
/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `with'
/usr/local/bundle/gems/sidekiq-4.2.10/lib/sidekiq.rb:92:in `redis'
/usr/local/bundle/gems/sidekiq-4.2.10/lib/sidekiq.rb:106:in `redis_info'
/usr/local/bundle/gems/sidekiq-4.2.10/lib/sidekiq/cli.rb:71:in `run'
/usr/local/bundle/gems/sidekiq-4.2.10/bin/sidekiq:12:in `<top (required)>'
/usr/local/bundle/bin/sidekiq:29:in `load'
/usr/local/bundle/bin/sidekiq:29:in `<main>'

My webapp.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checklist-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: railsapp
    spec:
      containers:
        - name: webapp
          image: masettyabhishek/checklist:latest
          command: ["rails", "s", "-p", "3001", "-b", "0.0.0.0", "-e", "PRODUCTION"]
          ports:
          - name: checklist-port
            containerPort: 3001
          env:
            - name: MYSQL_HOST
              value: database-service
            - name: MYSQL_USER
              value: root
            - name: MYSQL_PASSWORD
              value: Mission2019
            - name: MYSQL_DATABASE
              value: checklist
            - name: MYSQL_ROOT_PASSWORD
              value: Mission2019
            - name: REDIS_URL
              value: redis
            - name: REDIS_PORT
              value: "6379"
  selector:
    matchLabels:
      app: railsapp

webapp-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: webapp-service
spec:
  ports:
  - port: 3001
    protocol: TCP
  type: NodePort
  selector:
    app: railsapp

sidekiq.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sidekiq-deployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        instance: sidekiq
    spec:
      containers:
        - name: sidekiq
          image: masettyabhishek/checklist:latest
          command: ["sidekiq", "-C", "config/sidekiq.yml"]
          env:
            - name: MYSQL_HOST
              value: database-service
            - name: MYSQL_USER
              value: root
            - name: MYSQL_PASSWORD
              value: Mission2019
            - name: MYSQL_DATABASE
              value: checklist
            - name: MYSQL_ROOT_PASSWORD
              value: Mission2019
            - name: REDIS_URL
              value: redis
            - name: REDIS_PORT
              value: "6379"
          ports:
            - name: redis-port
              containerPort: 6379
  selector:
    matchLabels:
      instance: sidekiq

redis.yaml

apiVersion: v1
kind: Pod
metadata:
  name: redis-pod
spec:
  containers:
  - name: redis
    image: redis:alpine
    command: ["redis-server"]
    ports:
      - containerPort: 6379

---
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  selector:
    name: redis-pod
    instance: sidekiq
    app: railsapp
  type: NodePort
  ports:
  - port: 6379

This is sidekiq.yml in my rails app

Sidekiq.configure_server do |config|
    config.redis = { url: "redis://#{ENV['REDIS_URL']}:#{ENV['REDIS_PORT']}/0"}
end

Sidekiq.configure_client do |config|
    config.redis = { url: "redis://#{ENV['REDIS_URL']}:#{ENV['REDIS_PORT']}/0"}
end

This is Dockerfile if that helps to answer the question.

FROM ubuntu:16.04

ENV RUBY_MAJOR="2.6" \
    RUBY_VERSION="2.6.3" \
    RUBYGEMS_VERSION="3.0.8" \
    BUNDLER_VERSION="1.17.3" \
    RAILS_VERSION="5.2.1" \
    RAILS_ENV="production" \
    GEM_HOME="/usr/local/bundle"
ENV BUNDLE_PATH="$GEM_HOME" \
    BUNDLE_BIN="$GEM_HOME/bin" \
    BUNDLE_SILENCE_ROOT_WARNING=1 \
    BUNDLE_APP_CONFIG="$GEM_HOME"

ENV PATH="$BUNDLE_BIN:$GEM_HOME/bin:$GEM_HOME/gems/bin:$PATH"

USER root
RUN apt-get update && \
      apt-get -y install sudo
RUN echo "%sudo ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers && \
    addgroup --gid 1024 stars && \
    useradd -G stars,sudo -d /home/user --shell /bin/bash -m user
RUN mkdir -p /usr/local/etc \
    && echo 'install: --no-document' >> /usr/local/etc/gemrc \
    && echo 'update: --no-document' >> /usr/local/etc/gemrc

USER user
RUN sudo apt-get -y install --no-install-recommends vim make gcc zlib1g-dev autoconf build-essential libssl-dev libsqlite3-dev \
    curl htop unzip mc openssh-server openssl bison libgdbm-dev ruby git libmysqlclient-dev tzdata mysql-client
    
RUN sudo rm -rf /var/lib/apt/lists/* \
    && sudo curl -fSL -o ruby.tar.gz "http://cache.ruby-lang.org/pub/ruby/$RUBY_MAJOR/ruby-$RUBY_VERSION.tar.gz" \
    && sudo mkdir -p /usr/src/ruby \
    && sudo tar -xzf ruby.tar.gz -C /usr/src/ruby --strip-components=1 \
    && sudo rm ruby.tar.gz

USER root
RUN cd /usr/src/ruby \
    && { sudo echo '#define ENABLE_PATH_CHECK 0'; echo; cat file.c; } > file.c.new && mv file.c.new file.c \
    && autoconf \
    && ./configure --disable-install-doc

USER user
RUN cd /usr/src/ruby \
    && sudo make -j"$(nproc)" \
    && sudo make install \
    && sudo gem update --system $RUBYGEMS_VERSION \
    && sudo rm -r /usr/src/ruby
RUN sudo gem install bundler --version "$BUNDLER_VERSION"

RUN sudo mkdir -p "$GEM_HOME" "$BUNDLE_BIN" \
    && sudo chmod 777 "$GEM_HOME" "$BUNDLE_BIN" \
    && sudo gem install rails --version "$RAILS_VERSION"
RUN mkdir -p ~/.ssh && \
    chmod 0700 ~/.ssh && \
    ssh-keyscan github.com > ~/.ssh/known_hosts
ARG ssh_pub_key
ARG ssh_prv_key
RUN echo "$ssh_pub_key" > ~/.ssh/id_rsa.pub && \
    echo "$ssh_prv_key" > ~/.ssh/id_rsa && \
    chmod 600 ~/.ssh/id_rsa.pub && \
    chmod 600 ~/.ssh/id_rsa
USER root
RUN curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
RUN apt-get install -y nodejs
USER user
WORKDIR /data
RUN sudo mkdir /data/checklist
WORKDIR /data/checklist
ADD Gemfile Gemfile.lock ./
RUN sudo chown -R user /data/checklist
RUN bundle install
ADD . .
RUN sudo chown -R user /data/checklist
EXPOSE 3001
ENV RAILS_SERVE_STATIC_FILES true
ENV RAILS_LOG_TO_STDOUT true
RUN chmod +x ./config/docker/prepare-db.sh && sh ./config/docker/prepare-db.sh
ENTRYPOINT ["bundle", "exec"]
CMD ["sh", "./config/docker/startup.sh"]

kubectl describe svc redis

➜  checklist kubectl describe svc redis
Name:                     redis
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 app=railsapp,instance=sidekiq,name=redis-pod
Type:                     NodePort
IP:                       10.103.6.43
Port:                     <unset>  6379/TCP
TargetPort:               6379/TCP
NodePort:                 <unset>  31886/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Answer 1

As you can see the Endpoints section of the redis service is not having pod IPs which is the reason for Connection refused error. The Pod need to have label matching with selector of service. Updating the redis pod with labels as below should solve the issue.

apiVersion: v1
kind: Pod
metadata:
  name: redis-pod
  labels:
    instance: sidekiq
    app: ailsapp
    name: redis-pod
spec:
  containers:
  - name: redis
    image: redis:alpine
    command: ["redis-server"]
    ports:
      - containerPort: 6379