Kubernetes - not unique ip per pod

Question

I'm building a 3 VM (CentOS 7) cluster of Kubernetes 1.3.2. According to this kubernetes documentation page Networking in Kubernetes : “We give every pod its own IP address” and by that there is no port collision when few pods use the same ports on the same node. But as seen here, the pods do get the same IP addresses:

[root@gloom kuber-test]# kubectl get pods -o wide -l app=userloc
NAME                          READY     STATUS    RESTARTS   AGE       IP           NODE
userloc-dep-857294609-0am9d   1/1       Running   0          27m       172.17.0.5   157.244.150.86
userloc-dep-857294609-a4538   1/1       Running   0          27m       172.17.0.7   157.244.150.96
userloc-dep-857294609-c4wzy   1/1       Running   0          6h        172.17.0.3   157.244.150.86
userloc-dep-857294609-hbl9i   1/1       Running   0          6h        172.17.0.5   157.244.150.96
userloc-dep-857294609-rpgyd   1/1       Running   0          27m       172.17.0.5   157.244.150.198
userloc-dep-857294609-tnnho   1/1       Running   0          6h        172.17.0.3   157.244.150.198

What do I miss?

EDIT - 31/07/16:
Following Sven Walter's comments, maybe the issue is that somehow the IPs which the pods had received are of the docker bridge subnet 172.17.0.0/16 (which is not distinct per node) instead of flannel's subnets 10.xxx/24 (which are distinct per node). Can this be the issue?

In case needed, here is the deployment yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: userloc-dep 
spec:
  replicas: 6
  template:
    metadata:
      labels:
        app: userloc
    spec:
      containers:
      - name: userloc
        image: globe:5000/openlso/userlocation-ms:0.1 
        ports:
        - containerPort: 8081

Answer 1

The issue occured becuase following docker documentation I had added additional docker config in /etc/systemd/system/docker.service.d/docker.conf that overrides the config in /usr/lib/systemd/system/docker.service . Unfortunatelly the scripts I used to setup the cluster (master.sh and worker.sh) doesn't refer to the first file but to the second one.
Once I removed the docker.conf file the pods got flannel's subnet.

Answer 2

After configuring flannel, assuming you did so correctly, each node will grab a slice of the overall ip network cidr. You can figure out which cidr is assigned to which node by doing an etcd ls -r and looking for a key like "coreos.com". The subnet slices assigned to each node should be unique.

Once a node has a subnet, flannel assigns that cidr to flannel.0 (a vxlan device) and you need to restart docker, eg: https://github.com/coreos/flannel#docker-integration . If you failed to restart docker, or the options are wrong, or flannel isn't running on the node, or non-unique subnets are assigned to different nodes, things won't work as expected. Reply to this if you need more help debugging and we can take it from there.

Answer 3

maybe it can help you, I have the same problem when I had more than one network interface to fix that I defined the network interface that flannel use to communicate with other nodes.

flanneld --iface=enp0s8

in my case, I change that in /etc/sysconfig/flanneld

FLANNEL_ETCD="http://master.gary.local:2379"
FLANNEL_ETCD_KEY="/atomic.io/network"
FLANNEL_OPTIONS="--iface=enp0s8"

after change that obviously you need to restart docker and flanneld daemons.