[英]How does service discovery work with modern docker/docker-compose?
I'm using Docker 1.11.1 and docker-compose 1.8.0-rc2.我正在使用 Docker 1.11.1 和 docker-compose 1.8.0-rc2。
In the good old days (so, last year), you could set up a docker-compose.yml
file like this:在过去的美好时光(去年),您可以像这样设置
docker-compose.yml
文件:
app:
image: myapp
frontend:
image: myfrontend
links:
- app
And then start up the environment like this:然后像这样启动环境:
docker scale app=3 frontend=1
And your frontend container could inspect the environment variables for variables named APP_1_PORT
, APP_2_PORT
, etc to discover the available backend hosts and configure itself accordingly.并且您的前端容器可以检查名为
APP_1_PORT
、 APP_2_PORT
等变量的环境变量,以发现可用的后端主机并相应地配置自身。
Times have changed.时代变了。 Now, we do this...
现在,我们这样做...
version: '2'
services:
app:
image: myapp
frontend:
image: myfrontend
links:
- app
...and instead of environment variables, we get DNS. ...而不是环境变量,我们得到了 DNS。 So inside the
frontend
container, I can ask for app_app_1
or app_app_2
or app_app_3
and get the corresponding ip address.所以在
frontend
容器内,我可以请求app_app_1
或app_app_2
或app_app_3
并获取相应的 ip 地址。 I can also ask for app
and get the address of app_app_1
.我也可以要求
app
并获取app_app_1
的地址。
But how do I discover all of the available backend containers?但是我如何发现所有可用的后端容器呢? I guess I could loop over
getent hosts ...
until it fails:我想我可以遍历
getent hosts ...
直到它失败:
counter=1
while :; do
getent hosts app_$counter || break
backends="$backends app_$counter"
let counter++
done
But that seems ugly and fragile.但这看起来既丑陋又脆弱。
I've heard rumors about round-robin dns, but (a) that doesn't seem to be happening in my test environment, and (b) that doesn't necessarily help if your frontend needs simultaneous connections to the backends.我听说过关于循环 dns 的谣言,但是 (a) 这在我的测试环境中似乎没有发生,并且 (b) 如果您的前端需要同时连接到后端,这不一定有帮助。
How is simple container and service discovery meant to work in the modern Docker world?简单的容器和服务发现如何在现代 Docker 世界中工作?
Docker comes with a built-in nameserver . Docker 带有一个内置的名称服务器。 The server is, by default, reachable via
127.0.0.11:53
.默认情况下,服务器可通过
127.0.0.11:53
访问。
Every container has by default a nameserver entry in /etc/resolv.conf
, so it is not required to specify the address of the nameserver from within the container.默认情况下,每个容器在
/etc/resolv.conf
中都有一个名称服务器条目,因此不需要从容器内指定名称服务器的地址。 That is why you can find your service from within the network with service
or task_service_n
.这就是为什么您可以使用
service
或task_service_n
从网络中找到您的服务。
If you do task_service_n
then you will get the address of the corresponding service replica.如果你执行
task_service_n
那么你将获得相应服务副本的地址。
If you only ask for the service
docker will perform internal load balancing
between container in the same network and external load balancing
to handle requests from outside.如果只要求
service
docker 会在同一个网络中的容器之间进行internal load balancing
, external load balancing
来处理来自外部的请求。
When swarm is used, docker will additionally use two special networks.当使用 swarm 时,docker 会额外使用两个特殊网络。
ingress network
, which is actually an overlay network and handles incomming trafic to the swarm. ingress network
,实际上是一个覆盖网络,负责处理进入群的流量。 It allows to query any service from any node in the swarm.docker_gwbridge
, a bridge network , which connects the overlay networks of the individual hosts to an their physical network. docker_gwbridge
是一个桥接网络,它将单个主机的覆盖网络连接到它们的物理网络。 (including ingress) When using swarm to deploy services , the behavior as described in the examples below will not work unless endpointmode is set to dns roundrobin instead of vip.使用 swarm 部署服务时,除非将 endpointmode 设置为 dns roundrobin 而不是 vip,否则下面示例中描述的行为将不起作用。
endpoint_mode: vip - Docker assigns the service a virtual IP (VIP) that acts as the front end for clients to reach the service on a network.
endpoint_mode: vip - Docker 为服务分配一个虚拟 IP (VIP),作为客户端访问网络上的服务的前端。 Docker routes requests between the client and available worker nodes for the service, without client knowledge of how many nodes are participating in the service or their IP addresses or ports.
Docker 在客户端和服务的可用工作节点之间路由请求,而客户端不知道有多少节点参与服务或其 IP 地址或端口。 (This is the default.)
(这是默认设置。)
endpoint_mode: dnsrr - DNS round-robin (DNSRR) service discovery does not use a single virtual IP.
endpoint_mode: dnsrr - DNS 循环 (DNSRR) 服务发现不使用单个虚拟 IP。 Docker sets up DNS entries for the service such that a DNS query for the service name returns a list of IP addresses, and the client connects directly to one of these.
Docker 为服务设置 DNS 条目,以便对服务名称的 DNS 查询返回 IP 地址列表,并且客户端直接连接到其中之一。 DNS round-robin is useful in cases where you want to use your own load balancer, or for Hybrid Windows and Linux applications.
如果您想使用自己的负载均衡器,或者对于混合 Windows 和 Linux 应用程序,DNS 轮询非常有用。
For example deploy three replicas from dig/docker-compose.yml例如从dig/docker-compose.yml部署三个副本
version: '3.8'
services:
whoami:
image: "traefik/whoami"
deploy:
replicas: 3
You can use tools such as dig or nslookup to do a DNS lookup against the nameserver in the same network .您可以使用dig或nslookup等工具对同一网络中的名称服务器进行 DNS 查找。
docker run --rm --network dig_default tutum/dnsutils dig whoami
; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> whoami
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58433
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;whoami. IN A
;; ANSWER SECTION:
whoami. 600 IN A 172.28.0.3
whoami. 600 IN A 172.28.0.2
whoami. 600 IN A 172.28.0.4
;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Mon Nov 16 22:36:37 UTC 2020
;; MSG SIZE rcvd: 90
If you are only interested in the IP, you can provide the +short
option如果您只对 IP 感兴趣,可以提供
+short
选项
docker run --rm --network dig_default tutum/dnsutils dig +short whoami
172.28.0.3
172.28.0.4
172.28.0.2
Or look for specific service或寻找特定服务
docker run --rm --network dig_default tutum/dnsutils dig +short dig_whoami_2
172.28.0.4
The default loadbalancing happens on the transport layer or layer 4 of the OSI Model .默认负载平衡发生在OSI 模型的传输层或第 4 层。 So it is TCP/UDP based.
所以它是基于 TCP/UDP 的。 That means it is not possible to inpsect and manipulate http headers with this method.
这意味着无法使用此方法检查和操作 http 标头。 In the enterprise edition it is apparently possible to use labels similar to the ones treafik is using in the example a bit further down.
在企业版中,显然可以使用类似于 treafik 在示例中使用的标签。
docker run --rm --network dig_default curlimages/curl -Ls http://whoami
Hostname: eedc94d45bf4
IP: 127.0.0.1
IP: 172.28.0.3
RemoteAddr: 172.28.0.5:43910
GET / HTTP/1.1
Host: whoami
User-Agent: curl/7.73.0-DEV
Accept: */*
Here is the hostname from 10 times curl:这是 10 次 curl 的主机名:
Hostname: eedc94d45bf4
Hostname: 42312c03a825
Hostname: 42312c03a825
Hostname: 42312c03a825
Hostname: eedc94d45bf4
Hostname: d922d86eccc6
Hostname: d922d86eccc6
Hostname: eedc94d45bf4
Hostname: 42312c03a825
Hostname: d922d86eccc6
Health checks , by default, are done by checking the process id (PID) of the container on the host kernel.默认情况下, 健康检查是通过检查主机内核上容器的进程 ID (PID) 来完成的。 If the process is running successfully, the container is considered healthy.
如果进程成功运行,则容器被认为是健康的。
Oftentimes other health checks are required.通常需要其他健康检查。 The container may be running but the application inside has crashed.
容器可能正在运行,但里面的应用程序已经崩溃。 In many cases a TCP or HTTP check is preferred.
在许多情况下,首选 TCP 或 HTTP 检查。
It is possible to bake a custom health checks into images.可以将自定义健康检查烘焙到图像中。 For example, using curl to perform L7 health checks.
例如,使用curl执行 L7 健康检查。
FROM traefik/whoami
HEALTHCHECK CMD curl --fail http://localhost || exit 1
It is also possible to specify the health check via cli when starting the container.也可以在启动容器时通过 cli 指定健康检查。
docker run \
--health-cmd "curl --fail http://localhost || exit 1" \
--health-interval=5s \
--timeout=3s \
traefik/whoami
As initially mentioned, swarms behavior is different in that it will assign a virtual IP to services by default.正如最初提到的,swarm 行为的不同之处在于默认情况下它会为服务分配一个虚拟 IP。 Its actually not different its just docker or docker-compose doesn't create real services, it just imitates the behavior of swarm but still runs the container normally, as services can, in fact, only be created by manager nodes.
它实际上并没有什么不同,它只是 docker 或 docker-compose 不创建真正的服务,它只是模仿 swarm 的行为但仍然正常运行容器,因为服务实际上只能由管理器节点创建。
Keeping in mind we are on a swarm manager and thus the default mode is VIP请记住,我们在群管理器上,因此默认模式是 VIP
Create a overlay network that can be used by regular containers too创建一个也可以被常规容器使用的覆盖网络
$ docker network create --driver overlay --attachable testnet
create some service with 2 replicas用 2 个副本创建一些服务
$ docker service create --network testnet --replicas 2 --name digme nginx
Now lets use dig again and making sure we attach the container to the same network现在让我们再次使用 dig 并确保我们将容器附加到同一网络
$ docker run --network testnet --rm tutum/dnsutils dig digme
digme. 600 IN A 10.0.18.6
We see that indeed we only got one IP address back, so it appears that this is the virtual IP that has been assigned by docker.我们看到确实只取回了一个IP地址,所以看起来这是docker分配的虚拟IP。
Swarm allows actually to get the single IPs in this case without explicitly setting the endpoint mode.在这种情况下, Swarm 实际上允许获取单个 IP,而无需明确设置端点模式。
We can query for tasks.<servicename>
in this case that is tasks.digme
我们可以查询
tasks.<servicename>
在这种情况下是tasks.digme
$ docker run --network testnet --rm tutum/dnsutils dig tasks.digme
tasks.digme. 600 IN A 10.0.18.7
tasks.digme. 600 IN A 10.0.18.8
This has brought us 2 A records pointing to the individual replicas.这给我们带来了 2 个指向单个副本的 A 记录。
Now lets create another service with endpointmode set to dns roundrobin现在让我们创建另一个服务,端点模式设置为 dns roundrobin
docker service create --endpoint-mode dnsrr --network testnet --replicas 2 --name digme2 nginx
$ docker run --network testnet --rm tutum/dnsutils dig digme2
digme2. 600 IN A 10.0.18.21
digme2. 600 IN A 10.0.18.20
This way we get both IPs without adding the prefix tasks
.这样我们就可以在不添加前缀
tasks
情况下获得两个 IP。
If the built in features are not sufficent, some strategies can be implemented to achieve better control.如果内置功能不够,可以实施一些策略来实现更好的控制。 Below are some examples.
下面是一些例子。
Haproxy can use the docker nameserver in combination with dynamic server templates to discover the running container. Haproxy可以结合使用docker nameserver 和动态服务器模板来发现正在运行的容器。 Then the traditional proxy features can be leveraged to achieve powerful layer 7 load balancing with http header manipulation and chaos engeering such as retries.
然后可以利用传统的代理功能通过 http 标头操作和混沌工程(例如重试)来实现强大的第 7 层负载平衡。
version: '3.8'
services:
loadbalancer:
image: haproxy
volumes:
- ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
ports:
- 80:80
- 443:443
whoami:
image: "traefik/whoami"
deploy:
replicas: 3
...
resolvers docker
nameserver dns1 127.0.0.11:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold other 10s
hold refused 10s
hold nx 10s
hold timeout 10s
hold valid 10s
hold obsolete 10s
...
backend whoami
balance leastconn
option httpchk
option redispatch 1
retry-on all-retryable-errors
retries 2
http-request disable-l7-retry if METH_POST
dynamic-cookie-key MY_SERVICES_HASHED_ADDRESS
cookie MY_SERVICES_HASHED_ADDRESS insert dynamic
server-template whoami- 6 whoami:80 check resolvers docker init-addr libc,none
...
The previous method is already pretty decent.之前的方法已经很不错了。 However, you may have noticed that it requires knowing which services should be discovered and also the number of replicas to discover is hard coded.
但是,您可能已经注意到,它需要知道应该发现哪些服务以及要发现的副本数量是硬编码的。 Traefik , a container native edge router, solves both problems.
Traefik是一个容器原生边缘路由器,解决了这两个问题。 As long as we enable Traefik via label , the service will be discovered.
只要我们通过label启用 Traefik,就会发现该服务。 This decentralized the configuration.
这分散了配置。 It is as if each service registers itself.
就好像每个服务都注册了自己。
The label can also be used to inspect and manipulate http headers .该标签还可用于检查和操作 http 标头。
version: "3.8"
services:
traefik:
image: "traefik:v2.3"
command:
- "--log.level=DEBUG"
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
ports:
- "80:80"
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
whoami:
image: "traefik/whoami"
labels:
- "traefik.enable=true"
- "traefik.port=80"
- "traefik.http.routers.whoami.entrypoints=web"
- "traefik.http.routers.whoami.rule=PathPrefix(`/`)"
- "traefik.http.services.whoami.loadbalancer.sticky=true"
- "traefik.http.services.whoami.loadbalancer.sticky.cookie.name=MY_SERVICE_ADDRESS"
deploy:
replicas: 3
Consul is a tool for service discovery and configuration management. Consul是一个用于服务发现和配置管理的工具。 Services have to be registered via API request.
服务必须通过 API 请求注册。 It is a more complex solution that probably only makes sense in bigger clusters, but can be very powerful.
这是一个更复杂的解决方案,可能只在更大的集群中才有意义,但可能非常强大。 Usually it recommended running this on bare metal and not in a container.
通常它建议在裸机上而不是在容器中运行它。 You could install it alongside the docker host on each server in your cluster.
您可以将它与集群中每台服务器上的 docker 主机一起安装。
In this example it has been paired with the registrator image , which takes care of registering the docker services in consuls catalog.在这个例子中,它与注册器镜像配对,它负责在 consuls 目录中注册 docker 服务。
The catalog can be leveraged in many ways.可以通过多种方式利用目录。 One of them is to use consul-template .
其中之一是使用consul-template 。
Note that consul comes with its own DNS resolver so in this instance the docker DNS resolver is somewhat neglected.请注意,consul 带有自己的 DNS 解析器,因此在这种情况下,docker DNS 解析器在某种程度上被忽略了。
version: '3.8'
services:
consul:
image: gliderlabs/consul-server:latest
command: "-advertise=${MYHOST} -server -bootstrap"
container_name: consul
hostname: ${MYHOST}
ports:
- 8500:8500
registrator:
image: gliderlabs/registrator:latest
command: "-ip ${MYHOST} consul://${MYHOST}:8500"
container_name: registrator
hostname: ${MYHOST}
depends_on:
- consul
volumes:
- /var/run/docker.sock:/tmp/docker.sock
proxy:
build: .
ports:
- 80:80
depends_on:
- consul
whoami:
image: "traefik/whoami"
deploy:
replicas: 3
ports:
- "80"
Dockerfile for custom proxy image with consul template backed in.用于自定义代理镜像的 Dockerfile,支持 consul 模板。
FROM nginx
RUN curl https://releases.hashicorp.com/consul-template/0.25.1/consul-template_0.25.1_linux_amd64.tgz \
> consul-template_0.25.1_linux_amd64.tgz
RUN gunzip -c consul-template_0.25.1_linux_amd64.tgz | tar xvf -
RUN mv consul-template /usr/sbin/consul-template
RUN rm /etc/nginx/conf.d/default.conf
ADD proxy.conf.ctmpl /etc/nginx/conf.d/
ADD consul-template.hcl /
CMD [ "/bin/bash", "-c", "/etc/init.d/nginx start && consul-template -config=consul-template.hcl" ]
Consul template takes a template file and renders it according to the content of consuls catalog. Consul 模板接受一个模板文件,并根据 consuls 目录的内容进行渲染。
upstream whoami {
{{ range service "whoami" }}
server {{ .Address }}:{{ .Port }};
{{ end }}
}
server {
listen 80;
location / {
proxy_pass http://whoami;
}
}
After the template has been changed, the restart command is executed.更改模板后,执行重新启动命令。
consul {
address = "consul:8500"
retry {
enabled = true
attempts = 12
backoff = "250ms"
}
}
template {
source = "/etc/nginx/conf.d/proxy.conf.ctmpl"
destination = "/etc/nginx/conf.d/proxy.conf"
perms = 0600
command = "/etc/init.d/nginx reload"
command_timeout = "60s"
}
Built In![]() |
HAProxy![]() |
Traefik![]() |
Consul-Template![]() |
|
---|---|---|---|---|
Resolver![]() |
Docker![]() |
Docker![]() |
Docker![]() |
Consul![]() |
Service Discovery![]() |
Automatic![]() |
Server Templates![]() |
Label System![]() |
KV Store + Template ![]() |
Health Checks![]() |
Yes![]() |
Yes![]() |
Yes![]() |
Yes![]() |
Load Balancing![]() |
L4 ![]() |
L4, L7 ![]() |
L4, L7 ![]() |
L4, L7 ![]() |
Sticky Session![]() |
No![]() |
Yes![]() |
Yes![]() |
Depends on proxy![]() |
Metrics![]() |
No![]() |
Stats Page![]() |
Dashboard![]() |
Dashboard![]() |
You can view some of the code samples in more detail on github .您可以在 github 上更详细地查看一些代码示例。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.