使用 multiprocessing.manangers.BaseManager 作为 docker-compose 编排的一部分

Question

I have used docker-compose quite a bit in the past and have enjoyed success leveraging the internal network created as part of a docker-compose orchestration.

I have a very simple example where I have two services:

version: '3'

services:
  intent:
    restart: always
    build: ./dockerfs/intent
    command: gunicorn -w 2 --bind 0.0.0.0:5000 --timeout 999 --log-level debug client:app
    ports: 
      - 8075:5000
  base_s:
    restart: always
    build: ./dockerfs/base

where service base_s is a base_s service by connecting to the BaseManager server "holding" global data in connections :

import json
import uuid
from multiprocessing import Lock
from multiprocessing.managers import BaseManager


connections = {}
lock = Lock()


def get_connection(intent_id):
    with lock:
        if intent_id not in connections:
            print(f"{intent_id} not in connections. creating now...")
            connections[intent_id] = object()
            print(f"addining intent_id {intent_id} to connections")

        return connections[intent_id]


print("starting BaseManager!")
manager = BaseManager(("localhost", 8098), b"password")
manager.register("get_connection", get_connection)
server = manager.get_server()
server.serve_forever()

and service intent is a flask app that "gets" data from the BaseManager service:

import uuid
from multiprocessing.managers import BaseManager
from flask import g, session, Flask, jsonify


app = Flask(__name__)


client_id = uuid.uuid4().hex


def get_client():
    # if not hasattr(g, "rserve"):
    # calling http:<service> as I normally would to "interact" with service in shared network
    manager = BaseManager(("http://base_s", 8098), b"password")  # <<
    manager.register("get_connection")
    manager.connect()
    o = manager.get_connection(client_id)
    print(f"got object using id {o}")

    return f"got it {o}"


@app.route("/", methods=["GET", "POST"])
def client():
    o = get_client()

    return jsonify({"client_id": client_id, "object_id": o}

I can run the above code with success on local host but when I spin up these services using the docker-compose file shared above, I get a intent_1 | socket.gaierror: [Errno -2] Name or service not known intent_1 | socket.gaierror: [Errno -2] Name or service not known .

I was under the impression that I would be able to interact with the BaseManager service since docker-compose maintains an internal network which leads me to think that I maybe do not understand how the BaseManager is served.

Can anyone think of why I am not able to connect to the BaseManager service?

Thanks in advance.

update:

as part of the netstat -a command in the base_s service I see

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 ip-127-0-0-11.ec2:37121 0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8098            0.0.0.0:*               LISTEN     
udp        0      0 ip-127-0-0-11.ec2:53041 0.0.0.0:*   

so it looks like everything is listening as it should.

Answer 1

Got it working.

two changes that I made were:

In base_s service code

1. manager = BaseManager(("localhost", 8098), b"password") to manager = BaseManager(("", 8098), b"password")

this chooses a default address to ensure you are not listening on a loopback address

In client flask api code 2) manager = BaseManager(("http://base_s", 8098), b"password") to

I think the real fix was 2 - I assume that BaseManager does not expect a HTTP connection.