stackoom
  简体   繁体   中英

Prefect scheduling is not working with docker-compose

 原文  2022-11-25 12:14:40  9   1    docker-compose/ scheduling/ prefect

Question

I get an this error while running the code. It says file not found.

Logs:

prefect-docker-compose-database-1  | 
prefect-docker-compose-database-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
prefect-docker-compose-database-1  | 
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.293 UTC [1] LOG:  starting PostgreSQL 14.1 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20211027) 10.3.1 20211027, 64-bit
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.293 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.293 UTC [1] LOG:  listening on IPv6 address "::", port 5432
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.296 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.300 UTC [21] LOG:  database system was shut down at 2022-11-25 11:53:37 UTC
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.305 UTC [1] LOG:  database system is ready to accept connections
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     |  ___ ___ ___ ___ ___ ___ _____    ___  ___ ___ ___  _  _
prefect-docker-compose-orion-1     | | _ \ _ \ __| __| __/ __|_   _|  / _ \| _ \_ _/ _ \| \| |
prefect-docker-compose-orion-1     | |  _/   / _|| _|| _| (__  | |   | (_) |   /| | (_) | .` |
prefect-docker-compose-orion-1     | |_| |_|_\___|_| |___\___| |_|    \___/|_|_\___\___/|_|\_|
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | Configure Prefect to communicate with the server with:
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     |     prefect config set PREFECT_API_URL=http://0.0.0.0:4200/api
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | View the API reference documentation at http://0.0.0.0:4200/docs
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | Check out the dashboard at http://0.0.0.0:4200
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | 
prefect-docker-compose-agent-1     | Starting v2.6.8 agent connected to http://orion:4200/api...
prefect-docker-compose-agent-1     | 
prefect-docker-compose-agent-1     |   ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
prefect-docker-compose-agent-1     |  | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
prefect-docker-compose-agent-1     |  |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
prefect-docker-compose-agent-1     |  |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|
prefect-docker-compose-agent-1     | 
prefect-docker-compose-agent-1     | 
prefect-docker-compose-agent-1     | Agent started! Looking for work from queue(s): demo...
prefect-docker-compose-orion-1     | INFO:     Started server process [7]
prefect-docker-compose-orion-1     | INFO:     Waiting for application startup.
prefect-docker-compose-orion-1     | INFO:     Application startup complete.
prefect-docker-compose-orion-1     | INFO:     Uvicorn running on http://0.0.0.0:4200 (Press CTRL+C to quit)
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55756 - "GET /deployments/name/greetings/get_weather_docker_example HTTP/1.1" 404 Not Found
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55770 - "POST /flows/ HTTP/1.1" 201 Created
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55780 - "GET /block_types/slug/process HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55780 - "PATCH /block_types/9003f2ea-3325-4191-a0a3-67497f162165 HTTP/1.1" 204 No Content
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55780 - "GET /block_schemas/checksum/sha256%3A47c4ac364708f4a6f27fb10b18086ad92c0a53fbbdd9ba07c030467067979f84?version=2.6.8 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55780 - "POST /block_documents/ HTTP/1.1" 201 Created
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55770 - "POST /deployments/ HTTP/1.1" 201 Created
prefect-docker-compose-cli-1 exited with code 0
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46148 - "GET /assets/index.85f05129.css HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "GET /assets/index.f501c99b.js HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "GET /ui-settings HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "GET /health HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "POST /flow_runs/count HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "POST /flow_runs/count HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46162 - "POST /saved_searches/filter HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46148 - "POST /flow_runs/filter HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "POST /ui/flow_runs/history HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46172 - "POST /deployments/filter HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46162 - "POST /work_queues/filter HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46164 - "POST /flows/filter HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46148 - "GET /flows/91f4a98a-877d-4c40-93c7-41236a23d2e5 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46148 - "GET /deployments/c9398a97-4ecc-4b57-8acb-0a50d5a7b328 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46164 - "GET /work_queues/name/demo HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "GET /work_queues/name/demo HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "POST /work_queues/31d37abe-6aee-4674-beaf-4bc8d2cae7c4/get_runs HTTP/1.1" 200 OK
prefect-docker-compose-agent-1     | 11:54:55.107 | INFO    | prefect.agent - Submitting flow run '2f6ba8ff-364c-4ddf-974b-f49712fede9b'
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "POST /flow_runs/2f6ba8ff-364c-4ddf-974b-f49712fede9b/set_state HTTP/1.1" 201 Created
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "GET /deployments/c9398a97-4ecc-4b57-8acb-0a50d5a7b328 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "GET /flows/91f4a98a-877d-4c40-93c7-41236a23d2e5 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "GET /block_documents/7f22b67b-4947-46a1-b0d0-5ae260c71b62?include_secrets=true HTTP/1.1" 200 OK
prefect-docker-compose-agent-1     | 11:54:55.162 | INFO    | prefect.infrastructure.process - Opening process 'heavenly-macaw'...
prefect-docker-compose-agent-1     | 11:54:55.164 | INFO    | prefect.agent - Completed submission of flow run '2f6ba8ff-364c-4ddf-974b-f49712fede9b'
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33902 - "GET /flow_runs/2f6ba8ff-364c-4ddf-974b-f49712fede9b HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33902 - "GET /deployments/c9398a97-4ecc-4b57-8acb-0a50d5a7b328 HTTP/1.1" 200 OK
prefect-docker-compose-agent-1     | 11:54:56.745 | ERROR   | Flow run 'heavenly-macaw' - Flow could not be retrieved from deployment.
prefect-docker-compose-agent-1     | Traceback (most recent call last):
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 255, in retrieve_flow_then_begin_flow_run
prefect-docker-compose-agent-1     |     flow = await load_flow_from_flow_run(flow_run, client=client)
prefect-docker-compose-agent-1     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/contextlib.py", line 222, in __aexit__
prefect-docker-compose-agent-1     |     await self.gen.athrow(typ, value, traceback)
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 218, in asyncnullcontext
prefect-docker-compose-agent-1     |     yield
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
prefect-docker-compose-agent-1     |     return await fn(*args, **kwargs)
prefect-docker-compose-agent-1     |            ^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/site-packages/prefect/deployments.py", line 163, in load_flow_from_flow_run
prefect-docker-compose-agent-1     |     await storage_block.get_directory(from_path=deployment.path, local_path=".")
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/site-packages/prefect/filesystems.py", line 144, in get_directory
prefect-docker-compose-agent-1     |     shutil.copytree(from_path, local_path, dirs_exist_ok=True)
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/shutil.py", line 558, in copytree
prefect-docker-compose-agent-1     |     with os.scandir(src) as itr:
prefect-docker-compose-agent-1     |          ^^^^^^^^^^^^^^^
prefect-docker-compose-agent-1     | FileNotFoundError: [Errno 2] No such file or directory: '/root/flows'

Flow.py: This is flow file which is responsible for deployment scheduling 

from prefect import flow, task
from prefect.deployments import Deployment
from prefect.orion.schemas.schedules import IntervalSchedule, RRuleSchedule, CronSchedule
from datetime import datetime, timedelta



@task
def say_hello(name):
    print(f"hello {name}")

@task
def say_goodbye(name):
    print(f"goodbye {name}")

@flow
def greetings(names=["arthur", "trillian", "ford", "marvin"]):
    for name in names:
        say_hello(name)
        say_goodbye(name)

if __name__ == "__main__":
    deployment = Deployment.build_from_flow(
        name="get_weather_docker_example",  
        schedule = IntervalSchedule(interval=timedelta(seconds=30)),
        flow=greetings,        
        work_queue_name="demo"
    )
    deployment.apply()

docker-compose.yml 

version: "3.9"
services:

  ### Prefect Database
  database:
    image: postgres:14.1-alpine
    restart: always
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=orion
    expose:
      - 5432
    volumes: 
      - db:/var/lib/postgresql/data
    profiles: ["orion"]
  

  ### Prefect Orion API
  orion:
    image: prefecthq/prefect:2.6.8-python3.11
    restart: always
    volumes:
      - prefect:/root/.prefect
    entrypoint: ["prefect", "orion", "start"]
    environment:
      - PREFECT_ORION_API_HOST=0.0.0.0
      - PREFECT_ORION_DATABASE_CONNECTION_URL=postgresql+asyncpg://postgres:postgres@database:5432/orion
    ports:
      - 4200:4200
    depends_on:
      - database
    profiles: ["orion"]

  ## Prefect Agent
  agent:
    image: prefecthq/prefect:2.6.8-python3.11
    restart: always
    entrypoint: ["prefect", "agent", "start", "-q", "demo"]
    environment:
      - PREFECT_API_URL=http://orion:4200/api
    depends_on:
      - orion
    profiles: ["agent"]


  ### Prefect CLI
  cli:
    image: prefecthq/prefect:2.6.8-python3.11
    entrypoint: ["python", "flow.py"]
    working_dir: "/root/flows"
    volumes:
      - "./flows:/root/flows"
    environment:
      - PREFECT_API_URL=http://orion:4200/api
    profiles: ["cli"]


volumes:
  prefect:
  db:
  minio:
networks:
  default:
    name: prefect-network

directory strcture: This is how directory structure looks like and flows directory and docker-compose.yml are in same directory.


- flows
  - flow.py 
- docker-compose.yml are in same directory

Command to run the docker-compose:

docker compose --profile orion --profile agent --profile cli up

1 answers

solution1
 0  2022-12-15 16:13:12

Have you tried deploying your flow codes on storage? https://docs.prefect.io/concepts/storage/?h=storage

You need to upload your flow codes on storage and deploy it on your docker container.

block_prod = RemoteFileSystem(
    basepath="your storage choice path",
    settings=dict(
        token=KEY,
    ),
)
block_prod.save(flow_name, overwrite=True)

deployment = Deployment.build_from_flow(
    name=flow_name,
    flow=flow,
    storage=RemoteFileSystem.load(flow_name),
    infrastructure=DockerContainer(
        image=IMAGE,
        image_pull_policy="IF_NOT_PRESENT",
        networks=["prefect"],
    ),
)
deployment.apply()

暂无
暂无

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

Related Question Scheduling a docker Container with a docker-compose file docker-compose is not working Links working with `docker-compose` but not `docker compose` docker-compose tmpfs not working memcached not working in docker-compose docker-compose working but docker run not docker exec not working in docker-compose containers Docker-compose command not working on Windows 10 docker-compose syslog driver to loggly not working POST requests not working in docker-compose container
Related Tags
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM