Editing docker-compose.yml with PyYAML

Question

I have a pretty standard docker-compose.yml and I need to edit the password of the database programmatically.

Since its a YAML file, I thought it would be simple to edit and dump the content. So far I tried the PyYAML and it just mess the docker-compose file and I don't know why.

Loading and dumping the same content, it breaks the structure.

Content of the docker-compose.yml:

version: '2'
services:
  web:
    container_name: xxx
    ports:
     - "80:80"
    volumes:
      - .:/xxx
    depends_on:
      - mysql
    build: .
  mysql:
    ports:
     - "32768:32768"
     - "3306:3306"
    container_name: xxx-mysql
    restart: always
    image: mariadb:latest
    environment:
      MYSQL_ROOT_PASSWORD: 'thiswillbechangeonsetupscript'
      MYSQL_DATABASE: 'xxxdb'
    volumes:
     - ./database:/var/lib/mysql
    ports:
      - "3306:3306"

This is how I'm loading and dumping the content:

import yaml

with open("docker-compose.yml", 'r') as ymlfile:
    docker_config = yaml.load(ymlfile)

with open("docker-compose.yml", 'w') as newconf:
    yaml.dump(docker_config, newconf)

And this is how the file is saved.

services:
  mysql:
    container_name: xxx-mysql
    environment: {MYSQL_DATABASE: xxxdb, MYSQL_ROOT_PASSWORD: thiswillbechangeonsetupscript}
    image: mariadb:latest
    ports: ['3306:3306']
    restart: always
    volumes: ['./database:/var/lib/mysql']
  web:
    build: .
    container_name: xxx
    depends_on: [mysql]
    ports: ['80:80']
    volumes: ['.:/xxx']
version: '2'

Is there any better way to do this?! What I'm missing?

Answer 1

The default dump for PyYAML is to use flow style for leaf nodes ( [....] for sequences, {...} for mappings), so the minimum you should do is specify yaml.dump(....., default_flow_style=False)

Then the YAML specifications state that the order of the keys is not guaranteed, and what you see is that PyYAML dumps them in sorted order.

I can recommend to use ruamel.yaml (disclaimer: I am the author of that package), which had the specific goal to allow this kind of round-tripping with minimal, changes compared to the input, often none at all. Including key ordering, flow vs block style, quotes on strings etc.

There is another reason to use ruamel.yaml : if you run this program on your input:

import sys
import ruamel.yaml

yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
yaml.indent(sequence=3, offset=1)

with open("docker-compose.yml", 'r') as ymlfile:
    data = yaml.load(ymlfile)
yaml.dump(data, sys.stdout)

you'll get a DuplicateKeyError :

ruamel.yaml.constructor.DuplicateKeyError: while constructing a mapping
  in "docker-compose.yml", line 13, column 5
found duplicate key "ports" with value "[]" (original value: "[]")
  in "docker-compose.yml", line 24, column 5

because ports occurs two times as key in the mapping that is the value for the key mysql . This is not allowed according to the YAML specifications (the old 1.1 that PyYAML fallows and the newer 1.2), but PyYAML silently disposes of the first key-value pair, leaving port 32768 unmapped.

After deleting the last two lines from your input, the output of the program is:

version: '2'
services:
  web:
    container_name: xxx
    ports:
     - "80:80"
    volumes:
     - .:/xxx
    depends_on:
     - mysql
    build: .
  mysql:
    ports:
     - "32768:32768"
     - "3306:3306"
    container_name: xxx-mysql
    restart: always
    image: mariadb:latest
    environment:
      MYSQL_ROOT_PASSWORD: 'thiswillbechangeonsetupscript'
      MYSQL_DATABASE: 'xxxdb'
    volumes:
     - ./database:/var/lib/mysql

which is hopefully close enough for your intended purpose.

Please note that PyYAML remove the quotes in - "80:80" , which is fine because 80:80 cannot be falsely interpreted as a sexagesimal, but if you do something with port 25 changing - 80:80 to - 25:25 is vastly different when using a YAML 1.1 parser like PyYAML (as docker-compose does) from - "25:25" (the former equalling - 1525`)

Based on this I have made a utility ruamel.dcw , that uses this functionality to pre-process docker compose files, allowing defaults for environment variables (in case they are not set) and a few other tricks, writes out a temporary file and then calls docker-compose -f tmpfile , you should use a similar technique, disposing of your temporary file after the run.

Answer 2

You need to add default_flow_style=False when you write the yaml:

import yaml

with open("docker-compose.yml", 'r') as ymlfile:
    docker_config = yaml.load(ymlfile)

with open("docker-compose_new.yml", 'w') as newconf:
    yaml.dump(docker_config, newconf, default_flow_style=False)

Then you will get the following as output which, except using alphabetic order to write the lines, is similar to your input:

services:
  mysql:
    container_name: xxx-mysql
    environment:
      MYSQL_DATABASE: xxxdb
      MYSQL_ROOT_PASSWORD: thiswillbechangeonsetupscript
    image: mariadb:latest
    ports:
    - 3306:3306
    restart: always
    volumes:
    - ./database:/var/lib/mysql
  web:
    build: .
    container_name: xxx
    depends_on:
    - mysql
    ports:
    - 80:80
    volumes:
    - .:/xxx
version: '2'

Please note that, in your original docker-compose.yaml you are declaring the ports variable twice, so yaml parser will only consider the last variable. To fix that, remove the following lines:

ports:
  - "3306:3306"

Then, running the write operation as explained above gives the following ouput:

services:
  mysql:
    container_name: xxx-mysql
    environment:
      MYSQL_DATABASE: xxxdb
      MYSQL_ROOT_PASSWORD: thiswillbechangeonsetupscript
    image: mariadb:latest
    ports:
    - 32768:32768
    - 3306:3306
    restart: always
    volumes:
    - ./database:/var/lib/mysql
  web:
    build: .
    container_name: xxx
    depends_on:
    - mysql
    ports:
    - 80:80
    volumes:
    - .:/xxx
version: '2'