从Python脚本读取YAML文件中的值

Question

I have a Python script named app.py that has the value of the project ID,

project_id = "p007-999"

I hard code it inside the .gitlab-ci.yml file provided below,

# list of enabled stages, the default should be built, test, publish
stages:
  - build
  - publish

before_script:
  - export WE_PROJECT_ID="p007-999"
  - docker login -u "$WELANCE_REGISTRY_USER" -p "$WELANCE_REGISTRY_TOKEN" registry.welance.com

build:
  stage: build
  services:
    - docker:dind
  variables:
    DOCKER_HOST: docker:2375
  script:
    - echo $WE_PROJECT_ID
    - cd templates && pwd && yarn install && yarn prod && cd ..
    - docker build -t registry.welance.com/$WE_PROJECT_ID:$CI_COMMIT_REF_SLUG.$CI_COMMIT_SHA -f ./build/ci/Dockerfile .

I would like to automate this. I think the steps for that will be,

a. write the project_id value from the Python script to a shell script variables.sh .
b. In the before_script: of the YML file, execute the variables.sh and read the value from there.

How do I achieve it correctly?

Answer 1

You can do this with ruamel.yaml , which was specfically developed to do these kind of round-trip updates (disclaimer: I am the author of that package).

Assuming your input is:

# list of enabled stages, the default should be built, test, publish
stages:
  - build
  - publish

before_script:
  - PID_TO_REPLACE
  - docker login -u "$WELANCE_REGISTRY_USER" -p "$WELANCE_REGISTRY_TOKEN" registry.welance.com

build:
  stage: build
  services:
    - docker:dind
  variables:
    DOCKER_HOST: docker:2375
  script:
    - echo $WE_PROJECT_ID
    - cd templates && pwd && yarn install && yarn prod && cd ..
    - docker build -t registry.welance.com/$WE_PROJECT_ID:$CI_COMMIT_REF_SLUG.$CI_COMMIT_SHA -f ./build/ci/Dockerfile .

And your code something like:

import sys
from pathlib import Path
import ruamel.yaml


def update_project_id(path, pid):
   yaml = ruamel.yaml.YAML()
   yaml.indent(sequence=4, offset=2) # non-standard indent of 4 for sequences
   yaml.preserve_quotes = True
   data = yaml.load(path)

   data['before_script'][0] = 'export WE_PROJECT_ID="' + pid + '"'
   yaml.dump(data, path)

file_name = Path('.gitlab-ci.yml')
project_id = "p007-999"

update_project_id(file_name, project_id)

which gives as output:

# list of enabled stages, the default should be built, test, publish
stages:
  - build
  - publish

before_script:
  - export WE_PROJECT_ID="p007-999"
  - docker login -u "$WELANCE_REGISTRY_USER" -p "$WELANCE_REGISTRY_TOKEN" registry.welance.com

build:
  stage: build
  services:
    - docker:dind
  variables:
    DOCKER_HOST: docker:2375
  script:
    - echo $WE_PROJECT_ID
    - cd templates && pwd && yarn install && yarn prod && cd ..
    - docker build -t registry.welance.com/$WE_PROJECT_ID:$CI_COMMIT_REF_SLUG.$CI_COMMIT_SHA -f ./build/ci/Dockerfile .

(including the comment, which you lose when using by most other YAML loader/dumpers)

Answer 2

This is almost definitely inappropriate, but I really can't help myself.

WARNING: This is destructive, and will overwrite .gitlab-ci.yml .

awk '
  NR==FNR && $1=="project_id" {pid=$NF}
  /WE_PROJECT_ID=/ {sub(/\".*\"/, pid)}
  NR!=FNR {print > FILENAME}
' app.py .gitlab-ci.yml

1. In the first file only, assign the last column to pid only if the first column is exactly "project_id".

2. On any line in any file that assigns the variable WE_PROJECT_ID , replace the first quoted string with pid .

3. In any files other than the first, print all records to the current file. This is possible due to awk 's nifty buffers. If you have to be told to make a back-up, don't run this.