为不同的 GitHub Actions 作业共享相同的步骤

Question

I have a cross-platform project which is to be built on 2 platforms: mac and linux(ubuntu).

My pipeline contains 3 jobs:

1. prepare docker image with all nessesary too to build the project.
2. build on ubuntu in prepared docker container, depends on step 1
3. build on MacOS, needs nothing

Steps for linux and macos are definitely the same. But matrixes differs much, and linux build is run inside container.

Is there a way to share steps between two different jobs?

I tried YAML anchors but GitHub does not support them.

Full workflow

on:
  push:
    branches: [ main, support/1.2.x ]
  pull_request:
    branches: [ main, support/1.2.x ]

jobs:
  Docker-iroha-builder:
    runs-on: ubuntu-latest
    steps:
      -
        name: Checkout
        uses: actions/checkout@v2
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      -
        name: Cache Docker layers
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-
      -
        name: Login to DockerHub
        uses: docker/login-action@v1 
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      -
        name: Build and push
        uses: docker/build-push-action@v2
        with:
          file: docker/develop/Dockerfile.builder
          # context: .
          push: true
          tags: ${{ secrets.DOCKERHUB_ORG }}/iroha:builder
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache-new
      -
        # Temp fix
        # https://github.com/docker/build-push-action/issues/252
        # https://github.com/moby/buildkit/issues/1896
        name: Move cache
        run: |
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache


  build-iroha-ubuntu:
    needs: Docker-iroha-builder
    runs-on: ubuntu-latest
    container: ikyb/iroha:builder
    strategy:
      fail-fast: false
      matrix:
        cc: [ gcc-9, gcc-10, clang ]  ##todo g++-10
        USE_BURROW: [ -DUSE_BURROW=OFF ]
        debrel: [ Debug ] #,Release, RelWithDebInfo
    steps:
      - ## Takes 22 seconds with default github runner
        name: Homebrew
        run: brew install cmake ninja coreutils
        if: ${{ runner.os == 'MacOS' }}
      -
        name: Checkout
        uses: actions/checkout@v2
      -
        name: Cache vcpkg
        uses: actions/cache@v2
        with:
          path: |
            build-vcpkg
            build/vcpkg_installed
            $HOME/.cache/vcpkg
          key:          ${{ runner.os }}-vcpkg-${{ github.sha }}
          restore-keys: ${{ runner.os }}-vcpkg-
      - 
        name: Build Iroha vcpkg dependancies
        run: ./vcpkg/build_iroha_deps.sh $PWD/build-vcpkg
      - 
        name: CMake configure
        run: |
          export CC=${{ matrix.cc }} CXX=$(echo ${{ matrix.cc }} | sed -es,gcc,g++, -es,clang,clang++,)
          cmake -B build -DCMAKE_TOOLCHAIN_FILE=$PWD/build-vcpkg/scripts/buildsystems/vcpkg.cmake \
             ${{ matrix.USE_BURROW }} -GNinja #-DCMAKE_VERBOSE_MAKEFILE=ON
      -
        name: CMake build
        run: cmake --build build --config ${{ matrix.debrel }}

  build-iroha-macos:
    runs-on: macos-latest
    strategy:
      fail-fast: false
      matrix:
        USE_BURROW: [ -DUSE_BURROW=OFF ]
        debrel: [ Debug,Release ]
    steps:
      - ## Takes 22 seconds with default github runner
        name: Homebrew
        run: brew install cmake ninja coreutils
        if: ${{ runner.os == 'MacOS' }}
      -
        name: Checkout
        uses: actions/checkout@v2
      -
        name: Cache vcpkg
        uses: actions/cache@v2
        with:
          path: |
            build-vcpkg
            build/vcpkg_installed
            $HOME/.cache/vcpkg
          key:          ${{ runner.os }}-vcpkg-${{ github.sha }}
          restore-keys: ${{ runner.os }}-vcpkg-
      - 
        name: Build Iroha vcpkg dependancies
        run: ./vcpkg/build_iroha_deps.sh $PWD/build-vcpkg
      - 
        name: CMake configure
        run: |
          export CC=${{ matrix.cc }} CXX=$(echo ${{ matrix.cc }} | sed -es,gcc,g++, -es,clang,clang++,)
          cmake -B build -DCMAKE_TOOLCHAIN_FILE=$PWD/build-vcpkg/scripts/buildsystems/vcpkg.cmake \
             ${{ matrix.USE_BURROW }} -GNinja #-DCMAKE_VERBOSE_MAKEFILE=ON
      -
        name: CMake build
        run: cmake --build build --config ${{ matrix.debrel }}

Answer 1

TL;DR

I solved my problem with shell tool yq

yq eval 'explode(.)' file.yml

Long answer

GitHub Workflow description in YAML does not support anchors. There are several workarounds => anyway they come to building-editing workflow yaml from source. So I suggest yet another one make-workflows.sh based on YAML tool yq .

USAGE

0. Move your workflows to .github/*.src.yml
1. Put make-workflows.sh to directory .github/
2. (optional) Copy or link pre-commit.sh to .git/hooks/pre-commit Like ln -s../../.github/pre-commit.sh.git/hooks/pre-commit

---

File make-workflows.sh

#!/usr/bin/env bash
set -euo pipefail

## The script expands '*.src.yml' from $1(default: script's directory) 
## to $2 (default:subdirectory 'workflows') with corresponding name '*.yml'
## Main goal is to dereference YAML anchors.
## Deals only with Git cached/indexed files
## Set -x to debug

script_dir=$(dirname $(realpath "$0"))
dir_from=${1:-${script_dir}}
dir_to=${2:-workflows}
cd $dir_from

edited=
for f in $(git status -s -- \*.src.yml | sed 's,^.. ,,') ;do
    readonly out=$(echo $f | sed s,.src.yml\$,.yml,)
    readonly wout=$dir_to/$out
    readonly tempout=$(mktemp)
    trap "rm -f $tempout" EXIT
    echo >>$tempout "## DO NOT EDIT"
    echo >>$tempout "## Generated from $f with $(basename $0)"
    echo >>$tempout ""
    yq eval 'explode(.)' $f >>$tempout
    if ! diff -q $wout $tempout &>/dev/null ;then
        mv $tempout $wout
        edited+="'$out' "
    fi
done

if [[ -n "$edited" ]] 
then echo >&2 "make-workflows: these files were edited: $edited"
else echo >&2 "make-workflows: everything is up to date"
fi

---

File pre-commit.sh

#!/usr/bin/env bash
set -euo pipefail

gitroot=$(git rev-parse --show-toplevel)

cd $gitroot
./.github/make-workflows.sh
git add .github/workflows

Links

1. Share same steps for different GitHub Actions jobs
2. https://github.community/t/support-for-yaml-anchors/16128/60
3. https://github.com/mithro/actions-includes
4. https://github.com/allejo/gha-workflows

Answer 2

While github actions does not support YAML anchors directly, one can expand those eg by converting from YAML to JSON and then back to YAML. I am doing this here ( Makefile in .github/workflows ): https://github.com/agda/agda/blob/557681d04aae2100ccde2e045a8afcf30528c3a5/.github/workflows/Makefile

srcpath=../../src/github/workflows
sources=$(wildcard $(srcpath)/*.yml $(srcpath)/*.yaml)
targets=$(sort $(notdir $(sources)))

all : $(targets)

# Normalize YAML files by going via JSON.
# This expands anchors which are not understood by github workflows.

% : $(srcpath)/% 
    yaml2json $< | json2yaml - > $@

An example for a workflow file with anchors is here: https://github.com/agda/agda/blob/557681d04aae2100ccde2e045a8afcf30528c3a5/src/github/workflows/test.yml

jobs:
  build:
    runs-on: &runs_on ubuntu-22.04
    steps:
    - &checkout
      uses: actions/checkout@v3

    - &haskell_setup
      uses: haskell/actions/setup@v2
      with:
        ghc-version: ${{ env.GHC_VER }}
...
  test:
    needs: build
    runs-on: *runs_on

    steps:
    - *checkout
    - *haskell_setup
...