cURL command to GitLab API in GitLab pipeline succeeds with [0 bytes data] when cURL contains variable

Question

I have a GitLab pipeline that should generate a file last_changes.txt containing a checksum of a directory structure of myrepository and commit the file to a new branch in myrepository . myrepository is a different repository than the gitlab pipeline is running in.

The checksum is buildet with cksum databases/* | sort cksum databases/* | sort and stored in a variable. This variable is then submitted in the cURL command to the GitLab API to update an existing file in a repository ( https://docs.gitlab.com/ee/api/repository_files.html#update-existing-file-in-repository ).

The pipeline looks like this:

write-status:
  stage: post-build
  image: myrepo.domain.com/myimage
  script:
    - git clone --branch $CI_COMMIT_BRANCH https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.domain.com/project/myrepository.git
    - cd myrepository
    - |
      CHANGED_FILES=$(cksum databases/* | sort) 
      echo $CHANGED_FILES
      curl -v -w --request PUT --header 'PRIVATE-TOKEN: myPrivateToken' \
      --header "Content-Type: application/json" \
      --data "{\"branch\":\"newchanges\", \"start_branch\":\"main\", \"content\":\"${CHANGED_FILES}\", \"commit_message\":\"update file with checksum\"}" \
      "https://gitlab.domain.com/api/v4/projects/2808/repository/files/ressources%2Flast_changes1%2Etxt"    
  when: on_success

If I execute the commands from my local Git Bash, the commit works and the branch is being created. If I execute the commands stored as a shell script on localhost, the commit works as well and the branch is being created. However, if the same commands are executed in the GitLab pipeline in the script part (as shown in the code block above), the cURL command succeeds with [0 bytes data] but neither the commit nor the branch are created (output shown in the image below). This failure only seems to happen when the JSON content is added as a variable in the cURL command (cURL containing a variable). If the JSON content is a static string, everything works fine.

This error seems to happen with different linux distributions (tested with alpine 3.16 and rhel 8 docker images in the GitLab pipeline).

Is there a way to make GitLab pipelines accept variables in cURL commands?

GitLab Pipeline Status with 0 bytes data

Answer 1

Interesting issue.

I setup a test repo in which I was able to work around this issue by replacing the JSON-bash-escaping with a form which is - IMO - way easier to handle.

Sidenote: The -w in your curl command requires an argument if I'm not mistaken and might be part of the issue.

commit_variable:
  stage: build
  script: - |
        CHANGED_FILES=$(cksum /usr/bin/* | sort)
        echo $CHANGED_FILES
        curl -v --request PUT --header "PRIVATE-TOKEN: ${TOKEN}" \
            -F "branch=main" \
            -F "content=${CHANGED_FILES}" \
            -F "commit_message=changed files as variable" \
            "https://gitlab.com/api/v4/projects/38716506/repository/files/as_variable%2Etxt"

• CI job: https://gitlab.com/computerfoo/apitest/-/jobs/2905788304
• Commit: https://gitlab.com/computerfoo/apitest/-/commit/268457956218a1dbb43a9550f087debeeaec7e3c
• File: https://gitlab.com/computerfoo/apitest/-/blob/main/as_variable.txt

---

As an additional test I replaced the environment variable CHANGED_FILES with a file but the results were the same.

commit_file:
  stage: build
  script: - |
        cksum /usr/bin/* | sort > /tmp/changed_files.txt
        cat /tmp/changed_files.txt
        curl -v --request PUT --header "PRIVATE-TOKEN: ${TOKEN}" \
            -F "branch=main" \
            -F "content=</tmp/changed_files.txt" \
            -F "commit_message=changed files as file" \
            "https://gitlab.com/api/v4/projects/38716506/repository/files/as_file%2Etxt"

• CI job: https://gitlab.com/computerfoo/apitest/-/jobs/2905788305
• Commit: https://gitlab.com/computerfoo/apitest/-/commit/2694108c668c8d6077fbddf467e08988327f23be
• File: https://gitlab.com/computerfoo/apitest/-/blob/main/as_file.txt

---

See the complete .gitlab-ci.yml file.