How to run pylint only on changed files in Gitlab?

Question

I'm trying to run pylint only on changed python files, but my build keeps failing. I have extracted the changed files through git diff and saved them in a variable, but when I inject the variable into the pylint call, it fails. It works fine with a hardcoded filename however. Here is my yaml:

pylint:
stage: test
  before_script:
    - pip install pylint pylint-exit anybadge
  script:
      - echo CI_COMMIT_SHA=${CI_COMMIT_SHA}
      - echo CI_MERGE_REQUEST_TARGET_BRANCH_NAME=${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}
      - git fetch origin ${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}
      - FILES=$(git diff --name-only ${CI_COMMIT_SHA} origin/${CI_MERGE_REQUEST_TARGET_BRANCH_NAME} | grep '\.py'$)
      - echo "Changed files are $FILES"
      - pylint --output-format=text $(find -type f -name "$FILES" ! -path "**/.venv/**") | tee ./pylint/pylint.log || pylint-exit $?
      - PYLINT_SCORE=$(sed -n 's/^Your code has been rated at \([-0-9.]*\)\/.*/\1/p' ./pylint/pylint.log)
      - anybadge --label=Pylint --file=pylint/pylint.svg --value=$PYLINT_SCORE 2=red 4=orange 8=yellow 10=green
      - echo "Pylint score is $PYLINT_SCORE"
  artifacts:
    paths:
      - ./pylint/
    when: always
  only:
      refs:
          - merge_requests
      changes:
          - "**/*.py"

Answer 1

Darker permits to do that. It supports black, isort, mypy, pylint, and flake8 in february 2021.

This utility reformats and checks Python source code files in a Git repository. However, it only applies reformatting and reports errors in regions which have changed in the Git working tree since the last commit.

Answer 2

Found a way to do it. I the GitLab variables to get changed files in the merge request and input that into the pylint command

 script:
      - echo CI_COMMIT_SHA=${CI_COMMIT_SHA}
      - echo CI_MERGE_REQUEST_TARGET_BRANCH_NAME=${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}
      - git fetch origin ${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}
      - FILES=$(git diff --name-only ${CI_COMMIT_SHA} origin/${CI_MERGE_REQUEST_TARGET_BRANCH_NAME} | grep '\.py'$)
      - echo "Changed files are $FILES"
      - mkdir ./pylint
      - pylint --output-format=text $FILES | tee ./pylint/pylint.log || pylint-exit $?

Answer 3

This is what you can do

.gitlab-ci.yml

stages:
  - Lint

Lint:
  stage: Lint
  allow_failure: true
  script:
  - chmod +x lint.sh
  - ./lint.sh

lint.sh

#! /bin/sh

pip install pycodestyle
current_branch="$CI_BUILD_REF_NAME" 
echo $current_branch
all_changed_files=$(git diff --name-only origin/master origin/$current_branch)
echo "Checking changes!"
for each_file in $all_changed_files
do
# Checks each newly added file change with pycodestyle
pycodestyle $each_file
error_count=$(pycodestyle $each_file --count | wc -l)
if [ $error_count -ge 1 ]; then
    exit 1
fi
if [ $error_count -eq 0 ]; then
    exit 0
fi
done
echo "Completed checking"

Answer 4

While qathulu answer is adequate, it does not work if no python files are changed. The below prints all the changed files, then use grep to find the changed python files.

Unfortunately there is an error on gitlab, where assigning empty output from grep ends the pipeline. In order to fix that, grep is not assigned to variable until we are sure at least one python file is changed.

- echo CI_COMMIT_SHA=${CI_COMMIT_SHA}
- echo CI_MERGE_REQUEST_TARGET_BRANCH_NAME=${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}
- git fetch origin ${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}
- tmp_files=$(git diff --name-only ${CI_COMMIT_SHA} origin/${CI_MERGE_REQUEST_TARGET_BRANCH_NAME})
- echo "Changed files are $tmp_files"
- if [ -z "$(echo "$tmp_files" | grep "\.py")" ]; then exit 0; else echo "Python files are found"; fi
- tmp_pfiles=$(echo "$tmp_files" | grep "\.py")
- echo "Python files are $tmp_pfiles"
- mkdir ./pylint
- pylint --output-format=text $tmp_pfiles | tee ./pylint/pylint.log || pylint-exit $?