Flask async job/task submit to Celery or Jenkins

Question

I have a flask application where i submit tasks to celery(worker) to execute it. so that i can get the webpage back after submitting. Can i achieve same if i submit the task to Jenkins instead? Just wanted an opinion why would i use celery when i can ask Jenkins to schedule/execute the job through Jenkins API ? still get my webpage back. I may be wrong with my approach but anyone who can shed light on this would really appreciate.

The Main aim is that user submits the form which actually is task to execute and after hitting submit task detachs from web , reloads the form. Meanwhile task runs in background which celery does it efficiently but can it be done via jenkins.

Thanks

Answer 1

@Macintosh_89 - adding my (very late) answer for posterity's sake.

Everyone speaks HTTP, including Jenkins, so your architecture proposal is doable. (Unless there is some specific feature in Celery that makes it a winning solution for you, there's no reason you cannot achieve the same with, eg, Jenkins.) I recommend utilizing Jenkins' "trigger builds remotely" option, which provides you with a webhook.

Making an appropriate HTTP request to that webhook will trigger the build job to run. (On a side note: be sure to set the "Authentication Token" to a lengthy, strong value.)

As a bonus, I'm including a bash script that I wrote to test calls to a Jenkins job with "trigger builds remotely" enabled.

#!/bin/bash

# Doing an HTTP job build request with Jenkins is a two-part deal.
#
# 1. Get a crumb using auth credentials
# 2. Use that crumb (along with auth credentials) to submit the job
#
# Reference:
# www.inanzzz.com/index.php/post/jnrg/running-jenkins-build-via-command-line

set -u
set -o pipefail

# -------------------------------------------------------------------------- #
#                       VARIABLE DEFINITION
# -------------------------------------------------------------------------- #

PATH=/usr/bin:/bin

readonly _srv="${JENKINS_SERVER:-x}"
readonly _job_path="${JENKINS_JOB_PATH:-x}"
readonly _token="${JENKINS_JOB_TOKEN:-x}"
readonly _u="${JENKINS_SERVICE_ACCOUNT:-x}"
readonly _p="${JENKINS_SERVICE_PASSWORD:-x}"

readonly _crumb_path='crumbIssuer/api/xml'
readonly _crumb_query='xpath=concat(//crumbRequestField,":",//crumb)'
readonly _uri1="https://${_srv}/${_crumb_path}?${_crumb_query}"

readonly _token_query="token=${_token}"
readonly _uri2="https://${_srv}/${_job_path}?${_token_query}"

readonly _output="$(mktemp)"

# -------------------------------------------------------------------------- #
#                       FUNCTIONS
# -------------------------------------------------------------------------- #

errout() {
        local _msg="${0##*/} error: ${1}"
        printf '%s\n' "${_msg}"
        exit 1
}

cleanup() {
        rm -f "${_output}"
}

audit_env() {
        [ "${_srv}" == "x" ] ||                                         \
        [ "${_job_path}" == "x" ] ||                                    \
        [ "${_token}" == "x" ] ||                                       \
        [ "${_u}" == "x" ] ||                                           \
        [ "${_p}" == "x" ]

        if [ ${?} -eq 0 ] ; then
                errout "Missing ENV. Hint: grep -Eo 'JENKINS_[A-Z_]+' ${0}"
        fi
}

req_crumb() {
        curl                                                            \
        --silent                                                        \
        --show-error                                                    \
        --user "${_u}:${_p}"                                            \
        --request 'GET'                                                 \
        --output "${_output}"                                           \
        "${_uri1}"

        if [ ${?} -ne 0 ] ; then
                errout 'Problem with HTTP crumb request'
        fi
}

get_value_into_crumb_var() {
        local _s

        read -r _s < "${_output}" || [ -n "${_s}" ]

        if [ ${?} -ne 0 ] ; then
                errout "Problem reading crumb from ${_output}"
        fi

        readonly _crumb="${_s}"
}

req_job_launch() {
        curl                                                            \
        --silent                                                        \
        --show-error                                                    \
        --user "${_u}:${_p}"                                            \
        --header "${_crumb}"                                            \
        --request 'POST'                                                \
        "${_uri2}"

        if [ ${?} -ne 0 ] ; then
                errout 'Problem with HTTP job run request'
        fi
}

# -------------------------------------------------------------------------- #
#                       CLEANUP AND SIGNAL HANDLING
# -------------------------------------------------------------------------- #

trap 'cleanup' EXIT
trap 'exit 2' HUP INT QUIT TERM

# -------------------------------------------------------------------------- #
#                       MAIN LOGIC
# -------------------------------------------------------------------------- #

audit_env

req_crumb
get_value_into_crumb_var
req_job_launch

exit 0

If you'd like to test using this, just set each of the following in your environment first:

# automatically generated in build config under "Use the
# following to trigger build job remotely"
export JENKINS_JOB_PATH='job/Folder/job/aa/buildWithParameters'
# the "Authentication Token" set by you in build config
export JENKINS_JOB_TOKEN='xxxyyyzzzaaabbbccc'
# hostname of your build server
export JENKINS_SERVER='my-ci-system.some.org'
# account and password that are authorized to call webhook;
# if you're using "Project-based Matrix Authorization Strategy",
# this account will need READ for Overall, Job, and Run authz
export JENKINS_SERVICE_ACCOUNT='user'
export JENKINS_SERVICE_PASSWORD='secret'

Of course, you can implement something similar in your language of choice. If you'd like to see what's happening here more closely (especially WRT how the URIs are being built at runtime), then just run it using bash -x .

Answer 2

Couple of points below to consider to compare between celery and jenkins.

• Celery is specifically designed and built for running resource intensive tasks in background whereas jenkins is a more general tool for automation.

• jenkins is built on java so native integration is there although plugins are available whereas celery is built with python so you can directly program the tasks in python and send it to celery or just call your shell tasks from python.

• Message queuing - Again jenkins does not have builtin support for message brokers so queuing might be difficult for you. Celery uses rabbitmq as default to queue the tasks so your tasks never gets lost.

• Celery also provide simple callbacks so when task is completed you can run some function after it.

• Now if you ask about cpu comsumption then celery is not at all heavy