Ansible stdout empty or blank on some native commands but works fine in Ansible Tower

Question

I have an ansible playbook that loops a registered variable outputting the .stdout which works perfectly in ansible tower but stdout is blank or empty when I run it natively.

uname -a: Linux ******** 3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 9 16:09:48 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

(not sure if this is relevant to the original question but I'm including it because it might be)

While testing and checking syntax I also tried several shell commands natively on our three ansible test machines and discovered that most commands correctly capture stdout but a few don't. For instance "ls" and "whoami" work correctly and "who am i" doesn't but they all work properly from the command line. Also some of the shell commands involve piping and other don't.

I've also tried these in my ansible.cfg to no avail.

callback_whitelist = profile_tasks
callbacks_enables = ansible.posix.profile_tasks
callback = skippy

I wouldn't mind providing the entire code but since it works perfectly in ansible tower I'm not sure it's relevant. He're the relevant part(s).

- name: Looped results.
  debug:
    msg: "{{  item.item }} - {{ item.stdout }}."
  loop: "{{ pipe_result.results }}"
  loop_control: 
    label: ""
  # when: item.stdout != ""

I've also tested with when: item.stdout != "" and it's apparently not blank (but seems so to my human eyes). Notice stdout = "Copy " not all are zero but the first example loop is.

Output from Ansible Tower: (again only included one loops output for brevity)

{
    "msg": {
        "msg": "All items completed",
        "changed": true,
        "results": [
            {
                "stderr_lines": [],
                "ansible_loop_var": "item",
                "end": "2022-07-19 21:24:00.824032",
                "stderr": "",
                "stdout": "COPY 0",
                "changed": true,
                "rc": 0,
                "failed": false,
                "cmd": "awk -vRS='\\r\\n' 'FNR==1{if (match($0,/^\\\"loginName\\\",\\\"externalUserId\\\"/)) {t=1;} else {if (match($0,/^\\\"User Name\\\",\\\"First Name\\\"/)) t=2; else t=3; } next } {it=split(FILENAME,array,\"/\"); suffix=\",\\\"\" array[it] \"\\\",\\\"\" substr(FILENAME,index(FILENAME,\"I_\")+2,index(FILENAME,\".\")-1-index(FILENAME,\"I_\")-1) \"\\\"\"; } {split($0,line,\",\")} {if (t==3) next} {if (t==2) gensub(/\\([^\\\"]\\),/,\"/1\\\\\\\\,\",\"g\",line[1] \",\" line[1] \",\" line[2] \",\" line[4] \",\" line[5] \",\\\"\\\",\" line[17] \",\" line[18] \",\" line[9] \",\" line[24] \",\" line[23]\",\" line[19] \",\" line[21] \",\" line[12] \",\" line[20]\",\" line[11] \",\" line[03] \",\" line[22] \",\" line[25] \",\" line[28] \",\" line[06]) suffix} {if (t==1) print gensub(/[^\\\"],/,\"\\\\\\\\,\",\"g\",$0) suffix }' /tmp/percipiodata/********.csv | psql -h 10.18.3.4 postgres ansibleadmin@db-ansible-postgres -c 'copy percipio_users (login_name, external_userid, first_name, last_name, email, active_status, department_id, department_name, employee_id, full_time, hire_date, job_code, job_title, manager_employee_id, manager_level, manager_name, middle_name, pay_group, position_id, supervisor_email, sys_location, file_name, file_date) from stdin WITH csv'",
                "item": "********.csv",
                "delta": "0:00:00.288567",
                "invocation": {
                    "module_args": {
                        "creates": null,
                        "executable": null,
                        "_uses_shell": true,
                        "strip_empty_ends": true,
                        "_raw_params": "awk -vRS='\\r\\n' 'FNR==1{if (match($0,/^\\\"loginName\\\",\\\"externalUserId\\\"/)) {t=1;} else {if (match($0,/^\\\"User Name\\\",\\\"First Name\\\"/)) t=2; else t=3; } next } {it=split(FILENAME,array,\"/\"); suffix=\",\\\"\" array[it] \"\\\",\\\"\" substr(FILENAME,index(FILENAME,\"I_\")+2,index(FILENAME,\".\")-1-index(FILENAME,\"I_\")-1) \"\\\"\"; } {split($0,line,\",\")} {if (t==3) next} {if (t==2) gensub(/\\([^\\\"]\\),/,\"/1\\\\\\\\,\",\"g\",line[1] \",\" line[1] \",\" line[2] \",\" line[4] \",\" line[5] \",\\\"\\\",\" line[17] \",\" line[18] \",\" line[9] \",\" line[24] \",\" line[23]\",\" line[19] \",\" line[21] \",\" line[12] \",\" line[20]\",\" line[11] \",\" line[03] \",\" line[22] \",\" line[25] \",\" line[28] \",\" line[06]) suffix} {if (t==1) print gensub(/[^\\\"],/,\"\\\\\\\\,\",\"g\",$0) suffix }' /tmp/percipiodata/NTTA_I_20210818.csv | psql -h 10.18.3.4 postgres ansibleadmin@db-ansible-postgres -c 'copy percipio_users (login_name, external_userid, first_name, last_name, email, active_status, department_id, department_name, employee_id, full_time, hire_date, job_code, job_title, manager_employee_id, manager_level, manager_name, middle_name, pay_group, position_id, supervisor_email, sys_location, file_name, file_date) from stdin WITH csv'",
                        "removes": null,
                        "argv": null,
                        "warn": true,
                        "chdir": null,
                        "stdin_add_newline": true,
                        "stdin": null
                    }
                },
                "stdout_lines": [
                    "COPY 0"
                ],
                "start": "2022-07-19 21:24:00.535465"
            },
... chop, chop, chop ...
        ]
    },
    "changed": false,
    "_ansible_verbose_always": true,
    "_ansible_no_log": false
}

Native output:

TASK [Debug data piping] **************************************************************************************************************************************************************************
Wednesday 20 July 2022  15:21:33 -0500 (0:00:02.662)       0:00:25.941 ******** 
ok: [localhost] => {
    "msg": {
        "changed": true, 
        "msg": "All items completed", 
        "results": [
            {
                "ansible_loop_var": "item", 
                "changed": true, 
                "cmd": "awk -vRS='\\r\\n' 'FNR==1{if (match($0,/^\\\"loginName\\\",\\\"externalUserId\\\"/)) {t=1;} else {if (match($0,/^\\\"User Name\\\",\\\"First Name\\\"/)) t=2; else t=3; } next } {it=split(FILENAME,array,\"/\"); suffix=\",\\\"\" array[it] \"\\\",\\\"\" substr(FILENAME,index(FILENAME,\"I_\")+2,index(FILENAME,\".\")-1-index(FILENAME,\"I_\")-1) \"\\\"\"; } {split($0,line,\",\")} {if (t==3) next} {if (t==2) gensub(/\\([^\\\"]\\),/,\"/1\\\\\\\\,\",\"g\",line[1] \",\" line[1] \",\" line[2] \",\" line[4] \",\" line[5] \",\\\"\\\",\" line[17] \",\" line[18] \",\" line[9] \",\" line[24] \",\" line[23]\",\" line[19] \",\" line[21] \",\" line[12] \",\" line[20]\",\" line[11] \",\" line[03] \",\" line[22] \",\" line[25] \",\" line[28] \",\" line[06]) suffix} {if (t==1) print gensub(/[^\\\"],/,\"\\\\\\\\,\",\"g\",$0) suffix }' /tmp/percipiodata/********.csv | psql -h 10.18.3.4 postgres ansibleadmin@db-ansible-postgres -c 'copy percipio_users (login_name, external_userid, first_name, last_name, email, active_status, department_id, department_name, employee_id, full_time, hire_date, job_code, job_title, manager_employee_id, manager_level, manager_name, middle_name, pay_group, position_id, supervisor_email, sys_location, file_name, file_date) from stdin WITH csv'", 
                "delta": "0:00:00.322552", 
                "end": "2022-07-20 15:21:31.263278", 
                "failed": false, 
                "invocation": {
                    "module_args": {
                        "_raw_params": "awk -vRS='\\r\\n' 'FNR==1{if (match($0,/^\\\"loginName\\\",\\\"externalUserId\\\"/)) {t=1;} else {if (match($0,/^\\\"User Name\\\",\\\"First Name\\\"/)) t=2; else t=3; } next } {it=split(FILENAME,array,\"/\"); suffix=\",\\\"\" array[it] \"\\\",\\\"\" substr(FILENAME,index(FILENAME,\"I_\")+2,index(FILENAME,\".\")-1-index(FILENAME,\"I_\")-1) \"\\\"\"; } {split($0,line,\",\")} {if (t==3) next} {if (t==2) gensub(/\\([^\\\"]\\),/,\"/1\\\\\\\\,\",\"g\",line[1] \",\" line[1] \",\" line[2] \",\" line[4] \",\" line[5] \",\\\"\\\",\" line[17] \",\" line[18] \",\" line[9] \",\" line[24] \",\" line[23]\",\" line[19] \",\" line[21] \",\" line[12] \",\" line[20]\",\" line[11] \",\" line[03] \",\" line[22] \",\" line[25] \",\" line[28] \",\" line[06]) suffix} {if (t==1) print gensub(/[^\\\"],/,\"\\\\\\\\,\",\"g\",$0) suffix }' /tmp/percipiodata/********.csv | psql -h 10.18.3.4 postgres ansibleadmin@db-ansible-postgres -c 'copy percipio_users (login_name, external_userid, first_name, last_name, email, active_status, department_id, department_name, employee_id, full_time, hire_date, job_code, job_title, manager_employee_id, manager_level, manager_name, middle_name, pay_group, position_id, supervisor_email, sys_location, file_name, file_date) from stdin WITH csv'", 
                        "_uses_shell": true, 
                        "argv": null, 
                        "chdir": null, 
                        "creates": null, 
                        "executable": null, 
                        "removes": null, 
                        "stdin": null, 
                        "stdin_add_newline": true, 
                        "strip_empty_ends": true, 
                        "warn": true
                    }
                }, 
                "item": "********.csv", 
                "rc": 0, 
                "start": "2022-07-20 15:21:30.940726", 
                "stderr": "", 
                "stderr_lines": [], 
                "stdout": "", 
                "stdout_lines": []
            }, 
            ... chop, chop, chop ...
        ]
    }
}

Answer 1

What I understood from your question is that your playbook works fine on Tower, but it doesn't when you run it on your local machine.

My answer for that:
Since the code works successfully on Ansible Tower and locally it doesn't, the logical explanation from my point of view is that Ansible version differs locally from the version in the Ansible Tower.