tl;dr: Python2 strikes back! Always use Python3.
Ran into this issue after an upgrade of AWX from 14.0.0 to 15.0.1, where jobs will just fail with no output in AWX showing what happened. Just a blank box where stdout normally appears. Well great… Checked /var/log/, journalctl, and any other logging locations I could think of. The only thing I would see was a login event for awx user, then a disconnect by client (tower server). Pretty useless.
AWX_ISOLATED_NODE sshd[20060]: pam_unix(sshd:session): session opened for user awx by (uid=0) AWX_ISOLATED_NODE sshd[20063]: Received disconnect from 192.168.0.10 port 46462:11: disconnected by user AWX_ISOLATED_NODE sshd[20063]: Disconnected from 192.168.0.10 port 46462 AWX_ISOLATED_NODE sshd[20060]: pam_unix(sshd:session): session closed for user awx AWX_ISOLATED_NODE systemd-logind[1057]: Removed session 2269.
Turns out my isolated instance node is installed on a CentOS 7 box, which by default still runs Python2. Unbeknownst to me, this would be key, as most of my other isolated nodes are CentOS 8, which is Python3 default.
Well the ansible-runner version for python2 is 1.0.1, but it seems Ansible 2.9.14/AWX 15.0.1 do not support ansible-runner 1.0.1, and it needs to be at version 1.4.6.
In order to do this, I removed python-ansible-runner through yum, which is version 1.0.1. Then installed via pip3 install ansible-runner. Version 1.4.6!
This installation pumped out an warning during installation that /usr/local/bin was not in $PATH.
I added /usr/local/bin to /etc/environment, rebooted and I’m now able to run jobs!