Run Ansible Tower or AWX in Kubernetes or OpenShift with the Tower Operator

Note: Please note that the Tower Operator this post references is currently in early alpha status, and has no official support from Red Hat. If you are planning on using Tower for production and have a Red Hat Ansible Automation subscription, you should use one of the official Tower installation methods. Someday the operator may become a supported install method, but it is not right now.

I have been building a variety of Kubernetes Operators using the Operator SDK. Operators make managing applications in Kubernetes (and OpenShift/OCP) clusters very easy, because you can capture the entire application lifecycle in the Operator's logic.

AWX Tower Operator SDK built with Ansible for Kubernetes

Ansible Tower is a central system for Ansible automation scheduling and inventory management, with robust RBAC and many other useful features for managing automation tasks for teams. AWX is the open source upstream project where Tower's bleeding-edge development takes place. The relationship between the two is similar to Fedora (the less stable open source upstream which requires no license to use) and Red Hat Enterprise Linux (which is more stable and requires a license).

I am working on an Ansible Tower/AWX Operator for Kubernetes which makes installing and managing instances of Tower or AWX in your Kubernetes clusters easy.

Why an Operator?

Since there's already an install playbook for Ansible Tower on Kubernetes or OpenShift, why would there be a need for an Operator that does the same thing? There are a few good reasons:

  • There equivalent installation playbook for AWX is a little different than the one that ships with Tower, so it can be hard to see where things differ between the two systems.
  • The Operator pattern is the most Kubernetes-native way to manage complex applications like Tower.
  • Operators do more than just install and reconfigure applications; they can manage backups, upgrades, and even more, and they do it in real-time, all the time.

To the last point, the Operator doesn't yet manage the full lifecycle of Tower—it just does installation, configuration, and updates—but over time the hope is the Operator can do more and make it easy for you to concentrate on using Tower, not managing it.

Using the Operator

The Operator requires a Kubernetes cluster; if you just want to try it out, the best way to get a Kubernetes cluster locally is to Install Minikube, then start a Minikube instance:

$ minikube start --memory 6g --cpus 4

After a few minutes, you should have a Kubernetes environment running (and you'll probably know it by the sound of your computer's fans revving up!). Now it's time to install the Tower Operator, which is done by applying its manifest:

$ kubectl apply -f https://raw.githubusercontent.com/geerlingguy/tower-operator/master/deploy/tower-operator.yaml

After you run this command, you should be able to see the Tower Operator running in your Kubernetes cluster:

$ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
tower-operator-894f44fdb-wfrkv   2/2     Running   0          92s

To create an instance of Ansible Tower, create a manifest file for it named tower.yml, with the following contents:

---
apiVersion: tower.ansible.com/v1alpha1
kind: Tower
metadata:
  name: tower
  namespace: tower
spec:
  tower_hostname: tower.test
  tower_secret_key: aabbcc
 
  tower_admin_user: admin
  tower_admin_email: [email protected]
  tower_admin_password: changeme

If you want to install AWX instead of Tower, add the following two variables in spec to override the default Tower images:

tower_task_image: ansible/awx_task:9.0.1
tower_web_image: ansible/awx_web:9.0.1

Create the namespace for your Ansible Tower instance:

$ kubectl create namespace tower

Run the following command to deploy this instance of Ansible Tower into your cluster:

$ kubectl apply -f tower.yml

At this point, the Tower Operator will notice the existence of this new Tower resource, and then it will start setting it up in the tower namespace. You can watch as the Tower pods are brought up:

$ watch kubectl get pods -n tower

After a few minutes, you should see a number of pods running Postgres, RabbitMQ, Memcached, Tower, etc. After they are in Running status, Tower should be functional and responding to web requests. To access tower from your browser, you will need to make sure Minikube is exposing it at the tower_hostname you defined earlier. To do that, enable the ingress addon in Minikube:

$ minikube addons enable ingress

Then you need to get the IP address of the Minikube environment:

$ minikube ip

And edit your hosts file (usually /etc/hosts), adding a line like the following:

MINIKUBE_IP_HERE  tower.test

Finally, go into your web browser and open up http://tower.test/. On its first run (or after a version upgrade, when database migrations are running), you may see the "Tower is upgrading..." screen:

Ansible Tower is Upgrading screenshot

After the database migrations are complete, you should be redirected to the login screen:

Ansible Tower login screen

At this point, you can log in using the credentials in the Tower resource spec (in the example above, admin for username and changeme for password), and start using Tower! Note that use of Tower requires a license—you can get a free trial license here if you're evaluating Tower. If you use AWX, you will not be prompted for a license.

What the future holds

This operator was built using Ansible and the Operator SDK; so it's installing Ansible Tower on Kubernetes by running Ansible in an Operator on Kubernetes! Using Ansible for the operator makes it easy for anyone—not just Go developers—to build Kubernetes-native automation. I've written more about this in my book, Ansible for Kubernetes, which should be available soon.

There are a number of improvements I hope to make in the Tower operator (see the Tower operator issue queue), to make it more robust and helpful for those who want to manage one or more instance of Tower or AWX in a Kubernetes or OpenShift cluster. I also hope to get the operator listed on OperatorHub.io, and to see if I can grow it to become a supported installation method for Ansible Tower.

But even if not, the operator is a good example of the possibilities of Ansible-based operators in Kubernetes!

Comments

Thanks Jeff looks cool, will check it out. I'm curious to know after installing how would I upgrade aws/tower I would really like to see the operator doing the in place upgrade.

Right now, the way that would happen is:

  1. You update the tower_web_image and tower_task_image to the latest versions for each Tower resource CR.
  2. The operator will see the image update and run database migrations.

There may be other parts needed to make upgrades work for minor releases (e.g. 3.5 to 3.6), but at least for now, this should take care of patch releases (e.g. 3.6.1 to 3.6.2).

Note that it's something which may (probably will) need more refinement, and the process should also take a database backup or snapshot the volume it's on (right now it doesn't).

Hi Jeff!

Congratulations for the great work!

Some fixes:
1 - For Openshift, postgresql image is unable to write on data directory, so to work we need to set anyuid scc for the namespace.
2 - Maybe for the same reason, the status of "updating" still running forever. (I had to create the route manually).
3 - The App tower-task is unable to have 1 replica running.
4 - I had to change the yaml to place the tower-operator service account on the project I have created. It's all placed to run on default project.

This is really cool! Could you please elaborate on the benefits of running AWS on top of Kubernetes, and the benefits of using Operator? I assume we could scale out the web POD, but no the postgres DB POD?

Hi. Looks like something we could try out. How do you define to use AWX?

Hi Jeff,

1) Installed minikube
2) sudo minikube start --vm-driver=none
3) sudo minikube kubectl -- apply -f https://raw.githubusercontent.com/geerlingguy/tower-operator/master/dep…
4) sudo minikube kubectl -- get pods

Output shows error:
NAME READY STATUS RESTARTS AGE
tower-operator-7cbd9895cd-rhxmm 1/2 Error 2 38s
[root@awx2 kubernetes]# minikube kubectl -- get pods
NAME READY STATUS RESTARTS AGE
tower-operator-7cbd9895cd-rhxmm 1/2 CrashLoopBackOff 4 2m17s

Not sure what is going on here?

BR

If a pod is stuck in CrashLoopBackOff, that means something is wrong with it's startup, most likely. You can use kubectl logs to see what's going on: kubectl logs tower-operator-7cbd9895cd-rhxmm -c operator (or -c ansible for the ansible container) and see what's crashing.

Thank Jett !
I'm a newbie Tower, I have just installed Tower in Kubernetes, so i have some problem, I don't know how to redeploy Tower, and how can i find list of variable in spec to custom. I want change resource: CPU, Mem request... but don't know how to do. Thank you !

Thanks this looks awesome! is it possible to use an external database with it? and if so how :o?

Currently no, it is not configurable; however, that is something that could be supported without too much effort. Right now it uses an in-cluster managed Postgres instance.

Hi,
first, thinks for the great works!
I use your but i got the following error for postgres pod: Warning FailedScheduling default-scheduler error while running "VolumeBinding" filter plugin for pod "tower-postgres-0": pod has unbound immediate PersistentVolumeClaims

I have to create PV and PVC externally ?
Regards

I deployed a persistent volume on my cluster:

NAME         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
pvpostgres   2Gi        RWO            Recycle          Available           slow                    26h

Le persistent used with postgres still pending and dont use the pv previously deployed:

NAMESPACE       NAME                        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ansible-tower   postgres-tower-postgres-0   Pending                                                     27h

regards

You might need to run kubectl describe pv -n ansible-tower postgres-tower-postgres-0 and see why the PV is not being created, or also debug PVs elsewhere in the cluster. It could be permisssions or the PV structure in your cluster.

Hi,
Yes i already do that.
How can i modify the pvc ? i'm using nfs with no provisionner. I have to set stoarageclass="" in the pvc.

Regards

Hi,
After fixing my problems with pvc, i got this error on the container postgres:
chown: changing ownership of '/var/lib/postgresql/data': Operation not permitted

I have to create a specific user on nodes ?
Regards

Thanks , this is really. meanwhile i would like to understand how to attach one of the container to do manual project instead of Git , since i am trying to test it in my own environment.
Tried few options seems there is no TTY and also have trouble to find out which container host the /var/lib/awx/projects

Thanks for your tuto.
I install awx on a centos8 vm on a vmware infra
It works... But only from the server with the minikube IP.
How to Access by another computer on the same network?

Hi:

I'm trying to edit the deployment:

kubectl edit deployments awx

But all changes are reverted after save. How i can edit the deployment?

Thanks.

Haii Jeff,

Can we deploy awx with persisten volume using operator ?