Note: Please note that the Tower Operator this post references is currently in early alpha status, and has no official support from Red Hat. If you are planning on using Tower for production and have a Red Hat Ansible Automation subscription, you should use one of the official Tower installation methods. Someday the operator may become a supported install method, but it is not right now.
I have been building a variety of Kubernetes Operators using the Operator SDK. Operators make managing applications in Kubernetes (and OpenShift/OCP) clusters very easy, because you can capture the entire application lifecycle in the Operator's logic.
Ansible Tower is a central system for Ansible automation scheduling and inventory management, with robust RBAC and many other useful features for managing automation tasks for teams. AWX is the open source upstream project where Tower's bleeding-edge development takes place. The relationship between the two is similar to Fedora (the less stable open source upstream which requires no license to use) and Red Hat Enterprise Linux (which is more stable and requires a license).
I am working on an Ansible Tower/AWX Operator for Kubernetes which makes installing and managing instances of Tower or AWX in your Kubernetes clusters easy.
Why an Operator?
Since there's already an install playbook for Ansible Tower on Kubernetes or OpenShift, why would there be a need for an Operator that does the same thing? There are a few good reasons:
- There equivalent installation playbook for AWX is a little different than the one that ships with Tower, so it can be hard to see where things differ between the two systems.
- The Operator pattern is the most Kubernetes-native way to manage complex applications like Tower.
- Operators do more than just install and reconfigure applications; they can manage backups, upgrades, and even more, and they do it in real-time, all the time.
To the last point, the Operator doesn't yet manage the full lifecycle of Tower—it just does installation, configuration, and updates—but over time the hope is the Operator can do more and make it easy for you to concentrate on using Tower, not managing it.
Using the Operator
The Operator requires a Kubernetes cluster; if you just want to try it out, the best way to get a Kubernetes cluster locally is to Install Minikube, then start a Minikube instance:
$ minikube start --memory 6g --cpus 4
After a few minutes, you should have a Kubernetes environment running (and you'll probably know it by the sound of your computer's fans revving up!). Now it's time to install the Tower Operator, which is done by applying its manifest:
$ kubectl apply -f https://raw.githubusercontent.com/geerlingguy/tower-operator/master/deploy/tower-operator.yaml
After you run this command, you should be able to see the Tower Operator running in your Kubernetes cluster:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
tower-operator-894f44fdb-wfrkv 2/2 Running 0 92s
To create an instance of Ansible Tower, create a manifest file for it named tower.yml
, with the following contents:
---
apiVersion: tower.ansible.com/v1alpha1
kind: Tower
metadata:
name: tower
namespace: tower
spec:
tower_hostname: tower.test
tower_secret_key: aabbcc
tower_admin_user: admin
tower_admin_email: [email protected]
tower_admin_password: changeme
If you want to install AWX instead of Tower, add the following two variables in
spec
to override the default Tower images:tower_task_image: ansible/awx_task:9.0.1 tower_web_image: ansible/awx_web:9.0.1
Create the namespace for your Ansible Tower instance:
$ kubectl create namespace tower
Run the following command to deploy this instance of Ansible Tower into your cluster:
$ kubectl apply -f tower.yml
At this point, the Tower Operator will notice the existence of this new Tower resource, and then it will start setting it up in the tower
namespace. You can watch as the Tower pods are brought up:
$ watch kubectl get pods -n tower
After a few minutes, you should see a number of pods running Postgres, RabbitMQ, Memcached, Tower, etc. After they are in Running
status, Tower should be functional and responding to web requests. To access tower from your browser, you will need to make sure Minikube is exposing it at the tower_hostname
you defined earlier. To do that, enable the ingress
addon in Minikube:
$ minikube addons enable ingress
Then you need to get the IP address of the Minikube environment:
$ minikube ip
And edit your hosts file (usually /etc/hosts
), adding a line like the following:
MINIKUBE_IP_HERE tower.test
Finally, go into your web browser and open up http://tower.test/. On its first run (or after a version upgrade, when database migrations are running), you may see the "Tower is upgrading..." screen:
After the database migrations are complete, you should be redirected to the login screen:
At this point, you can log in using the credentials in the Tower resource spec (in the example above, admin
for username and changeme
for password), and start using Tower! Note that use of Tower requires a license—you can get a free trial license here if you're evaluating Tower. If you use AWX, you will not be prompted for a license.
What the future holds
This operator was built using Ansible and the Operator SDK; so it's installing Ansible Tower on Kubernetes by running Ansible in an Operator on Kubernetes! Using Ansible for the operator makes it easy for anyone—not just Go developers—to build Kubernetes-native automation. I've written more about this in my book, Ansible for Kubernetes, which should be available soon.
There are a number of improvements I hope to make in the Tower operator (see the Tower operator issue queue), to make it more robust and helpful for those who want to manage one or more instance of Tower or AWX in a Kubernetes or OpenShift cluster. I also hope to get the operator listed on OperatorHub.io, and to see if I can grow it to become a supported installation method for Ansible Tower.
But even if not, the operator is a good example of the possibilities of Ansible-based operators in Kubernetes!
Comments
Thanks Jeff looks cool, will check it out. I'm curious to know after installing how would I upgrade aws/tower I would really like to see the operator doing the in place upgrade.
Right now, the way that would happen is:
tower_web_image
andtower_task_image
to the latest versions for each Tower resource CR.There may be other parts needed to make upgrades work for minor releases (e.g. 3.5 to 3.6), but at least for now, this should take care of patch releases (e.g. 3.6.1 to 3.6.2).
Note that it's something which may (probably will) need more refinement, and the process should also take a database backup or snapshot the volume it's on (right now it doesn't).
Thanks
Hi Jeff!
Congratulations for the great work!
Some fixes:
1 - For Openshift, postgresql image is unable to write on data directory, so to work we need to set anyuid scc for the namespace.
2 - Maybe for the same reason, the status of "updating" still running forever. (I had to create the route manually).
3 - The App tower-task is unable to have 1 replica running.
4 - I had to change the yaml to place the tower-operator service account on the project I have created. It's all placed to run on default project.
@Luciano - Thanks for the comment! I've added a new issue in the Tower Operator's issue queue to track progress making it easier to run on OpenShift out of the box: Make sure Tower Operator can be deployed easily on OpenShift.
This is really cool! Could you please elaborate on the benefits of running AWS on top of Kubernetes, and the benefits of using Operator? I assume we could scale out the web POD, but no the postgres DB POD?
Hi. Looks like something we could try out. How do you define to use AWX?
Please see the README: https://github.com/geerlingguy/tower-operator#deploy-awx-instead-of-tower
Hi Jeff,
1) Installed minikube
2) sudo minikube start --vm-driver=none
3) sudo minikube kubectl -- apply -f https://raw.githubusercontent.com/geerlingguy/tower-operator/master/dep…
4) sudo minikube kubectl -- get pods
Output shows error:
NAME READY STATUS RESTARTS AGE
tower-operator-7cbd9895cd-rhxmm 1/2 Error 2 38s
[root@awx2 kubernetes]# minikube kubectl -- get pods
NAME READY STATUS RESTARTS AGE
tower-operator-7cbd9895cd-rhxmm 1/2 CrashLoopBackOff 4 2m17s
Not sure what is going on here?
BR
If a pod is stuck in
CrashLoopBackOff
, that means something is wrong with it's startup, most likely. You can usekubectl logs
to see what's going on:kubectl logs tower-operator-7cbd9895cd-rhxmm -c operator
(or-c ansible
for the ansible container) and see what's crashing.Thank Jett !
I'm a newbie Tower, I have just installed Tower in Kubernetes, so i have some problem, I don't know how to redeploy Tower, and how can i find list of variable in spec to custom. I want change resource: CPU, Mem request... but don't know how to do. Thank you !
See all the available CR options here: https://github.com/geerlingguy/tower-operator/blob/master/deploy/crds/tower_v1alpha1_tower_cr_awx.yaml (e.g.
tower_task_mem_request
,tower_task_cpu_request
, etc.).Thanks this looks awesome! is it possible to use an external database with it? and if so how :o?
Currently no, it is not configurable; however, that is something that could be supported without too much effort. Right now it uses an in-cluster managed Postgres instance.
Hi,
first, thinks for the great works!
I use your but i got the following error for postgres pod: Warning FailedScheduling default-scheduler error while running "VolumeBinding" filter plugin for pod "tower-postgres-0": pod has unbound immediate PersistentVolumeClaims
I have to create PV and PVC externally ?
Regards
If you're deploying to OpenShift there's currently an issue with the Postgres setup that will be resolved once this PR is merged in: https://github.com/geerlingguy/tower-operator/pull/34. Please post any other issues you might find in the project's repository. Thanks!
no, i'm deploying on a cluster kubernetes without openshift.
I deployed a persistent volume on my cluster:
Le persistent used with postgres still pending and dont use the pv previously deployed:
regards
You might need to run
kubectl describe pv -n ansible-tower postgres-tower-postgres-0
and see why the PV is not being created, or also debug PVs elsewhere in the cluster. It could be permisssions or the PV structure in your cluster.Hi,
Yes i already do that.
How can i modify the pvc ? i'm using nfs with no provisionner. I have to set stoarageclass="" in the pvc.
Regards
It might be best if you look through / open new issues in the repository (this blog post is not a great place to offer support :) — https://github.com/geerlingguy/tower-operator/issues
Hi,
After fixing my problems with pvc, i got this error on the container postgres:
chown: changing ownership of '/var/lib/postgresql/data': Operation not permitted
I have to create a specific user on nodes ?
Regards
Thanks , this is really. meanwhile i would like to understand how to attach one of the container to do manual project instead of Git , since i am trying to test it in my own environment.
Tried few options seems there is no TTY and also have trouble to find out which container host the /var/lib/awx/projects
Thanks for your tuto.
I install awx on a centos8 vm on a vmware infra
It works... But only from the server with the minikube IP.
How to Access by another computer on the same network?
I'm having the same trouble, but cannot find any simple documentation as to what I've done wrong.
Hi:
I'm trying to edit the deployment:
kubectl edit deployments awx
But all changes are reverted after save. How i can edit the deployment?
Thanks.
Haii Jeff,
Can we deploy awx with persisten volume using operator ?