Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

I realized I haven't posted about my DrupalCon Seattle 2019 session titled Everything I know about Kubernetes I learned from a cluster of Raspberry Pis, so I thought I'd remedy that. First, here's a video of the recorded session:

The original Raspberry Pi Dramble Cluster
The original Pi Dramble 6-node cluster, running the LAMP stack.

I started running the Raspberry Pi Dramble in 2014, after I realized I could automate the setup of everything in a LAMP stack on a set of Raspberry Pi 2s using Ansible (one Pi for an HTTP load balancer/reverse proxy, two for PHP app backends, and two for MySQL redundancy. Kubernetes was the logical next step, so I moved things towards Kubernetes in 2017, but running Kubernetes was a lesson in pain due to the Pi's limited memory (1 GB maximum) at the time.

When the Raspberry Pi 4 came around, I acquired some 2 GB models as quickly as I could, and redeployed onto them. Gone were the restrictions that were causing Kubernetes' API to be flaky with the older Pis, and now the Pi 4 cluster is extremely reliable. Early on, cooling was an issue, but the recent firmware update has made that less problematic. I now power the cluster using the official PoE HAT, which means I only have one plug for each Pi, and everything fits nicely into a small travel case (if I want to bring the cluster with me anywhere).

Raspberry Pi Dramble 4 Kubernetes edition cluster
The current Pi Dramble, running Kubernetes and sporting BlinkStick Nanos.

I even got it all to run off a 10,000 mAh battery pack with a bunch of USB splitters... but it did not stay powered long, and kept giving low power warnings—so I'll have to consider other options for a highly-mobile four-node Kubernetes bare-metal cluster.

I've continually updated the cluster so it is tested in a Docker-based Kubernetes environment, a Vagrant-based Kubernetes local development environment, and of course, the Pi environment. The latter environment causes much consternation, as many common container images are not maintained in an armv6 or linux/arm format, which is required when running on the 32-bit ARM OS the Pi uses, Raspbian. But the Pi Dramble abides, and it quietly goes on, serving up traffic for https://www.pidramble.com through the years.

The official Pi Dramble Wiki has all the instructions for building your own Pi Kubernetes cluster, with links to buy all the parts I have, along with every step to get it running using open source Ansible roles to install Kubernetes and Docker for ARM, then configure a new four-node cluster.

Further reading

appearances
drupalcon
drupal
drupal planet
kubernetes
raspberry pi
dramble
presentations

Comments

Thanks for the hint on that little blinky thing called BlinkStickNanos.

This is very cool. I’ve been thinking about doing something like this and this post got me excited to try it.

Do you have any recommendations for hosting the pi rack? I imagine you’d need to co-locate?

I've often wondered why people use things like RPs for work like this rather than a multi-core desktop (or server) machine with a bunch of VMs. These days such machines can be found for real cheap, and they wouldn't present the issues you mentioned in your article. Sure, you can't put them in a brief case (unless you're using a laptop) and they might take more power than a cluster of RPs. But, if time is money, then I don't understand the attraction of RP-like boards for purposes other than special projects that will ultimately run on such boards.

Agreed. It's a great visual technique. Keep up the great work. Big fan.

I've been curious to do something like this with the latest Raspberry Pi 4's as well -- but AFAIK power requirements are now an issue. Thanks for your power tips (including using the POE hat)

I followed your pidramble Kubernetes GitHub page. I am confused.

When I do a kubectl get svc I get the following:

NAMESPACE     NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP                  5d21h
kube-system   service/kube-dns     ClusterIP   10.96.0.10     <none>        53/UDP,53/TCP,9153/TCP   5d21h
wordpress     service/mysql        ClusterIP   None           <none>        3306/TCP                 59s
wordpress     service/wordpress    NodePort    10.98.65.212   <none>        80:30820/TCP             36s

I cannot access my world press site by domain name that I identified in deploy-ingress.yaml file:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: wordpress-ingress
  namespace: wordpress
  annotations: {}
spec:
  rules:
  - host: mikeventarola.net
    http:
      paths:
      - path: /
        backend:
          serviceName: wordpress
          servicePort: 80

I can only access it by the nodeport.

Am I missing something?

Thanks
Mike

Do you have the DNS for mikeventarola.net pointed at your Raspberry Pi's IP address (any of the Pi nodes, that is)? You can edit your computer's hosts file to do this.

I'm keen to learn Kubernetes and I'm looking to setup a similar environment using Raspberry Pi's. What model of the Pi 4 Model B are you using the 2Gb or the 4Gb model? Between this page and the Pi Dramble GitHub page it's not clear as this page refers to the 2Gb model and the GitHub page the 4Gb model.

More is always better; but these are all 2 GB models. I also sometimes do some testing with K8s on a single Pi, and for that I do it only on the 4 GB Pi, otherwise I usually run out of memory pretty quick.