debian

Debugging networking issues with multi-node Kubernetes on VirtualBox

Since this is the third time I've burned more than a few hours on this particular problem, I thought I'd finally write up a blog post. Hopefully I find this post in the future, the fourth time I run into the problem.

What problem is that? Well, when I build a new Kubernetes cluster with multiple nodes in VirtualBox (usually orchestrated with Vagrant and Ansible, using my geerlingguy.kubernetes role), I get everything running. kubectl works fine, all pods (including CoreDNS, Flannel or Calico, kube-apiserver, the scheduler) report Running, and everything in the cluster seems right. But there are lots of strange networking issues.

Sometimes internal DNS queries work. Most of the time not. I can't ping other pods by their IP address. Some of the debugging I do includes:

Updating all your servers with Ansible

From time to time, there's a security patch or other update that's critical to apply ASAP to all your servers. If you use Ansible to automate infrastructure work, then updates are painless—even across dozens, hundreds, or thousands of instances! I've written about this a little bit in the past, in relation to protecting against the shellshock vulnerability, but that was specific to one package.

I have an inventory script that pulls together all the servers I manage for personal projects (including the server running this website), and organizes them by OS, so I can run commands like ansible [os] command. Then that enables me to run commands like:

Format eMMC storage on an Orange Pi, Radxa, etc.

To use eMMC modules on the Orange Pi, Radxa, Milk-V, etc. as a writable volume in Linux, you need to delete the existing partitions (on my old Orange Pi, it was formatted as FAT/WIN32), create a new partition, format the partition, then mount it:

  1. Delete the existing partitions, and create a new partition:
    1. sudo fdisk /dev/mmcblk1
    2. p to list all partitions, then d and a number, once for each of the existing partitions.
    3. n to create a new partition, then use all the defaults, then w to write the changes.
  2. Format the partition: sudo mkfs.ext4 -L "emmc" /dev/mmcblk1p1
  3. Create a mount point: sudo mkdir -p /mnt/emmc
  4. Mount the disk: sudo mount /dev/mmcblk1p1 /mnt/emmc

Note your eMMC device may be a different ID, e.g. mmcblk2 or mmcblk0, depending on the order the board firmware loads multiple devices in. Check with lsblk to see which device you would like to modify.