Recently my team at work has been exploring the idea of using docker/kubernetes for deploying a disparate set of web applications. To date, we’ve always just deployed web applications directly to a VM/cloud instance, but that pattern is starting to feel antiquated as docker and Kubernetes have made considerable leaps in progress in recent history.

I’ve always been a fan of Docker on some level — I was actually a very early adopter/user of docker before it was even publicly released. I had built some early infrastructure for running data processing pipelines on top of OG docker, but we eventually killed it for a more stable and less complex platform. It turns out; building production-grade systems atop an API that changes and gets completely rewritten every other week is hard 😬

It seems like Kubernetes has made major leaps and bounds towards making containers usable for deploying web applications that require multiple components (like databases, redis, etc). To the point that deploying applications with Kubernetes seems EASIER in some ways that manually maintaining virtual machines, keeping the kernels up to date/secure, and monitoring your entire stack. With Kubernetes you build tooling to maintain your cluster as a whole, but then all of the complexity of application-layer stuff gets defined inside the Dockerfile and podfile yml configurations. So in some ways, it might actually be easier to deploy web apps in Kubernetes than to have to worry about the entire stack of software that you don’t use which lies on the host operating system.

Anyway, I’m still a noob at Kubernetes, and I’m actively trying to build some intuition around it, to see whether it would actually be a good fit for what I’m looking for (run 20-50 small web apps with minimal maintenance and effort).

Hardware > Virtual Machines

I’ve found that I often build a more visceral intuition for how distributed systems are working if I have a physical understanding of how everything is laid out. So while it’s 100% possible to use virtual machines, and likely the best (cost to benefit ratio) strategy for building a production Kubernetes cluster — I wanted to build a physical cluster.

So I started poking around at the idea of making a little Raspberry Pi Kubernetes cluster with a minimal form factor.

The idea is to make an all-in-one lab environment that I can grab from the closet, plug in a single ethernet port, and a single power cord, and have everything up and running — and ready for experimentation!

The best resource I found on this stuff is http://www.pidramble.com/ by Jeff Geerling. On the pidramble.com website, he put together a bunch of parts lists and general thoughts on how he organized his Raspberry Pi cluster.

The best part about his cluster setup, in my opinion, is the form factor. If you include a Raspberry Pi PoE hat, you can pull power directly from the ethernet connection which makes things significantly more simple. It checks all of my boxes in terms of a self-contained cluster that’s easy to set aside, and easy to bust out for experimentation.

Check this out:

To Raspberry Pi or Not???

I shopped around a bunch and floated this idea to some of my friends. The most critical piece of feedback I received was that Raspberry Pis are underpowered; and that there may actually be better options.

One such option is the ODROID-MC1. The ODROID-MC1 is a compact 4-host compute cluster with a total of 32 CPU cores and 8GB ram. At first, this seemed like a huge win, as the compute per dollar capability is roughly double that of a Raspberry Pi 4B cluster; which weighs in at 16CPU cores and 8GB ram.

The downside, however, is the form factor and effort required to set things up. At first glance, it looks amazing. But if you want to run all of the nodes off of a single power supply, you need to purchase a cheapo whitelabel Chinese power supply, and manually wire up all of the 5.5mm power connectors like the image below. At first, I thought this sounded fun, but I have a long history of destroying electronics by making stupid mistakes with electricity — so this approach comes with an elevated risk.

The other option is to purchase 4x 5V/10A power bricks, which means I would also need to add a power strip to the build, which further hurts the form factor.

Learning LED Indicators (future improvement)

Another super interesting addition to this cluster project would be to hook up some indicator LEDs like the BlinkStick Nano. You could write some code that uses these USB LED indicators to change colors based on different behavior/states of each host. This would be super cool because it would help you develop a clear intuition for how a high availability Kubernetes cluster works.

img_3487

Small additions like this make this an even more interesting learning environment for distributed systems. But it takes a lot more work to write the monitoring code to trigger the LEDs, so this would definitely be an in-the-future project addition.

An example scenario for how these LEDs could work would be something like:

  1. Disconnect ethernet
  2. Disconnected host LED turns orange – to indicate degraded state.
  3. Containers get automagically re-provisioned (scheduled) to other hosts.
  4. LEDs on other hosts start blinking yellow to indicate that new pods are scheduled, but have not yet been started.
  5. After a minute, all hosts but the unplugged host turn green, while the degraded host remains orange.

So what’s the game plan?

So I’ve decided to roll with the Raspberry Pi cluster, for the form factor alone. Though I also suspect that the software support and resources are significantly better for the Raspberry Pi, given projects HypriotOS (a container operating system setup for running docker).

Below you’ll find a parts list including links and breakdowns of both the Raspberry Pi and ODROID-MC1 options.

At the time of writing, the Raspberry Pi 4B cluster weighs in at ~$385, while the ODROID-MC1 setup costs ~$355.

The Pi cluster is a little more expensive but provides a better form factor for being a learning tool.

I’m purchasing everything today, so sometime in the next month or so I may have a followup post for how this all goes.

Raspberry Pi 4B 2GB Cluster

Specs:

  • Cost: ~$385
  • 16 Cores (Broadcom BCM2711)
    • 4 cores per host @ 1.5GHz
  • 8GB RAM
    • 2GB per host

Raspberry 4B 2GB Parts List (~$385)

ItemQty.PriceCostNotes
Raspberry Pi 4B 2GB4$35.00140https://www.canakit.com/raspberry-pi-4-2gb.html
PoE Pi Hat4$20.0080https://www.canakit.com/raspberry-pi-poe-hat.html
128GB Samsung Evo MicroSD4$22.4189.64https://www.amazon.com/gp/product/B07BS3HLY9
Netgear GS305P 5port PoE Switch1$50.0050https://www.amazon.com/NETGEAR-Gigabit-Ethernet-Unmanaged-Desktop/dp/B01MRO4M73
Stackable Raspberry Pi Chasis1$25.0025https://www.amazon.com/GeeekPi-Raspberry-Cluster-Cooling-Heatsink/dp/B07MW3GM1T

ODROID-MC1

Specs:

  • Cost: ~$355
  • 32 Cores (Exynos 5 Octa 5422)
    • 4x per host @ 2.1GHz
    • 4x per host @ 1.4GHz
  • 8GB RAM
    • 2GB per host

ODROID-MC1 Parts List (~$355)

ItemQty.PriceCostNotes
ODROID-MC11$220.00220https://www.hardkernel.com/shop/odroid-mc1-my-cluster-one-with-32-cpu-cores-and-8gb-dram/
5v/20A Power Supply1$20.0020https://www.amazon.com/gp/product/B01IMP3V4Q
128GB MicroSD card4$22.4189.64https://www.amazon.com/gp/product/B07BS3HLY9
12x 5.5mm pigtail power plugs1$7.007https://www.amazon.com/gp/product/B07199792W
NEMA 5-15P to Unterminated1$17.0017https://www.amazon.com/Parts-Express-Power-Cord-Black/dp/B072YNYPZR