Incomparably Square

Saturday February 24, 2024

Well, I’ve finally reformatted my Dell R730 project system and replaced TrueNAS SCALE with Proxmox VE once again. Now that I want to go deeper into virtualizing so I can do more Kubernetes experimentation, using a NAS device as the Hypervisor only serves to complicate the process.

There wasn’t anything critical on the system, so that made it convenient to reformat, and just start from scratch. Why all the re-shuffling? EDB wants me to start digging into our cloud products, and that means I need a more convenient place to stage and experiment with that kind of thing. While I could theoretically have just install something like minikube on my local system and leave my poor TrueNAS alone, I was pretty much done with its shenanigans anyway.

The thing about TrueNAS is that it’s meant to essentially be an appliance. It basically locks everything down, and there’s no real guarantee any modifications you make will survive an upgrade. The irony here is that it actually has a K3s environment for its application ecosystem, but you have to jump through so many hoops to use it as such, it’s not worth the trouble. The other problem is that some of the apps rely on databases, making it difficult or impossible to properly upgrade them. Meaning you have to dump the database somehow, upgrade the app, and import the old data. If I’m going to have to do all of that anyway, why not just run my own damn cluster?

I ran across Jim’s Garage in my YouTube recommendations and that just happened to be the topic for the last couple months. So why not give setting up a “real” Kubernetes cluster a shot? I needed to do it anyway, and it gave me just enough impetus to put the TrueNAS out of its misery. I even submitted a pull request to spruce up his deployment script a bit.

I also have to note that using prebuilt cloud images in Proxmox’s cloud-init system is a huge time saver. Once I downloaded a Debian Trixie qcow2 image and made a template, I generated a half dozen VMs pretty quickly:

for x in {1..5}; do
  let node=$[499 + $x]
  qm clone 5000 ${node} --name kubernetes-${x} --full 1
  qm resize ${node} scsi0 +18G
  qm set ${node} --ipconfig0 ip=10.0.5.${x}/16,gw=10.0.0.1
done

I had already initialized the template VM with dual sockets and cores, 4GB of RAM, and 2GB of minimal storage. The appropriate SSH keys, system user and password, and any other necessary OS installation information are also associated with the template in the cloud-init system. Meaning all of these clones are dead easy to deploy, and I was able to SSH to them almost immediately once the loop completed.

Since the template VM is 2GB, adding 18 makes it an even 20. I haven’t decided how big each node should be yet, so once I’m done with the experiment, I’ll resize to something more reasonable. I suspect I’ll need to rebuild the worker nodes at minimum once I actually start making them do things. I’ve also segregated my network such that 10.0.5.* is essentially set aside for Kubernetes, which “should be enough” for any node configs and ingress settings I want to use. If not, I can always add another C-Block.

With all of that out of the way, it’s time to start fiddling with the cluster itself. I know there’s a thing called kubectl and… that’s about it. Hopefully I haven’t bitten off more than I can chew.

Until Tomorrow

See Also