Virtual machines setup for homelab
Sometimes, you might want to test out something before applying it to a target machine - reason being a botched deployment is harder to fix. If you have a spare machine lying around, you could test on that, but then you'll have to fire it up and make sure the base configurations are usable, otherwise you'll run into configuration drift between your target machine and the test environment. A different OS version is enough to break a deployment sometimes, from my experience. But even without a homelab, the same applies for your workstation as well - trying out something before applying it to your daily driver.
A virtual machine is a portion of your machine that's used for another environment - its own OS, disk and a portion of CPU and memory. You can mount supported peripherals - a USB flash drive is one example. VMs are especially very useful if you run into cases where you have a large machine and you want to share it between different users. Years as a sysadmin taught me that a shared box with multiple users will turn into a ball of mud eventually, it's just a matter of time. But if you are on the adventurous side, no one is stopping you from using a single machine to do everything, but I hope you document all your configurations so you can reproduce it.
General Setup
Setting up VMs usually follows a few steps (in no particular order):
- Obtaining a base image (Ubuntu, Fedora, etc.)
- Setup cloud-init config (for username/password, ssh keys, base packages, etc.)
- Define a volume, specify disk size and attach it
- Provision a VM with defined CPU / memory
Setups
All solutions here are based on QEMU, which makes it even more fun when you realize it's doing almost the same thing under the hood, just using different APIs.
KubeVirt
If you already have a kubernetes cluster and don't want to dedicate a node for Proxmox, KubeVirt would be a good setup. It utilizes k8s CRDs, and running VMs are launched as pods, so you can use your existing k8s monitoring setup to see VM usage and utilization.
Follow the docs on how to set it up. KubeVirt also provides its own CLI for managing KubeVirt VMs (it also provides access to serial console and vnc). Below is an example k8s manifest:
---
apiVersion: v1
kind: Service
metadata:
name: debian-vm-ssh
namespace: vms
spec:
type: NodePort
selector:
kubevirt.io/vm: debian-vm
ports:
- protocol: TCP
port: 22
targetPort: 22
nodePort: 30080
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: debian-vm
namespace: vms
spec:
runStrategy: Halted
template:
metadata:
labels:
kubevirt.io/vm: debian-vm
spec:
domain:
devices:
disks:
- name: rootdisk
disk:
bus: virtio
- name: cloudinitdisk
disk:
bus: virtio
interfaces:
- name: default
masquerade: {}
resources:
requests:
memory: 4Gi
cpu: 4
networks:
- name: default
pod: {}
volumes:
- name: rootdisk
dataVolume:
name: debian-dv
- name: cloudinitdisk
cloudInitNoCloud:
userData: |
#cloud-config
hostname: debian-vm
users:
- name: debian
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/bash
ssh_authorized_keys:
- ssh-ed25519 xxxxxxxxxxxxxxxxx
ssh_pwauth: true
disable_root: false
chpasswd:
list: |
debian:debian
expire: false
package_update: true
packages:
- qemu-guest-agent
runcmd:
- systemctl enable ssh
- systemctl start ssh
dataVolumeTemplates:
- metadata:
name: debian-dv
spec:
storage:
resources:
requests:
storage: 32Gi
accessModes:
- ReadWriteOnce
source:
registry:
url: "docker://quay.io/containerdisks/debian:13"You can reach this VM via $K8S_NODE_IP:$VM_PORT.
Personally I think the API is too verbose, but if your requirement is to use k8s as a control plane, it's a good option.
Proxmox
Proxmox would be easier to use if you are not comfortable with k8s. You can use the web ui to control and manage VMs. You can install a normal ISO image for a desktop OS as well. The web ui also provides vnc console to access your VMs without requiring setting up third-party tools.
The only downside is, Proxmox is also an OS, which means your machine will be used as a Proxmox node. You also need to use ethernet, wi-fi won't work because Proxmox needs ethernet to use bridge network (to assign real DHCP IPs to VMs).
Also if you want to extend the lifespan of your SSD, Proxmox might not be ideal because it writes to disk a lot.
If you prefer using terraform to manage Proxmox VMs, it is possible. For my setup I define VMs in a local block and let terraform do the rest:
locals {
vms = {
scratch = {
id = 101
on_boot = false
template = "docker"
cpu = 2
memory = 2048
disk = 8
}
kube-scratch = {
id = 103
on_boot = false
template = "kubernetes"
cpu = 4
memory = 5000
disk = 20
}
}
}I gave a talk about this, in case you are interested.
Nomad
Kubernetes won the orchestrator war. But if you have legacy workloads which are not container-based Nomad is a good option. Nomad can also schedule QEMU workloads with very minimal overhead. There's more prep required compared to KubeVirt or Proxmox, since you'll have to create disk images for QEMU manually (or you can automate it, but it doesn't work out of the box).
To use this in a homelab, you'll have to install Nomad and setup systemd to launch it, then modify the config to add QEMU config. Next, define a Nomad job to wire up everything together. You'll need to stitch this yourself, but it's quite a lot to go through. This setup is only ideal if you want minimal orchestration overhead.
Upside is you can also use terraform, in which you can define VMs as:
locals {
vms = {
foo = {
cpu = 1
memory = 512
disk = 8
ssh_port = 2222
}
scratch = {
cpu = 2
memory = 1024
disk = 8
ssh_port = 2223
}
}
}(Yes, you'll have to stitch a terraform setup as well.)
VirTofu
If you want little to no orchestration overhead, VirTofu would fit the bill. You'll need to install some packages and define a few configurations before you can get it up and running. It uses libvirt as an API to talk to QEMU, in which terraform is the one telling libvirt what to do. This setup has been used in production as well, so it's battle-tested.
However, this setup expects you to understand the basics of QEMU and networking. It might not work for you if you just want something that works out of the box.
Personally I started with Proxmox, then found it shredded my SSD (also using a whole node for Proxmox means I can't use the machine for something else). Then I migrated to KubeVirt because I already have k8s running. Not a fan of KubeVirt API but it gets the job done. Recently for funsies I tinkered with Nomad, got it working so I'm using it for scratch VMs because it's less yaml. I haven't used VirTofu, but I helped its creator refine the docs before he made it public.
A tool isn't a silver bullet - what works for you might not work for others. Understand what you need, then find the tools that fit the bill - it'll be better for your own sanity.
Happy tinkering!