Multi-site fleet management with warpgate, tailscale and iroh

The story so far

Over time it's entirely possible to accumulate multiple virtual machines across sites. Could be your home, your friend's home, your other friend's home. You might have a single virtual machine on aws, because it's what you are most familiar with. Then you need $awesomeFeatureAvailableOnOtherCloud so you created a few vms on gcp because you need to fetch data from BigQuery (because it's cheaper than network egress outside gcp). And you thought you can avoid Windows, but you inherited a legacy app which requires windows, so you need a few Windows vms on Azure.

Right now your fleet is located across different locations:

%%{init: {'theme': 'dark'}}%%
graph TD
subgraph cloud ["Cloud"]
subgraph azure ["Azure"]
K
L
end
subgraph gcp ["GCP"]
H
I
J
end
subgraph aws ["AWS"]
G
end
end
subgraph onprem [On-Prem]
subgraph SiteC ["Your Other Friend's Home"]
D
E
F
end
subgraph SiteB ["Your Friend's Home"]
B
C
end
subgraph SiteA ["Your Home"]
A
end
end

Normally you would not expose each vm to public internet, rather you would create a bastion host as an access gateway to your vms:

%%{init: {'theme': 'dark'}}%%
graph LR
subgraph VPC
BastionHost[Bastion Host]
A
B
C
end
Workstation -->|SSH| BastionHost
BastionHost --> A
BastionHost --> B
BastionHost --> C

There are 6 sites in the fleet diagram, which means you would need to setup 6 instances of bastion hosts. Theoretically it's a sound decision, but practically it's an operation hell. Imagine the fun when you need to add more bastion host users (who might need multiple-site access as well). So that's a single user which need to exist on all bastion hosts (and we are not talking about permissions yet).

But you can do a magic networking incantation

You can use a networking trick to connect different sites together, which means in the end you only need a single instance of bastion host.

Previously I talked about warpgate, which you can easily set up to act as a bastion host with SSO support.

To punch a hole between different sites, you can use tailscale so devices are discoverable when direct links are not available. Although you should set tailscale acl so it would allow the bastion host to reach vms, but each vms are not discoverable via tailscale pipe - this is to limit the attack surface, in case a single vm is compromised. Without acl, attackers can reach other vms through a compromised vm.

The bastion host can be placed anywhere, either on-prem or cloud. To decide where to put the bastion host depends on your context and requirements. With tailscale acl applied, the network diagram would be:

%%{init: {'theme': 'dark'}}%%
graph TD
subgraph cloud ["Cloud"]
subgraph azure ["Azure"]
end
subgraph gcp ["GCP"]
end
subgraph aws ["AWS"]
end
end
subgraph onprem [On-Prem]
subgraph SiteC ["Your Other Friend's Home"]
end
subgraph SiteB ["Your Friend's Home"]
end
subgraph SiteA ["Your Home"]
end
end
BastionHost[Bastion Host] --> |tailscale pipe| SiteA
BastionHost[Bastion Host] --> |tailscale pipe| SiteB
BastionHost[Bastion Host] --> |tailscale pipe| SiteC
BastionHost[Bastion Host] --> |tailscale pipe| aws
BastionHost[Bastion Host] --> |tailscale pipe| gcp
BastionHost[Bastion Host] --> |tailscale pipe| azure

When you specify a bastion host target (that will be your vms), you can use assigned tailscale ip to refer to each vm. A single machine can have different IPs, depending on the network interface. Different network interfaces can be active at the same time, which means your vm is still accessible via direct ip, if you are within the same vpc, or you can use tailscale ip to access your vm regardless of which vpc you're in, as long as your tailscale tailnet is active.

A second network incantation

I have a setup where I have my own fleet, my friend has his own fleet. We both use our own tailscale. But I have a server at my friend's, which means this server uses my friend's tailnet. Multiple tailscale tailnets can't be active at the same time.

You can't use WireGuard either, because it occupies the same CIDR as tailscale's. But you can use iroh to punch a hole, caveat being you'll have to create a tunnel manually so the devices are discoverable through iroh. It's like tailscale, but at the application level. You can also create a thin wrapper for a target machine to expose ssh port, then on the bastion host you do a handshake via iroh endpoint id to allow the bastion host to reach the target machine. For my setup, the iroh app exposes a target vm's ssh port to 2222 on localhost, which means I can reference a remote vm in bastion host via localhost:2222.

But setting all this up is only half the battle. You should also put time into using IaC for this, at least for bastion host targets. Warpgate has an official terraform provider so it's a good start. Because it won't be fun when you have to do an inventory for your fleet. It's too easy for things to fall through the cracks.