diff options
Diffstat (limited to 'post/framework-server.typ')
| -rw-r--r-- | post/framework-server.typ | 558 |
1 files changed, 558 insertions, 0 deletions
diff --git a/post/framework-server.typ b/post/framework-server.typ new file mode 100644 index 0000000..4e9d5b6 --- /dev/null +++ b/post/framework-server.typ @@ -0,0 +1,558 @@ +#import "/global/common.typ": * + +#let doc = [ +How I re-purposed my Framework as a standalone server for my home-lab. + +#quote(block: true)[ + This is not sponsored by Framework ... though I wish it were :) +] + += Primary Motivation + +I \#feelsbad for abusing my lowly NAS - + +#img("framework-server/feelsbadman.png", alt: "feelsbadman", width: "150px") + +Its not it's fault that its owner expected it to run Collabora Online while +running all his other services without instantly crashing... + +#fig("framework-server/error.jpg", alt: "meme of error page") + +Did I learn anything? Basically a NAS should just do NAS things like worry +about storage. Compute should be handled by something else. + +#quote(block: true)[Lesson learned.] + += Secondary Motivation + +Battery life is an issue with this gen. + +For my personal laptop I'd want something that has "all day" battery like the +newer gen Intels or Ryzens. + +My plan is to snag one of those when prices become more attractive and retrofit +it into my now hollow laptop chassis. + +Isn't this the Framework +#extlink("https://frame.work/ca/en/about#what-we-re-fixing")[mission]? + +#quote(block: true)[ + "Even better, what we've done to enable repair also opens up upgradability + and customization. This lets you get exactly the product you need and + extends usable lifetime too." +] + += Idea + +Re-purpose my 99%-of-the-time idle personal Framework laptop which is already +overkill for the remaining 1%-of-the-time tasks, to be a hypervisor server. + +Setup my home-lab services there, and provide data storage for those services +over NFS from my NAS. + +I'll setup a dedicated "workstation" VM for those 1%-of-the-time tasks. + +#fig("framework-server/physical.png", alt: "physical architecture image") + += Prior to starting! + +To avoid making the same mistakes I made, do the following *while the mainboard +is still in the laptop chassis!* + +#fig("framework-server/mistakes.jpg", alt: "mistakes meme") + ++ *Update your BIOS firmware to at least 3.17.* It is *impossible* to do so + once the battery is un-plugged from the board on older firmwares. I learned + this the hard-way. This update will provide the critical options below for + this process to be feasible. ++ In your BIOS settings, enable options for optimizing CPU performance over + battery life. ++ In your BIOS settings, enable "Standalone operation". ++ In your BIOS settings, under "Boot" enable the option to power-on on + power-attach. + #fig("framework-server/bios.jpg", alt: "bios image") ++ Install Proxmox VE. Since this will require a screen, and keyboard input, you + might as well do it now. See the #link(<hypervisor>)[hypervisor] section + below. + +#html.hr() + += Build + +== Server + +I'm using my original +#extlink("https://frame.work/ca/en/products/mainboard-11th-gen-intel-core?v=FRANFG000B")[Framework + 13] laptop. + +#fig("framework-server/case.jpg", alt: "case image") + +Its an i7-1165G7 - yes a "mobile" CPU. Am I asking for trouble? We'll see. Its +more than powerful enough for me (for now). + +Alternatively you could build/use another server. + +If you are also using the Framework 13, then you'll want to make sure that you +are running at least *BIOS 3.17*. See the #link(<issues>)[issues] section below +about my troubles. + +== Case + +As you can see above, I opted for the +#extlink("https://frame.work/ca/en/products/cooler-master-mainboard-case")[Coolermaster + Case] because honestly it looks cool. + +Alternatively, you could self-print the +#extlink("https://github.com/FrameworkComputer/Framework-Laptop-13")[community] +3D case or get someone else to print it. + +== Network Connectivity + +I opted for the 2.5G +#extlink("https://frame.work/ca/en/products/ethernet-expansion-card")[ethernet + expansion card] from Framework. + +#fig("framework-server/eth.jpg", alt: "framework eth adapter image") + +I suppose one could also use a WiFi card, but it makes more sense to go wired +in a 24/7 "on" configuration. + +Alternatively, most thunderbolt-ethernet dongles should do the job. You'd want +to check that there are Linux drivers for it. + +Since I host a few publicly accessible services (this blog for example), I also +need a public static IP under which an internet user can access these services. + +So I opt for a cheap VPS from +#extlink("https://www.racknerd.com/kvm-vps")[Racknerd]. These are about a \$1/mo +for a public IPv4 IP, and a decent amount of network bandwidth. + +#fig("framework-server/vps.png", alt: "racknerd vps pricing image") + +== Storage + +=== for VMs + +I used the 1TB NVME drive I had on my Framework. + +=== for Data + +_You should have a storage solution (NAS) in place already!_ + +#fig("framework-server/nas.jpg", alt: "nas image") + +Setting this up is beyond the scope of this post. You can check out my DIY ZFS +backed NAS setup +#extlink("https://gitlab.com/kdam0/vps#nas-equipment")[here]. I've had no +(storage related) issues with my NAS since its inception over a year ago. + +Then when I need to allocate space for a new service I do: + +```bash +zfs create <pool>/<dataset> +zfs set compression=on/off +zfs set sharenfs="..." sharesmb=".." +``` + +Then when the time comes, configure the NFS mount on the client VM that +requires access to this data store: + +```bash +mount -t nfs hostname-of-your-nas:/mnt/pool/dataset /mnt/data +``` + +There are also many options for pre-made NAS appliances such as Synology. + +== Power + +Highly recommend a UPS such as +#extlink("https://www.amazon.ca/APC-BE600M1-Back-Uninterrupted-Electronics-Computers/dp/B01FWAZEIU/ref=sr_1_6?crid=28ZH7JP9IYXPU&keywords=ups&qid=1690928556&sprefix=ups%2Caps%2C172&sr=8-6")[this]. + +#fig("framework-server/ups.jpg", alt: "ups image") + +Even if you don't care about uptime, *you should care about data integrity* +issues resulting from abrupt power-loss. + +A UPS should buy you enough time to power things down gracefully. + += Hypervisor <hypervisor> + +I opted for +#extlink("https://www.proxmox.com/en/proxmox-virtual-environment/overview")[Proxmox + VE] since v8 was just released, I've been itching to try it out. Now felt +like the right time. + +There is also a Terraform +#extlink("https://registry.terraform.io/providers/Telmate/proxmox/latest/docs")[provider] +allowing provisioning of VMs, and LXC Containers through code. Neat! + +#quote(block: true)[ + I recommend you do this *before* taking the mainboard out of the laptop + chassis as you will need to provide keyboard input, as well as see whats + going on the screen. +] + +- Install it. I just followed the + #extlink("https://www.proxmox.com/en/proxmox-virtual-environment/get-started")[official + instructions]. + +- Under your host (mine's `pve`), under System, Network, create a bridge + `vmbr0` with: + +#fig("framework-server/net1.png", alt: "proxmox net image") + +This will make your Proxmox host available in your home network. *Set this to +something else if you like.* + +#quote(block: true)[ + You should also reserve the corresponding IP in your home router for this + host. +] + +While you're here, create another bridge `vmbr1` with `VLAN Aware` checked: + +#fig("framework-server/net2.png", alt: "proxmox net image") + +#quote(block: true)[ + We are going to create our home-lab network primarily on `vmbr1`. Think of + `vmbr0` as a _management_ bridge. +] + += Network + +I always find it best to sort out networking *before* we start building stuff. + +Since I'm basically starting over, I want to do it properly. A few design +goals: + ++ I want segregation of my home-lab services from my non-home-lab hosts (my + "workstation" VM for example), as well as the rest of my home network. ++ I don't want to keep track of static IP leases, everything should be DHCP, + and DNS. ++ I want all traffic to/from my home-lab segment to be encrypted. Non-home-lab + traffic does not need to be encrypted. ++ I need some way to make these services publicly accessible over the internet. + +The first two goals necessitate a virtual firewall/router deployment capable of +_selectively_ routing a segment of my network over a VPN. + +Here is what I would like: + +#fig("framework-server/network.png", alt: "network architecture") + +== Goals: 1,2 + +At first I went with the popular pfSense VM. For a #link(<pfsense>)[few] +reasons, I abandoned it quickly. + +#html.hr() + +Then I decided to try out OpenWRT, and boy am I glad I did. + +#html.hr() + +I gave it two networking interfaces, each `vbmrX` we created earlier: + +#fig("framework-server/pfnet.png", alt: "openwrt bridge image") + +- `vmbr0` will be used to access our router from the home-network. It will also + be how our router gets access to the internet. Aka. *WAN*. +- `vmbr1` will be used to create various subnets for our home-lab. Aka. *LAN*. + +#quote(block: true)[ + You should reserve another IP in your home router using the MAC address + generated for `net0`. This will be how to access your home-lab router from + your home network. +] + +I was able to setup a network with a few VLANs: + +#fig("framework-server/wrtnet.png", alt: "openwrt network image") + +#quote(block: true)[ + This means that the next time we create a new VM or LXC container, we can + place it on `vmbr1`, with a particular VLAN Tag/`id` and it will start with + the corresponding IP for that network. +] + +Few observations: + +- The Luci web UI is much more minimal and to the point. The terminology used + for things are more in-line with what I'm familiar with. +- Soon I had the segregation I was looking for thanks to Firewall settings: + #fig("framework-server/fw1.png", alt: "firewall settings image") +- DHCP, *and* DNS working as expected. + +== Goals: 3 + +Then I started working on setting up my VPN (Wireguard) remote on my VPS. + +- I followed the script from + #extlink("https://github.com/angristan/wireguard-install")[here]. Worked + flawlessly. +- Added a peer through the script - my OpenWRT VM. +- Setup a domain - eg. `vpn.mydomain.com` to point to my VPS's public IP. +- Then I enabled the "DMZ" setting (but *not* "Advanced DMZ" - which is + probably the _real_ DMZ) on my home ISP router settings, and added my OpenWRT + VM to it. + #fig("framework-server/dmz.png", alt: "dmz image") +- I also setup a port-forward for the Wireguard port 51820 from my ISP router + settings, to point to my OpenWRT VM's internal port 51820. + #fig("framework-server/port1.png", alt: "port forward image") + +After all that, I started setting up Wireguard on OpenWRT. + +- The corresponding OpenWRT package is called `luci-app-wireguard`. +- This will give you the ability to create a Wireguard interface on OpenWRT. + #fig("framework-server/wg.png", alt: "wg menu image") + +- I followed the following excellent tutorial for setting up the Wireguard + interface, configured to use my VPS as its peer: + #extlink("https://www.youtube.com/watch?v=04q41GEPvKA")[Setting up Wireguard + on OpenWRT] +- Created the new interface and added the firewall settings: + #fig("framework-server/wrtnet2.png", alt: "openwrt wg ifrace") + #fig("framework-server/fw2.png", alt: "firewall wg settings image") + +- As is, this will mean that *only* LAB will have internet access (using the + Wireguard interface) - which is not what I want. I still want other subnets + to be routed via *WAN*. + +*_If only there was a way to route by VLANs..._* + +#quote(block: true)[ + This is known as VPN _split-tunnelling_, or _Policy based routing_. +] + +Luckily OpenWRT has a way to accomplish exactly this with a package called +`luci-app-pbr`. This will add the PBR setting to your menu: + +#fig("framework-server/pbr.png", alt: "pbr setting image") + +- I followed the following tutorial: + #extlink("https://www.youtube.com/watch?v=FN2qfxNIs2g")[Setting up PBR on + OpenWRT] + +- Set up the rule for my PBR: + #fig("framework-server/pbr2.png", alt: "pbr rule image") + +- The default interface is set to `wan`, so everything else should behave as + normal: + #fig("framework-server/pbr1.png", alt: "pbr global gateways image") + +However, it did take many trail & error sessions (see the +#link(<pbr-dns-leakage>)[dns leakage issues] section below) to get this working +exactly as I wanted. + +*In summary*, + +When I create a VM/container in my _Lab_ subnet, + +- its external IP is that of my VPS +- when it reaches out to the internet, it goes through the Wireguard tunnel + encrypted, to the VPS, and out the VPS's ISP. + #fig("framework-server/lab.png", alt: "lab flow image") + +When I create a VM/container *not* in my Lab subnet, + +- its external IP is the same as my home-network's public IP +- when it reaches out to the internet, it goes through my home ISP (NAT), just + like all other devices on my home-network. + #fig("framework-server/lan.png", alt: "lan flow image") + +PBR provides quite granular control over this, for example, if you run a Plex +server, it frequently needs to access the `plex.tv` domain. You can choose to +route any source going to `plex.tv` to route via the `wan` interface instead. + +== Goals: 4 + +For publicly accessible services, I need to point their domains to my VPS. I +use sub-domains, so this is just a matter of creating `CNAME` records for these +services. + +Then we setup our Wireguard on the VPS to forward all 80,443 TCP traffic to our +Wireguard on OpenWRT. We'd need to add these lines *in addition* to your +existing VPS `wg0` interface config: + +``` +PostUp = iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 80 -j DNAT --to-destination <ip of your OpenWRT wg0>:80 +PostUp = iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 443 -j DNAT --to-destination <ip of your OpenWRT wg0>:443 +... +PostDown = iptables -t nat -D PREROUTING -p tcp -i eth0 --dport 80 -j DNAT --to-destination <ip of your OpenWRT wg0>:80 +PostDown = iptables -t nat -D PREROUTING -p tcp -i eth0 --dport 443 -j DNAT --to-destination <ip of your OpenWRT wg0>:443 +``` + +Then we also need to setup port-forwards for 80,443 on OpenWRT going to our +Reverse Proxy VM/Container. + +#fig("framework-server/port2.png", alt: "port forwards image") + +I just setup a temporary Caddy container in reverse-proxy mode to test this +out. + +#fig("framework-server/rproxy.png", alt: "reverse-proxy setup image") + +Once this works, we can move on to building out our services :) + += Home Lab Services + +These are _some_ of the services I run ad LXC container: + +- #extlink("https://nextcloud.com/athome/")[Nextcloud] - files, and image + storage and sharing (thanks to the excellent + #extlink("https://apps.nextcloud.com/apps/memories")[Memories] app). I used + the official #extlink("https://github.com/nextcloud/all-in-one")[AIO] + install. +- #extlink("https://www.plex.tv/")[Plex] - media streaming. +- #extlink("https://github.com/dani-garcia/vaultwarden")[Vaultwarden] - a + self-hosted password manager based on Bitwarden. +- A few static websites. + +How I set up these services is beyond the scope of this post. You should check +out my git repo's #extlink("https://gitlab.com/kdam0/home-lab")[readme] for +instructions. I will just briefly summarize my process here. + +- I use LXC wherever possible as it has lower overhead than VMs. +- LXC or VM hosts are provisioned by Terraform and services are configured by + Ansible. + +One thing to note is that as of now, there is no automated way to pass-through +my iGPU to my LXC containers that would benefit from this - such as Nextcloud +(for Memories app), Plex etc. + +Instead you must edit the config on Proxmox for the corresponding container, +and add the following lines and restart the container. Edit +`/etc/pve/lxc/<id of your container>.conf`: + +``` +lxc.cgroup2.devices.allow: c 226:0 rwm +lxc.cgroup2.devices.allow: c 226:128 rwm +lxc.cgroup2.devices.allow: c 29:0 rwm +lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir +lxc.mount.entry: /dev/dri/renderD128 dev/renderD128 none bind,optional,create=file +``` + +Then inside your LXC container do `ls -al /dev/dri/` and you should see some +devices. + +#fig("framework-server/dri.png", alt: "gpu devices image") + += Issues <issues> + +== Intermittent connectivity loss with Framework + +- When I initially started this project, I had already installed Proxmox, and + my setup my networking while on BIOS 3.16. +- However every couple of days (sometimes hours) I was observing loss of + connectivity to both my Proxmox host, and my virtual router. +- When I plugged in a monitor to see what was going on, I saw a few messages on + the TTY regarding `usbX disconnected...`. +- Unplugging and re-plugging in the ethernet card did not fix it. +- Usually a hard reset was required to bring things back to normal. + +I've encountered issues with other Framework peripherals in the past, so I +reached out to the Framework support folks, and they immediately suggested a +BIOS update. + +So I proceeded to give that a shot from within Proxmox using `fwupmgr`: + +#fig("framework-server/fail.png", alt: "bios up fail image") + +And to my surprise it was complaining about _battery level_? Its not even +connected to the battery... + +I brought this up to the support and they confirmed my fear that it was not +possible to update the BIOS while having the battery detached. Yikes! + +This was a deal-breaker for this whole project, until they confirmed that once +I was on 3.17, it would be possible to do future updates without the battery +being connected. + +- So I went ahead and pretty much attached the whole thing back into the laptop + chassis +- Updated to 3.17, and applied "Standalone operation" + #fig("framework-server/bios2.png", alt: "bios update image") + +- Thankfully, all my connectivity issues have gone away now. +- Framework support service is excellent! + +== Intermittent loss of internet for hosts in Lab Vlan. + +This one kept me up for many nights. To summarize: + +- in a host in the Lab network, doing an `apt update` the first time would + hang, mid-way through +- running it a few times again, would eventually succeed the operation + #fig("framework-server/wtf.jpg", alt: "wtf meme") + +Could it be DNS? Could it be PBR? Could it be Wireguard? + +After many hours of troubleshooting, I happened to have seen errors during a +`tcpdump` session on my VPN interface. + +Something about `...need to fragment...`. DOH! + +- Lab internet access is via VPN (due to PBR). +- *VPN remote interface has an MTU of 1420, not 1500.* +- Proxmox `vmbr1` set MTU to 1500 by default. +- Therefore all hosts in Lab have a MTU mismatch trying to get to the internet. + +#quote(block: true)[Fixed. It was not DNS... it was MTU.] + +== PBR DNS leakage <pbr-dns-leakage> + +The whole point having my Lab traffic go through Wireguard is to prevent my +home ISP from knowing what I am up to. *DNS queries leaking to my ISP +completely defeats this purpose!* + +Although my external IP would correctly report to being my VPS one, tests like +#extlink("https://github.com/macvk/dnsleaktest")[this] would show that DNS was +still being sent to my home ISP. + +#fig("framework-server/dns.jpg", alt: "dns meme image") + +To fix this, I had to: + +- *Un-check* setting default route on the Lab interface. + #fig("framework-server/fix1.png", alt: "dns fix 1 image") + +- Explicitly set DHCP DNS settings for my Lab gateway interface to use my VPS's + `wg0` interface IP. + #fig("framework-server/fix2.png", alt: "dns fix 2 image") + +- Needed a static route defied for the `wg0` interface targeting the VPN + network via the OpenWRT local IP for `wg0` as a gateway. + #fig("framework-server/fix3.png", alt: "dns fix 3 image") + +== pfSense <pfsense> + +Few observations: + +- Segregation was working how I wanted it to. +- A few firewall rules were required to allow access to parts of my + home-network - NAS. +- DHCP, worked out of the box. +- However, DNS did not. + +I'm still not sure what the issue was, but it was probably something small. + +More importantly, I was quite overwhelmed with all the options in the UI for +which I had no knowledge of. + +Also, everything in pfSense pretty much _requires_ a UI, which for now is fine, +but I'd like the option to move to a more automation friendly configuration in +the future. + += Summary + +The Framework laptop is happily fulfilling its destiny as a compute server, and +so is my NAS as a storage server. + +Since switching to this setup, I've observed a dramatic improvement in +responsiveness for all of my services. + +#fig("framework-server/imp.png", alt: "improvement meme image") + +Overall I'm quite pleased with how everything turned out once the appropriate +BIOS was running on the Framework. +] |