diff options
Diffstat (limited to 'content')
38 files changed, 488 insertions, 0 deletions
diff --git a/content/post/framework-server/bios.jpg b/content/post/framework-server/bios.jpg Binary files differnew file mode 100644 index 0000000..76cda35 --- /dev/null +++ b/content/post/framework-server/bios.jpg diff --git a/content/post/framework-server/bios2.png b/content/post/framework-server/bios2.png Binary files differnew file mode 100644 index 0000000..094e37e --- /dev/null +++ b/content/post/framework-server/bios2.png diff --git a/content/post/framework-server/case.jpg b/content/post/framework-server/case.jpg Binary files differnew file mode 100644 index 0000000..a40c172 --- /dev/null +++ b/content/post/framework-server/case.jpg diff --git a/content/post/framework-server/dmz.png b/content/post/framework-server/dmz.png Binary files differnew file mode 100644 index 0000000..372c12b --- /dev/null +++ b/content/post/framework-server/dmz.png diff --git a/content/post/framework-server/dns.jpg b/content/post/framework-server/dns.jpg Binary files differnew file mode 100644 index 0000000..0b32528 --- /dev/null +++ b/content/post/framework-server/dns.jpg diff --git a/content/post/framework-server/dri.png b/content/post/framework-server/dri.png Binary files differnew file mode 100644 index 0000000..a578c79 --- /dev/null +++ b/content/post/framework-server/dri.png diff --git a/content/post/framework-server/error.jpg b/content/post/framework-server/error.jpg Binary files differnew file mode 100644 index 0000000..429eef8 --- /dev/null +++ b/content/post/framework-server/error.jpg diff --git a/content/post/framework-server/eth.jpg b/content/post/framework-server/eth.jpg Binary files differnew file mode 100644 index 0000000..f8ad12a --- /dev/null +++ b/content/post/framework-server/eth.jpg diff --git a/content/post/framework-server/fail.png b/content/post/framework-server/fail.png Binary files differnew file mode 100644 index 0000000..f0e06db --- /dev/null +++ b/content/post/framework-server/fail.png diff --git a/content/post/framework-server/feelsbadman.png b/content/post/framework-server/feelsbadman.png Binary files differnew file mode 100644 index 0000000..b1c2a26 --- /dev/null +++ b/content/post/framework-server/feelsbadman.png diff --git a/content/post/framework-server/fix1.png b/content/post/framework-server/fix1.png Binary files differnew file mode 100644 index 0000000..b36489e --- /dev/null +++ b/content/post/framework-server/fix1.png diff --git a/content/post/framework-server/fix2.png b/content/post/framework-server/fix2.png Binary files differnew file mode 100644 index 0000000..5b64900 --- /dev/null +++ b/content/post/framework-server/fix2.png diff --git a/content/post/framework-server/fix3.png b/content/post/framework-server/fix3.png Binary files differnew file mode 100644 index 0000000..9af2433 --- /dev/null +++ b/content/post/framework-server/fix3.png diff --git a/content/post/framework-server/fw1.png b/content/post/framework-server/fw1.png Binary files differnew file mode 100644 index 0000000..72a95ee --- /dev/null +++ b/content/post/framework-server/fw1.png diff --git a/content/post/framework-server/fw2.png b/content/post/framework-server/fw2.png Binary files differnew file mode 100644 index 0000000..cec00e1 --- /dev/null +++ b/content/post/framework-server/fw2.png diff --git a/content/post/framework-server/imp.png b/content/post/framework-server/imp.png Binary files differnew file mode 100644 index 0000000..c11b806 --- /dev/null +++ b/content/post/framework-server/imp.png diff --git a/content/post/framework-server/index.md b/content/post/framework-server/index.md new file mode 100644 index 0000000..8e57e60 --- /dev/null +++ b/content/post/framework-server/index.md @@ -0,0 +1,488 @@ ++++ +author = "Kumar Damani" +title = "Helping my Framework Laptop reach its destiny" +date = "2023-07-29" +draft = false +description = "Framework laptop as a standalone server." +tags = [ + "technology" +] +math = false ++++ + +How I re-purposed my Framework as a standalone server for my home-lab. +<!--more--> + +> This is not sponsored by Framework ... though I wish it were :) + +# Primary Motivation +I #feelsbad for abusing my lowly NAS - + +{{< figure src="feelsbadman.png" title="" width="150px" >}} + +Its not it's fault that its owner expected it to run Collabora Online +while running all his other services without instantly crashing... + + + +Did I learn anything? +Basically a NAS should just do NAS things like worry about storage. +Compute should be handled by something else. + +> Lesson learned. + + +# Secondary Motivation +Battery life is an issue with this gen. + +For my personal laptop I'd want something that has "all day" +battery like the newer gen Intels or Ryzens. + +My plan is to snag one of those when prices become +more attractive and retrofit it into my now hollow laptop chassis. + +Isn't this the Framework [mission](https://frame.work/ca/en/about#what-we-re-fixing)? + +> "Even better, what we've done to enable repair also opens up upgradability and customization. +This lets you get exactly the product you need and extends usable lifetime too." + + +# Idea +Re-purpose my 99%-of-the-time idle personal Framework laptop which is already +overkill for the remaining 1%-of-the-time tasks, to be a hypervisor server. + +Setup my home-lab services there, and provide data storage for those services over NFS from my NAS. + +I'll setup a dedicated "workstation" VM for those 1%-of-the-time tasks. + + + + +# Prior to starting! +To avoid making the same mistakes I made, do the following **while +the mainboard is still in the laptop chassis!** + + + +1. **Update your BIOS firmware to at least 3.17.** +It is **impossible** to do so once the battery is un-plugged from the board on older firmwares. +I learned this the hard-way. +This update will provide the critical options below for this process to be feasible. +1. In your BIOS settings, enable options for optimizing CPU performance over +battery life. +1. In your BIOS settings, enable "Standalone operation". +1. In your BIOS settings, under "Boot" enable the option to power-on on power-attach. + +1. Install Proxmox VE. Since this will require a screen, and keyboard input, you might as +well do it now. See the [hypervisor]({{< relref "#hypervisor" >}}) section below. + + +--- + +# Build +## Server +I'm using my original [Framework 13](https://frame.work/ca/en/products/mainboard-11th-gen-intel-core?v=FRANFG000B) laptop. + + + +Its an i7-1165G7 - yes a "mobile" CPU. +Am I asking for trouble? We'll see. +Its more than powerful enough for me (for now). + +Alternatively you could build/use another server. + +If you are also using the Framework 13, then you'll want to make sure +that you are running at least **BIOS 3.17**. +See the [issues]({{< relref "#issues" >}}) section below about my troubles. + + +## Case +As you can see above, I opted for the +[Coolermaster Case](https://frame.work/ca/en/products/cooler-master-mainboard-case) +because honestly it looks cool. + +Alternatively, you could self-print the [community](https://github.com/FrameworkComputer/Framework-Laptop-13) 3D case +or get someone else to print it. + + +## Network Connectivity +I opted for the 2.5G [ethernet expansion card](https://frame.work/ca/en/products/ethernet-expansion-card) +from Framework. + + + +I suppose one could also use a WiFi card, +but it makes more sense to go wired in a 24/7 "on" configuration. + +Alternatively, most thunderbolt-ethernet dongles should do the job. +You'd want to check that there are Linux drivers for it. + +Since I host a few publicly accessible services (this blog for example), +I also need a public static IP under which an internet user can access these services. + +So I opt for a cheap VPS from [Racknerd](https://www.racknerd.com/kvm-vps). +These are about a $1/mo for a public IPv4 IP, and a decent amount of network bandwidth. + + + +## Storage +### for VMs +I used the 1TB NVME drive I had on my Framework. + +### for Data +*You should have a storage solution (NAS) in place already!* + + + +Setting this up is beyond the scope of this post. +You can check out my DIY ZFS backed NAS setup [here](https://gitlab.com/kdam0/vps#nas-equipment). +I've had no (storage related) issues with my NAS since its inception over a year ago. + +Then when I need to allocate space for a new service I do: +```shell +zfs create <pool>/<dataset> +zfs set compression=on/off +zfs set sharenfs="..." sharesmb=".." +``` + +Then when the time comes, configure the NFS mount on the client VM that requires access to this +data store: +```bash +mount -t nfs hostname-of-your-nas:/mnt/pool/dataset /mnt/data +``` + +There are also many options for pre-made NAS appliances such as Synology. + + +## Power +Highly recommend a UPS such as [this](https://www.amazon.ca/APC-BE600M1-Back-Uninterrupted-Electronics-Computers/dp/B01FWAZEIU/ref=sr_1_6?crid=28ZH7JP9IYXPU&keywords=ups&qid=1690928556&sprefix=ups%2Caps%2C172&sr=8-6). + + + +Even if you don't care about uptime, **you should care about data integrity** issues +resulting from abrupt power-loss. + +A UPS should buy you enough time to power things down gracefully. + + +# Hypervisor +I opted for [Proxmox VE](https://www.proxmox.com/en/proxmox-virtual-environment/overview) +since v8 was just released, I've been itching to try it out. Now felt like the right time. + +There is also a Terraform [provider](https://registry.terraform.io/providers/Telmate/proxmox/latest/docs) +allowing provisioning of VMs, and LXC Containers through code. Neat! + +> I recommend you do this **before** taking the mainboard out of the laptop chassis as you +will need to provide keyboard input, as well as see whats going on the screen. + +* Install it. +I just followed the [official instructions](https://www.proxmox.com/en/proxmox-virtual-environment/get-started). + +* Under your host (mine's `pve`), under System, Network, create a bridge `vmbr0` with: + + + +This will make your Proxmox host available in your home network. +**Set this to something else if you like.** +> You should also reserve the corresponding IP in your home router for this host. + +While you're here, create another bridge `vmbr1` with `VLAN Aware` checked: + + + +> We are going to create our home-lab network primarily on `vmbr1`. +> Think of `vmbr0` as a *management* bridge. + +# Network +I always find it best to sort out networking **before** we start building stuff. + +Since I'm basically starting over, I want to do it properly. +A few design goals: + +1. I want segregation of my home-lab services from my non-home-lab hosts +(my "workstation" VM for example), as well as the rest of my home network. +1. I don't want to keep track of static IP leases, everything should be DHCP, and DNS. +1. I want all traffic to/from my home-lab segment to be encrypted. Non-home-lab +traffic does not need to be encrypted. +1. I need some way to make these services publicly accessible over the internet. + +The first two goals necessitate a virtual firewall/router deployment capable of +*selectively* routing a segment of my network over a VPN. + +Here is what I would like: + + + +## Goals: 1,2 +At first I went with the popular [pfSense]() VM. +For a [few]({{< relref "#pfsense" >}}) reasons, I abandoned it quickly. + +--- + +Then I decided to try out [OpenWRT](), and boy am I glad I did. + +--- + +I gave it two networking interfaces, each `vbmrX` we created earlier: + + +* `vmbr0` will be used to access our router from the home-network. +It will also be how our router gets access to the internet. Aka. **WAN**. +* `vmbr1` will be used to create various subnets for our home-lab. Aka. **LAN**. + +> You should reserve another IP in your home router using the MAC address generated for `net0`. +This will be how to access your home-lab router from your home network. + +I was able to setup a network with a few VLANs: + + +> This means that the next time we create a new VM or LXC container, we +can place it on `vmbr1`, with a particular VLAN Tag/`id` and it will start with +the corresponding IP for that network. + + +Few observations: +* The Luci web UI is much more minimal and to the point. +The terminology used for things are more in-line with what I'm familiar with. +* Soon I had the segregation I was looking for thanks to Firewall settings: + +* DHCP, **and** DNS working as expected. + +## Goals: 3 +Then I started working on setting up my VPN (Wireguard) remote on my VPS. + +* I followed the script from [here](https://github.com/angristan/wireguard-install). +Worked flawlessly. +* Added a peer through the script - my OpenWRT VM. +* Setup a domain - eg. `vpn.mydomain.com` to point to my VPS's public IP. +* Then I enabled the "DMZ" +setting (but **not** "Advanced DMZ" - which is probably the *real* DMZ) +on my home ISP router settings, and added my OpenWRT VM to it. + +* I also setup a port-forward for the Wireguard port 51820 from my ISP router settings, +to point to my OpenWRT VM's internal port 51820. + + +After all that, I started setting up Wireguard on OpenWRT. +* The corresponding OpenWRT package is called `luci-app-wireguard`. +* This will give you the ability to create a Wireguard interface on OpenWRT. + + +* I followed the following excellent tutorial for setting up the Wireguard interface, +configured to use my VPS as its peer: +{{< youtube id="04q41GEPvKA" title="Setting up Wireguard on OpenWRT" >}} +* Created the new interface and added the firewall settings: + + + +* As is, this will mean that **only** LAB will have internet access (using the Wireguard interface) - +which is not what I want. I still want other subnets to be routed via **WAN**. + +***If only there was a way to route by VLANs...*** + +> This is known as VPN *split-tunnelling*, or *Policy based routing*. + +Luckily OpenWRT has a way to accomplish exactly this with a package called `luci-app-pbr`. +This will add the PBR setting to your menu: + + + +* I followed the following tutorial: +{{< youtube id="FN2qfxNIs2g" title="Setting up PBR on OpenWRT" >}} + +* Set up the rule for my PBR: + + +* The default interface is set to `wan`, so everything else should behave as normal: + + +However, it did take many trail & error sessions +(see the [dns leakage issues]({{< relref "#pbr-dns-leakage" >}}) section below) +to get this working exactly as I wanted. + +**In summary**, + +When I create a VM/container in my *Lab* subnet, +* its external IP is that of my VPS +* when it reaches out to the internet, it goes through the Wireguard tunnel encrypted, +to the VPS, and out the VPS's ISP. + + +When I create a VM/container **not** in my Lab subnet, +* its external IP is the same as my home-network's public IP +* when it reaches out to the internet, it goes through my home ISP +(NAT), just like all other devices on my home-network. + + +PBR provides quite granular control over this, for example, if you run a Plex server, it frequently +needs to access the `plex.tv` domain. +You can choose to route any source going to `plex.tv` to route via the `wan` interface instead. + + + +## Goals: 4 +For publicly accessible services, I need to point their domains to my VPS. +I use sub-domains, so this is just a matter of creating `CNAME` records for these services. + +Then we setup our Wireguard on the VPS to forward all 80,443 TCP traffic to our +Wireguard on OpenWRT. +We'd need to add these lines **in addition** to your existing VPS `wg0` interface config: +``` +PostUp = iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 80 -j DNAT --to-destination <ip of your OpenWRT wg0>:80 +PostUp = iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 443 -j DNAT --to-destination <ip of your OpenWRT wg0>:443 +... +PostDown = iptables -t nat -D PREROUTING -p tcp -i eth0 --dport 80 -j DNAT --to-destination <ip of your OpenWRT wg0>:80 +PostDown = iptables -t nat -D PREROUTING -p tcp -i eth0 --dport 443 -j DNAT --to-destination <ip of your OpenWRT wg0>:443 +``` + +Then we also need to setup port-forwards for 80,443 on OpenWRT going to our Reverse Proxy VM/Container. + + +I just setup a temporary Caddy container in reverse-proxy mode to test this out. + + +Once this works, we can move on to building out our services :) + +# Home Lab Services +These are *some* of the services I run ad LXC container: +* [Nextcloud](https://nextcloud.com/athome/) - files, and image storage and sharing (thanks to the excellent [Memories](https://apps.nextcloud.com/apps/memories) app). I used the official [AIO](https://github.com/nextcloud/all-in-one) install. +* [Plex](https://www.plex.tv/) - media streaming. +* [Vaultwarden](https://github.com/dani-garcia/vaultwarden) - a self-hosted password manager based on Bitwarden. +* A few static websites. + +How I set up these services is beyond the scope of this post. +You should check out my git repo's [readme]() for instructions. +I will just briefly summarize my process here. +(I will update the link on this page when its ready) + +* I use LXC wherever possible as it has lower overhead than VMs. +* LXC or VM hosts are provisioned by Terraform and services are configured +by Ansible. + +One thing to note is that as of now, there is no automated way to pass-through my +iGPU to my LXC containers that would benefit from this - +such as Nextcloud (for Memories app), Plex etc. + +Instead you must edit the config on Proxmox for the corresponding container, and add +the following lines and restart the container. Edit `/etc/pve/lxc/<id of your container>.conf`: +``` +lxc.cgroup2.devices.allow: c 226:0 rwm +lxc.cgroup2.devices.allow: c 226:128 rwm +lxc.cgroup2.devices.allow: c 29:0 rwm +lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir +lxc.mount.entry: /dev/dri/renderD128 dev/renderD128 none bind,optional,create=file +``` +Then inside your LXC container do `ls -al /dev/dri/` and you should see some devices. + + + +# Issues +## Intermittent connectivity loss with Framework +* When I initially started this project, I had already installed +Proxmox, and my setup my networking while on BIOS 3.16. +* However every couple of days (sometimes hours) I was observing loss of connectivity +to both my Proxmox host, and my virtual router. +* When I plugged in a monitor to see what was going on, I saw a few +messages on the TTY regarding `usbX disconnected...`. +* Unplugging and re-plugging in the ethernet card did not fix it. +* Usually a hard reset was required to bring things back to normal. + +I've encountered issues with other Framework peripherals in the past, +so I reached out to the Framework support folks, +and they immediately suggested a BIOS update. + +So I proceeded to give that a shot from within Proxmox using `fwupmgr`: + + +And to my surprise it was complaining about *battery level*? +Its not even connected to the battery... + +I brought this up to the support and they confirmed my fear +that it was not possible to update the BIOS while having the battery detached. +Yikes! + +This was a deal-breaker for this whole project, until they confirmed +that once I was on 3.17, +it would be possible to do future updates without the battery being connected. + +* So I went ahead and pretty much attached the whole thing back into the laptop chassis +* Updated to 3.17, and applied "Standalone operation" + + +* Thankfully, all my connectivity issues have gone away now. +* Framework support service is excellent! + + +## Intermittent loss of internet for hosts in Lab Vlan. +This one kept me up for many nights. To summarize: +* in a host in the Lab network, doing an `apt update` the first time would hang, +mid-way through +* running it a few times again, would eventually succeed the operation + + +Could it be DNS? +Could it be PBR? +Could it be Wireguard? + +After many hours of troubleshooting, I happened to have +seen errors during a `tcpdump` session on my VPN interface. + +Something about `...need to fragment...`. DOH! +* Lab internet access is via VPN (due to PBR). +* **VPN remote interface has an MTU of 1420, not 1500.** +* Proxmox `vmbr1` set MTU to 1500 by default. +* Therefore all hosts in Lab have a MTU mismatch trying +to get to the internet. + +> Fixed. It was not DNS... it was MTU. + +## PBR DNS leakage +The whole point having my Lab traffic go through Wireguard is +to prevent my home ISP from knowing what I am up to. +**DNS queries leaking to my ISP completely defeats this purpose!** + +Although my external IP would correctly report to being my VPS one, +tests like [this](https://github.com/macvk/dnsleaktest) +would show that DNS was still being sent to my home ISP. + + + +To fix this, I had to: +* **Un-check** setting default route on the Lab interface. + + +* Explicitly set DHCP DNS settings for my Lab gateway interface to +use my VPS's `wg0` interface IP. + + +* Needed a static route defied for the `wg0` interface targeting the VPN network +via the OpenWRT local IP for `wg0` as a gateway. + + +## pfSense +Few observations: +* Segregation was working how I wanted it to. +* A few firewall rules were required to allow access to parts of my home-network - NAS. +* DHCP, worked out of the box. +* However, DNS did not. + +I'm still not sure what the issue was, but it was probably something small. + +More importantly, I was quite overwhelmed with all the options in the UI for which I had no +knowledge of. + +Also, everything in pfSense pretty much *requires* a UI, which for now is fine, but +I'd like the option to move to a more automation friendly configuration in the future. + + +# Summary +The Framework laptop is happily fulfilling its destiny as a compute server, +and so is my NAS as a storage server. + +Since switching to this setup, I've observed a dramatic improvement +in responsiveness for all of my services. + + + +Overall I'm quite pleased with how everything turned out once the +appropriate BIOS was running on the Framework. diff --git a/content/post/framework-server/lab.png b/content/post/framework-server/lab.png Binary files differnew file mode 100644 index 0000000..6591a8b --- /dev/null +++ b/content/post/framework-server/lab.png diff --git a/content/post/framework-server/lan.png b/content/post/framework-server/lan.png Binary files differnew file mode 100644 index 0000000..461d598 --- /dev/null +++ b/content/post/framework-server/lan.png diff --git a/content/post/framework-server/mistakes.jpg b/content/post/framework-server/mistakes.jpg Binary files differnew file mode 100644 index 0000000..d4b2e12 --- /dev/null +++ b/content/post/framework-server/mistakes.jpg diff --git a/content/post/framework-server/nas.jpg b/content/post/framework-server/nas.jpg Binary files differnew file mode 100644 index 0000000..75f5251 --- /dev/null +++ b/content/post/framework-server/nas.jpg diff --git a/content/post/framework-server/net1.png b/content/post/framework-server/net1.png Binary files differnew file mode 100644 index 0000000..a0a4eff --- /dev/null +++ b/content/post/framework-server/net1.png diff --git a/content/post/framework-server/net2.png b/content/post/framework-server/net2.png Binary files differnew file mode 100644 index 0000000..c5ad118 --- /dev/null +++ b/content/post/framework-server/net2.png diff --git a/content/post/framework-server/network.png b/content/post/framework-server/network.png Binary files differnew file mode 100644 index 0000000..1b1fba6 --- /dev/null +++ b/content/post/framework-server/network.png diff --git a/content/post/framework-server/pbr.png b/content/post/framework-server/pbr.png Binary files differnew file mode 100644 index 0000000..2b1e330 --- /dev/null +++ b/content/post/framework-server/pbr.png diff --git a/content/post/framework-server/pbr1.png b/content/post/framework-server/pbr1.png Binary files differnew file mode 100644 index 0000000..4d7f075 --- /dev/null +++ b/content/post/framework-server/pbr1.png diff --git a/content/post/framework-server/pbr2.png b/content/post/framework-server/pbr2.png Binary files differnew file mode 100644 index 0000000..0a07839 --- /dev/null +++ b/content/post/framework-server/pbr2.png diff --git a/content/post/framework-server/pfnet.png b/content/post/framework-server/pfnet.png Binary files differnew file mode 100644 index 0000000..67cc5f4 --- /dev/null +++ b/content/post/framework-server/pfnet.png diff --git a/content/post/framework-server/physical.png b/content/post/framework-server/physical.png Binary files differnew file mode 100644 index 0000000..fc1cb81 --- /dev/null +++ b/content/post/framework-server/physical.png diff --git a/content/post/framework-server/port1.png b/content/post/framework-server/port1.png Binary files differnew file mode 100644 index 0000000..1b72ab1 --- /dev/null +++ b/content/post/framework-server/port1.png diff --git a/content/post/framework-server/port2.png b/content/post/framework-server/port2.png Binary files differnew file mode 100644 index 0000000..37ff887 --- /dev/null +++ b/content/post/framework-server/port2.png diff --git a/content/post/framework-server/rproxy.png b/content/post/framework-server/rproxy.png Binary files differnew file mode 100644 index 0000000..0d3f0bf --- /dev/null +++ b/content/post/framework-server/rproxy.png diff --git a/content/post/framework-server/ups.jpg b/content/post/framework-server/ups.jpg Binary files differnew file mode 100644 index 0000000..4cd7aee --- /dev/null +++ b/content/post/framework-server/ups.jpg diff --git a/content/post/framework-server/vps.png b/content/post/framework-server/vps.png Binary files differnew file mode 100644 index 0000000..7392495 --- /dev/null +++ b/content/post/framework-server/vps.png diff --git a/content/post/framework-server/wg.png b/content/post/framework-server/wg.png Binary files differnew file mode 100644 index 0000000..18896a0 --- /dev/null +++ b/content/post/framework-server/wg.png diff --git a/content/post/framework-server/wrtnet.png b/content/post/framework-server/wrtnet.png Binary files differnew file mode 100644 index 0000000..73d435d --- /dev/null +++ b/content/post/framework-server/wrtnet.png diff --git a/content/post/framework-server/wrtnet2.png b/content/post/framework-server/wrtnet2.png Binary files differnew file mode 100644 index 0000000..34cd6b9 --- /dev/null +++ b/content/post/framework-server/wrtnet2.png diff --git a/content/post/framework-server/wtf.jpg b/content/post/framework-server/wtf.jpg Binary files differnew file mode 100644 index 0000000..995fefe --- /dev/null +++ b/content/post/framework-server/wtf.jpg |
