path: root/post/framework-server.typ

#import "/global/common.typ": *

#let doc = [
How I re-purposed my Framework as a standalone server for my home-lab.

#quote(block: true)[
    This is not sponsored by Framework ... though I wish it were :)
]

= Primary Motivation

I \#feelsbad for abusing my lowly NAS -

#img("framework-server/feelsbadman.webp", alt: "feelsbadman", width: "150px")

Its not it's fault that its owner expected it to run Collabora Online while
running all his other services without instantly crashing...

#fig("framework-server/error.webp", alt: "meme of error page")

Did I learn anything? Basically a NAS should just do NAS things like worry
about storage. Compute should be handled by something else.

#quote(block: true)[Lesson learned.]

= Secondary Motivation

Battery life is an issue with this gen.

For my personal laptop I'd want something that has "all day" battery like the
newer gen Intels or Ryzens.

My plan is to snag one of those when prices become more attractive and retrofit
it into my now hollow laptop chassis.

Isn't this the Framework
#extlink("https://frame.work/ca/en/about#what-we-re-fixing")[mission]?

#quote(block: true)[
    "Even better, what we've done to enable repair also opens up upgradability
    and customization. This lets you get exactly the product you need and
    extends usable lifetime too."
]

= Idea

Re-purpose my 99%-of-the-time idle personal Framework laptop which is already
overkill for the remaining 1%-of-the-time tasks, to be a hypervisor server.

Setup my home-lab services there, and provide data storage for those services
over NFS from my NAS.

I'll setup a dedicated "workstation" VM for those 1%-of-the-time tasks.

#fig("framework-server/physical.webp", alt: "physical architecture image")

= Prior to starting!

To avoid making the same mistakes I made, do the following *while the mainboard
is still in the laptop chassis!*

#fig("framework-server/mistakes.webp", alt: "mistakes meme")

+ *Update your BIOS firmware to at least 3.17.* It is *impossible* to do so
  once the battery is un-plugged from the board on older firmwares. I learned
  this the hard-way. This update will provide the critical options below for
  this process to be feasible.
+ In your BIOS settings, enable options for optimizing CPU performance over
  battery life.
+ In your BIOS settings, enable "Standalone operation".
+ In your BIOS settings, under "Boot" enable the option to power-on on
  power-attach.
  #fig("framework-server/bios.webp", alt: "bios image")
+ Install Proxmox VE. Since this will require a screen, and keyboard input, you
  might as well do it now. See the #link(<hypervisor>)[hypervisor] section
  below.

#html.hr()

= Build

== Server

I'm using my original
#extlink("https://frame.work/ca/en/products/mainboard-11th-gen-intel-core?v=FRANFG000B")[Framework
  13] laptop.

#fig("framework-server/case.webp", alt: "case image")

Its an i7-1165G7 - yes a "mobile" CPU. Am I asking for trouble? We'll see. Its
more than powerful enough for me (for now).

Alternatively you could build/use another server.

If you are also using the Framework 13, then you'll want to make sure that you
are running at least *BIOS 3.17*. See the #link(<issues>)[issues] section below
about my troubles.

== Case

As you can see above, I opted for the
#extlink("https://frame.work/ca/en/products/cooler-master-mainboard-case")[Coolermaster
  Case] because honestly it looks cool.

Alternatively, you could self-print the
#extlink("https://github.com/FrameworkComputer/Framework-Laptop-13")[community]
3D case or get someone else to print it.

== Network Connectivity

I opted for the 2.5G
#extlink("https://frame.work/ca/en/products/ethernet-expansion-card")[ethernet
  expansion card] from Framework.

#fig("framework-server/eth.webp", alt: "framework eth adapter image")

I suppose one could also use a WiFi card, but it makes more sense to go wired
in a 24/7 "on" configuration.

Alternatively, most thunderbolt-ethernet dongles should do the job. You'd want
to check that there are Linux drivers for it.

Since I host a few publicly accessible services (this blog for example), I also
need a public static IP under which an internet user can access these services.

So I opt for a cheap VPS from
#extlink("https://www.racknerd.com/kvm-vps")[Racknerd]. These are about a \$1/mo
for a public IPv4 IP, and a decent amount of network bandwidth.

#fig("framework-server/vps.webp", alt: "racknerd vps pricing image")

== Storage

=== for VMs

I used the 1TB NVME drive I had on my Framework.

=== for Data

_You should have a storage solution (NAS) in place already!_

#fig("framework-server/nas.webp", alt: "nas image")

Setting this up is beyond the scope of this post. You can check out my DIY ZFS
backed NAS setup
#extlink("https://gitlab.com/kdam0/vps#nas-equipment")[here]. I've had no
(storage related) issues with my NAS since its inception over a year ago.

Then when I need to allocate space for a new service I do:

```bash
zfs create <pool>/<dataset>
zfs set compression=on/off
zfs set sharenfs="..." sharesmb=".."
```

Then when the time comes, configure the NFS mount on the client VM that
requires access to this data store:

```bash
mount -t nfs hostname-of-your-nas:/mnt/pool/dataset /mnt/data
```

There are also many options for pre-made NAS appliances such as Synology.

== Power

Highly recommend a UPS such as
#extlink("https://www.amazon.ca/APC-BE600M1-Back-Uninterrupted-Electronics-Computers/dp/B01FWAZEIU/ref=sr_1_6?crid=28ZH7JP9IYXPU&keywords=ups&qid=1690928556&sprefix=ups%2Caps%2C172&sr=8-6")[this].

#fig("framework-server/ups.webp", alt: "ups image")

Even if you don't care about uptime, *you should care about data integrity*
issues resulting from abrupt power-loss.

A UPS should buy you enough time to power things down gracefully.

= Hypervisor <hypervisor>

I opted for
#extlink("https://www.proxmox.com/en/proxmox-virtual-environment/overview")[Proxmox
  VE] since v8 was just released, I've been itching to try it out. Now felt
like the right time.

There is also a Terraform
#extlink("https://registry.terraform.io/providers/Telmate/proxmox/latest/docs")[provider]
allowing provisioning of VMs, and LXC Containers through code. Neat!

#quote(block: true)[
    I recommend you do this *before* taking the mainboard out of the laptop
    chassis as you will need to provide keyboard input, as well as see whats
    going on the screen.
]

- Install it. I just followed the
  #extlink("https://www.proxmox.com/en/proxmox-virtual-environment/get-started")[official
    instructions].

- Under your host (mine's `pve`), under System, Network, create a bridge
  `vmbr0` with:

#fig("framework-server/net1.webp", alt: "proxmox net image")

This will make your Proxmox host available in your home network. *Set this to
something else if you like.*

#quote(block: true)[
    You should also reserve the corresponding IP in your home router for this
    host.
]

While you're here, create another bridge `vmbr1` with `VLAN Aware` checked:

#fig("framework-server/net2.webp", alt: "proxmox net image")

#quote(block: true)[
    We are going to create our home-lab network primarily on `vmbr1`. Think of
    `vmbr0` as a _management_ bridge.
]

= Network

I always find it best to sort out networking *before* we start building stuff.

Since I'm basically starting over, I want to do it properly. A few design
goals:

+ I want segregation of my home-lab services from my non-home-lab hosts (my
  "workstation" VM for example), as well as the rest of my home network.
+ I don't want to keep track of static IP leases, everything should be DHCP,
  and DNS.
+ I want all traffic to/from my home-lab segment to be encrypted. Non-home-lab
  traffic does not need to be encrypted.
+ I need some way to make these services publicly accessible over the internet.

The first two goals necessitate a virtual firewall/router deployment capable of
_selectively_ routing a segment of my network over a VPN.

Here is what I would like:

#fig("framework-server/network.webp", alt: "network architecture")

== Goals: 1,2

At first I went with the popular pfSense VM. For a #link(<pfsense>)[few]
reasons, I abandoned it quickly.

#html.hr()

Then I decided to try out OpenWRT, and boy am I glad I did.

#html.hr()

I gave it two networking interfaces, each `vbmrX` we created earlier:

#fig("framework-server/pfnet.webp", alt: "openwrt bridge image")

- `vmbr0` will be used to access our router from the home-network. It will also
  be how our router gets access to the internet. Aka. *WAN*.
- `vmbr1` will be used to create various subnets for our home-lab. Aka. *LAN*.

#quote(block: true)[
    You should reserve another IP in your home router using the MAC address
    generated for `net0`. This will be how to access your home-lab router from
    your home network.
]

I was able to setup a network with a few VLANs:

#fig("framework-server/wrtnet.webp", alt: "openwrt network image")

#quote(block: true)[
    This means that the next time we create a new VM or LXC container, we can
    place it on `vmbr1`, with a particular VLAN Tag/`id` and it will start with
    the corresponding IP for that network.
]

Few observations:

- The Luci web UI is much more minimal and to the point. The terminology used
  for things are more in-line with what I'm familiar with.
- Soon I had the segregation I was looking for thanks to Firewall settings:
  #fig("framework-server/fw1.webp", alt: "firewall settings image")
- DHCP, *and* DNS working as expected.

== Goals: 3

Then I started working on setting up my VPN (Wireguard) remote on my VPS.

- I followed the script from
  #extlink("https://github.com/angristan/wireguard-install")[here]. Worked
  flawlessly.
- Added a peer through the script - my OpenWRT VM.
- Setup a domain - eg. `vpn.mydomain.com` to point to my VPS's public IP.
- Then I enabled the "DMZ" setting (but *not* "Advanced DMZ" - which is
  probably the _real_ DMZ) on my home ISP router settings, and added my OpenWRT
  VM to it.
  #fig("framework-server/dmz.webp", alt: "dmz image")
- I also setup a port-forward for the Wireguard port 51820 from my ISP router
  settings, to point to my OpenWRT VM's internal port 51820.
  #fig("framework-server/port1.webp", alt: "port forward image")

After all that, I started setting up Wireguard on OpenWRT.

- The corresponding OpenWRT package is called `luci-app-wireguard`.
- This will give you the ability to create a Wireguard interface on OpenWRT.
  #fig("framework-server/wg.webp", alt: "wg menu image")

- I followed the following excellent tutorial for setting up the Wireguard
  interface, configured to use my VPS as its peer:
  #extlink("https://www.youtube.com/watch?v=04q41GEPvKA")[Setting up Wireguard
    on OpenWRT]
- Created the new interface and added the firewall settings:
  #fig("framework-server/wrtnet2.webp", alt: "openwrt wg ifrace")
  #fig("framework-server/fw2.webp", alt: "firewall wg settings image")

- As is, this will mean that *only* LAB will have internet access (using the
  Wireguard interface) - which is not what I want. I still want other subnets
  to be routed via *WAN*.

*_If only there was a way to route by VLANs..._*

#quote(block: true)[
    This is known as VPN _split-tunnelling_, or _Policy based routing_.
]

Luckily OpenWRT has a way to accomplish exactly this with a package called
`luci-app-pbr`. This will add the PBR setting to your menu:

#fig("framework-server/pbr.webp", alt: "pbr setting image")

- I followed the following tutorial:
  #extlink("https://www.youtube.com/watch?v=FN2qfxNIs2g")[Setting up PBR on
    OpenWRT]

- Set up the rule for my PBR:
  #fig("framework-server/pbr2.webp", alt: "pbr rule image")

- The default interface is set to `wan`, so everything else should behave as
  normal:
  #fig("framework-server/pbr1.webp", alt: "pbr global gateways image")

However, it did take many trail & error sessions (see the
#link(<pbr-dns-leakage>)[dns leakage issues] section below) to get this working
exactly as I wanted.

*In summary*,

When I create a VM/container in my _Lab_ subnet,

- its external IP is that of my VPS
- when it reaches out to the internet, it goes through the Wireguard tunnel
  encrypted, to the VPS, and out the VPS's ISP.
  #fig("framework-server/lab.webp", alt: "lab flow image")

When I create a VM/container *not* in my Lab subnet,

- its external IP is the same as my home-network's public IP
- when it reaches out to the internet, it goes through my home ISP (NAT), just
  like all other devices on my home-network.
  #fig("framework-server/lan.webp", alt: "lan flow image")

PBR provides quite granular control over this, for example, if you run a Plex
server, it frequently needs to access the `plex.tv` domain. You can choose to
route any source going to `plex.tv` to route via the `wan` interface instead.

== Goals: 4

For publicly accessible services, I need to point their domains to my VPS. I
use sub-domains, so this is just a matter of creating `CNAME` records for these
services.

Then we setup our Wireguard on the VPS to forward all 80,443 TCP traffic to our
Wireguard on OpenWRT. We'd need to add these lines *in addition* to your
existing VPS `wg0` interface config:

```
PostUp = iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 80 -j DNAT --to-destination <ip of your OpenWRT wg0>:80
PostUp = iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 443 -j DNAT --to-destination <ip of your OpenWRT wg0>:443
...
PostDown = iptables -t nat -D PREROUTING -p tcp -i eth0 --dport 80 -j DNAT --to-destination <ip of your OpenWRT wg0>:80
PostDown = iptables -t nat -D PREROUTING -p tcp -i eth0 --dport 443 -j DNAT --to-destination <ip of your OpenWRT wg0>:443
```

Then we also need to setup port-forwards for 80,443 on OpenWRT going to our
Reverse Proxy VM/Container.

#fig("framework-server/port2.webp", alt: "port forwards image")

I just setup a temporary Caddy container in reverse-proxy mode to test this
out.

#fig("framework-server/rproxy.webp", alt: "reverse-proxy setup image")

Once this works, we can move on to building out our services :)

= Home Lab Services

These are _some_ of the services I run ad LXC container:

- #extlink("https://nextcloud.com/athome/")[Nextcloud] - files, and image
  storage and sharing (thanks to the excellent
  #extlink("https://apps.nextcloud.com/apps/memories")[Memories] app). I used
  the official #extlink("https://github.com/nextcloud/all-in-one")[AIO]
  install.
- #extlink("https://www.plex.tv/")[Plex] - media streaming.
- #extlink("https://github.com/dani-garcia/vaultwarden")[Vaultwarden] - a
  self-hosted password manager based on Bitwarden.
- A few static websites.

How I set up these services is beyond the scope of this post. You should check
out my git repo's #extlink("https://gitlab.com/kdam0/home-lab")[readme] for
instructions. I will just briefly summarize my process here.

- I use LXC wherever possible as it has lower overhead than VMs.
- LXC or VM hosts are provisioned by Terraform and services are configured by
  Ansible.

One thing to note is that as of now, there is no automated way to pass-through
my iGPU to my LXC containers that would benefit from this - such as Nextcloud
(for Memories app), Plex etc.

Instead you must edit the config on Proxmox for the corresponding container,
and add the following lines and restart the container. Edit
`/etc/pve/lxc/<id of your container>.conf`:

```
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.cgroup2.devices.allow: c 29:0 rwm
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/dri/renderD128 dev/renderD128 none bind,optional,create=file
```

Then inside your LXC container do `ls -al /dev/dri/` and you should see some
devices.

#fig("framework-server/dri.webp", alt: "gpu devices image")

= Issues <issues>

== Intermittent connectivity loss with Framework

- When I initially started this project, I had already installed Proxmox, and
  my setup my networking while on BIOS 3.16.
- However every couple of days (sometimes hours) I was observing loss of
  connectivity to both my Proxmox host, and my virtual router.
- When I plugged in a monitor to see what was going on, I saw a few messages on
  the TTY regarding `usbX disconnected...`.
- Unplugging and re-plugging in the ethernet card did not fix it.
- Usually a hard reset was required to bring things back to normal.

I've encountered issues with other Framework peripherals in the past, so I
reached out to the Framework support folks, and they immediately suggested a
BIOS update.

So I proceeded to give that a shot from within Proxmox using `fwupmgr`:

#fig("framework-server/fail.webp", alt: "bios up fail image")

And to my surprise it was complaining about _battery level_? Its not even
connected to the battery...

I brought this up to the support and they confirmed my fear that it was not
possible to update the BIOS while having the battery detached. Yikes!

This was a deal-breaker for this whole project, until they confirmed that once
I was on 3.17, it would be possible to do future updates without the battery
being connected.

- So I went ahead and pretty much attached the whole thing back into the laptop
  chassis
- Updated to 3.17, and applied "Standalone operation"
  #fig("framework-server/bios2.webp", alt: "bios update image")

- Thankfully, all my connectivity issues have gone away now.
- Framework support service is excellent!

== Intermittent loss of internet for hosts in Lab Vlan.

This one kept me up for many nights. To summarize:

- in a host in the Lab network, doing an `apt update` the first time would
  hang, mid-way through
- running it a few times again, would eventually succeed the operation
  #fig("framework-server/wtf.webp", alt: "wtf meme")

Could it be DNS? Could it be PBR? Could it be Wireguard?

After many hours of troubleshooting, I happened to have seen errors during a
`tcpdump` session on my VPN interface.

Something about `...need to fragment...`. DOH!

- Lab internet access is via VPN (due to PBR).
- *VPN remote interface has an MTU of 1420, not 1500.*
- Proxmox `vmbr1` set MTU to 1500 by default.
- Therefore all hosts in Lab have a MTU mismatch trying to get to the internet.

#quote(block: true)[Fixed. It was not DNS... it was MTU.]

== PBR DNS leakage <pbr-dns-leakage>

The whole point having my Lab traffic go through Wireguard is to prevent my
home ISP from knowing what I am up to. *DNS queries leaking to my ISP
completely defeats this purpose!*

Although my external IP would correctly report to being my VPS one, tests like
#extlink("https://github.com/macvk/dnsleaktest")[this] would show that DNS was
still being sent to my home ISP.

#fig("framework-server/dns.webp", alt: "dns meme image")

To fix this, I had to:

- *Un-check* setting default route on the Lab interface.
  #fig("framework-server/fix1.webp", alt: "dns fix 1 image")

- Explicitly set DHCP DNS settings for my Lab gateway interface to use my VPS's
  `wg0` interface IP.
  #fig("framework-server/fix2.webp", alt: "dns fix 2 image")

- Needed a static route defied for the `wg0` interface targeting the VPN
  network via the OpenWRT local IP for `wg0` as a gateway.
  #fig("framework-server/fix3.webp", alt: "dns fix 3 image")

== pfSense <pfsense>

Few observations:

- Segregation was working how I wanted it to.
- A few firewall rules were required to allow access to parts of my
  home-network - NAS.
- DHCP, worked out of the box.
- However, DNS did not.

I'm still not sure what the issue was, but it was probably something small.

More importantly, I was quite overwhelmed with all the options in the UI for
which I had no knowledge of.

Also, everything in pfSense pretty much _requires_ a UI, which for now is fine,
but I'd like the option to move to a more automation friendly configuration in
the future.

= Summary

The Framework laptop is happily fulfilling its destiny as a compute server, and
so is my NAS as a storage server.

Since switching to this setup, I've observed a dramatic improvement in
responsiveness for all of my services.

#fig("framework-server/imp.webp", alt: "improvement meme image")

Overall I'm quite pleased with how everything turned out once the appropriate
BIOS was running on the Framework.
]