1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
|
#import "/global/common.typ": *
#let doc = [
Doing it right this time.
Until now I've been using a
#extlink("https://pine64.org/devices/rockpro64/")[RockPro64] in the (excellent
for starters) Pine64 #extlink("https://wiki.pine64.org/wiki/NASCase")[NAS case]
for my NAS. It runs #extlink("https://www.armbian.com/rockpro64/")[Armbian],
`openzfs`, and I manage my ZFS as needed on the CLI. I share my datasets over
NFS to my Proxmox guest VMs for access. This has served me reasonably well for
the past few years. But we have a baby coming soon, and along with that a
flurry of new photos and videos for my family to store on my NAS. And although
I _probably_ have enough buffer to survive the initial few months, I certainly
wish to do better than my current 2xHDD mirrored pool giving me ~4TB of
storage.
#fig("nas-upgrade/oldnas.webp", alt: "nas nas")
= The Plan
Luckily I already possess the hard-drives I need for my desired pool. In total,
I have 2 HDDs from my current NAS, and 2 spares - all the same 4TB capacity.
I still want to keep my current NAS running as a backup store, but I only need
1 drive for this purpose (for now). That leaves me with 3 HDDs we can use for
our new `raidz1` pool which would give me 2 drives for storage and 1 for
parity.
Now for the platform. I have been a very happy user of
#extlink("https://www.proxmox.com/en/proxmox-virtual-environment/overview")[Proxmox]
(on my compute node) and it already comes ready with ZFS support so I'll stick
with that. I did consider #extlink("https://www.truenas.com")[TrueNAS] (both
standalone, and virtualized within Proxmox with disk-passthrough), but when it
comes to storage, I want as few surprises as possible so I stuck with what I
know. I'm already used to ZFS on the CLI and Proxmox is based on Debian so
if/when something goes wrong, I don't want to be messing around with an
unfamiliar UI.
Although I don't want to manage ZFS via a GUI, I'd like a way to manage
_access_ to my datasets via a GUI.
#extlink("https://cockpit-project.org")[Cockpit] seems like a good light-weight
choice. It runs as an LXC container within Proxmox using bind-mounts for the
dataset paths. I can then select which paths I allow access to over my network
using NFS, and SMB protocols on a per-user basis.
= Shopping
Needs:
- `>= 3` hot-swappable HDD bays
- `>= 16` GB RAM (for ZFS)
- runs Proxmox
Wants:
- IPMI
- 1U
- 10Gb networking
On eBay, I ended up checking out with:
#fig("nas-upgrade/nas-chassis.webp", alt: "nas chassis")
#fig("nas-upgrade/ethernet.webp", alt: "ethernet adapter")
#table(
columns: 2,
[Part], [Price (shipped)],
[1U 20" Short Depth Supermicro Server X9SCL-F XEON E3-1270 V2 16GB NIC Rails], [\$236],
[Mellanox ConnectX-2 PCIe x8 10Gbe SFP+ network card], [\$23],
)
which gave me everything I needed *and* wanted!
= Build
First I free one of my HDDs from my RockPro64 NAS for use in my new pool. This
means I am running on a single HDD for the remainder of the migration.
With the free HDD, and my two spares, I get everything assembled, and set up a
new Proxmox node with a ZFS pool from the UI using the 3 disks.
#fig("nas-upgrade/zfs-pool.webp", alt: "proxmox zfs pool")
= Sync
To migrate all of my data from the old pool to the new pool I use
#extlink("https://github.com/jimsalterjrs/sanoid/tree/master?tab=readme-ov-file#syncoid")[`syncoid`].
It uses ZFS snapshots to accomplish this. It's awesome.
This takes many hours depending on the size of your pool and your connection,
but since I saw almost full utilization of my network link I was convinced it's
the best I can do right now.
I've seen in a few Reddit posts suggesting ways this can be optimized using
some combination of `zfs send` with `mbuffer` rather than `ssh` which `syncoid`
uses, but I'm happy with the convenience of `syncoid`.
The best part is that everything can still be running while this is going on!
This is because only the first sync takes a long time since it needs to copy
_everything_ over. All subsequent syncs only transfer the delta since the last
sync which should be pretty quick if your data does not change all that much.
That being said, since I am now running on a single drive, and hitting it hard
during sync, I do not want to stay in this state for any longer than I need to
due to potentially leading my single disk to failure.
Once the sync is complete, I do a second sync (for the delta), and promptly
shut down all of my services to prepare for the cut-over.
= Cockpit
Before I can re-enable my services. I need to expose my datasets on Cockpit.
I create a Debian LXC, and assign the generated MAC a static IP on my network.
Other than that, I just make sure my container has the NFS feature enabled:
#fig("nas-upgrade/cockpit-lxc.webp", alt: "cockpit lxc features")
Setting up user permissions is a pain, but once done I expose the relevant NFS
as well as Samba paths for my services - replicating my old NAS's shared paths.
= Voila
In my homelab nameserver, I update the A record for my NAS to point to the new
IP of my Cockpit LXC container, and start my services back up.
I confirm everything is still good by turning off the old NAS, and yep still
good.
Luckily for me all good on the first try!
= Finishing touches
I set up a cron on my old NAS to sync data nightly from the new pool. This will
only work while the actual data size on my new pool is less than 4TB, so I'll
need to get a new drive with higher capacity to keep the backups going in the
future.
Luckily I should have about a year until I exceed 4TB on the new pool so I'll
be keeping an eye on deals to snag an 8TB backup drive along with a few spares
for my new NAS.
= Resources
- #extlink("https://www.apalrd.net/posts/2023/ultimate_nas/")[https://www.apalrd.net/posts/2023/ultimate_nas/]
- #extlink("https://blog.kye.dev/proxmox-cockpit")[https://blog.kye.dev/proxmox-cockpit]
]
|
