1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
|
---
title: "NAS Upgrade"
date: 2024-06-01T15:56:10-04:00
draft: true
description: "I upgrade my NAS hardware, and the platform it runs on."
author: "Kumar Damani"
---
Doing it right this time.
<!--more-->
Upto now I've been using a [RockPro64](https://pine64.org/devices/rockpro64/) in the (excellent for starters) Pine64 [NAS case](https://wiki.pine64.org/wiki/NASCase) for my NAS.
It runs [Armbian](https://www.armbian.com/rockpro64/), and `openzfs`, and I managed my ZFS as needed on the CLI.
I share my datasets over NFS to my Proxmox guest VMs for access.
This has served me reasonably well for the past few years.
But we have a baby comming soon, and along with that a flurry of new photos and videos
for my family to store on my NAS.
And although I *probably* have enough buffer to survive the inital few months,
I certainly wish to do better than my current 2xHDD mirrorred pool giving me ~4TB of storage.
# The Plan
Luckily I already possess the hard-drives I need for my desired pool.
In total I have 2 HDDs from my current NAS, and 2 spares -
all the same 4TB capacity.
I still want to keep my current NAS running as a backup store,
but I only need 1 drive for this purpose (for now).
That leaves me with 3 HDDs we can use for our new `raidz1` pool which would give
me 2 drives for storage and 1 for parity.
Now for the platform. I have been a very happy user of [Proxmox](https://www.proxmox.com/en/proxmox-virtual-environment/overview) (on my compute node)
and it already comes ready with ZFS support so I'll stick with that.
I did consider [TrueNAS](https://www.truenas.com) (both standalone, and virtualized within Proxmox with
disk-passthrough), but I when it comes to storage, I want as few surprises as possible
so I stuck with what I know. I'm already used to ZFS on the CLI and Proxmox is basically
Debian so if/when something goes wrong, I don't want to be messing around with an unfamiliar UI.
Although I don't want to manage ZFS via a GUI,
I'd like a way to manage *access* to my datasets via a GUI.
[Cockpit](https://cockpit-project.org) seems like a good light-weight choice.
It runs as an LXC container within Proxmox using bind-mounts for the dataset paths.
I can then select which paths I allow access to over my network
using NFS, and SMB protocols on a per-user basis.
# Shopping
Needs:
- `>= 3` hot-swappable HDD bays
- `>= 16` GB RAM (for ZFS)
- runs Proxmox
Wants:
- IPMI
- 1U
- 10Gb networking
On Ebay, I ended up checking-out with:
| Part | Price (shipped) |
| -------------- | --------------- |
| 1U 20" Short Depth Supermicro Server X9SCL-F XEON E3-1270 V2 16GB NIC Rails | $236 |
| Mellanox ConnectX-2 PCIe x8 10Gbe SFP+ network card | $23 |
which gave me everything I needed **and** wanted!
# Build
First I free one of my HDDs from my RockPro64 NAS for use in my new pool.
This means I am running on a single HDD for the remainder of the migration.
With the free HDD, and my two spares,
I get everything assembled, and setup a new Proxmox node with a ZFS pool
from the UI using the 3 disks.
# Sync
To migrate all of my data from the old pool to the new pool I use
[`syncoid`](https://github.com/jimsalterjrs/sanoid/tree/master?tab=readme-ov-file#syncoid).
It uses ZFS snapshots to accomplish this. Its awesome.
This takes many hours depending on the size of your pool and your connection,
but I since I saw almost full utilization of my network link
I was convinced its the best I can do right now.
I've seen in a few Reddit posts suggesting ways this can be optimized using
some combination of `zfs send` with `mbuffer` rather than `ssh` which
`syncoid` uses, but I'm happy with the convenience of `syncoid`.
The best part is that everything can still be running while this is going on!
This is because only the first sync takes a long time since it needs to copy
*everything* over. All subsequent syncs only transfer the delta
since the last sync which should be pretty quick if your data does not change
all that much.
That being said, since I am now running on a single drive, and hitting it hard during
sync, I do not want to stay in this state for any longer than I need to due to
potentially leading my single disk to failure.
Once the sync is complete, I do a second sync (for the delta), and promptly
shutdown all of my services to prepare for the cut-over.
# Cockpit
Before I can re-enable my services. I need to expose my datasets on Cockpit.
I create a Debian LXC, and assign the generated MAC a static IP on my network.
Other than that, I just make sure my container has the NFS feature enabled:
Setting up user permissions is a pain, but once done
I expose the relevant NFS as well as Samba paths for my services -
basically replicating my old NAS's shared paths.
# Voila
In my homelab nameserver, I update the A record
for my NAS to point to the new IP of my Cockpit LXC container,
and start my services back up.
I confirm everything is still good by turning off the old NAS, and yep still good.
Luckily for me all good on the first try!
# Finishing touches
I setup a cron on my old NAS to sync data nightly from the new pool.
This will only work while my the actual data size on my new pool is less than 4TB,
so I'll need to get a new drive with higher capacity to keep the backups going
in the future.
Luckily I should have about a year until I exceed 4TB on the new pool so I'll be keeping
an eye on deals to snag an 8TB backup drive along with a few spares for my new NAS.
# Resources
- https://www.apalrd.net/posts/2023/ultimate_nas/
- https://blog.kye.dev/proxmox-cockpit
|
