Category Archives: HomeLab

The Hyperconverged Homelab—Growing RAIDZ vDevs

Share Button

Quickly approaching 85% utilization of my pool, I found myself in need of more storage capacity. Since the first revision of this project’s hardware was scrounged together on a small budget and utilized some already-owned drives, one of my vDevs ended up being a RAIDZ1 vDev of only 3x2TB. Adding more vDevs to my pool would require either an additional HBA (not possible with my now-undersized motherboard’s single PCIe slot) or a SAS expander. In either case, I would need the drives themselves. I figured that this was a good opportunity to experience growing a ZFS pool by increasing the size of a vDev’s disks.

ZFS does not support “growing” of arrays by adding disks (yet!), unlike some other RAID and RAID-like products. The only way to increase the size of a pool (think of it as pooling the capacity of a bunch of individual RAID arrays) is to add vDevs (the individual RAID arrays in this example), or to replace every single disk in a vDev with a larger capacity. vDevs can be constructed out of mixed-size disks, but are limited to the maximum capacity of the smallest disk. For example, a ZFS vDev containing 2x 2TB and 1x 1TB disks has the same usable capacity as one containing 3x 1TB disks: the “extra” is ignored and unused. Replace the lone undersized disk, however, and ZFS can grow the vDev to the full available size.

Expanding vDevs is a replace-in-place strategy that essentially works the same as rebuilding (“resilvering”) after a disk failure. Recent versions of ZFS support manually replacing a disk without first failing it out of the vDev, which means that on single-parity (RAIDZ1) vDevs this process can be accomplished safely, without losing fault-tolerance. The FreeNAS documentation provides more information and instructions.

Growing by “too much” is not recommended and will result in poor performance, as some metadata will be an non-optimal size for the new disk size. As far as I have read (unfortunately I can’t find a link for this), it’s definitely “too much” around an order of magnitude, although aiming for no more than a factor of five is probably wise. For my case, as an example, we’re growing from 2TB disks to 6TB disks, which is only a factor of 3. This should be perfectly fine.

Speaking of 6TB drives… Hard drives may be cheap in historical terms, but there’s still value in being thrifty. For my use-case, which currently includes read-oriented archival storage, grown mostly write-only and used for backups and media storage, accessed by 1Gb network links, the performance requirements are rather low. The data is (mostly) replaceable, so single redundancy is adequate. This means that I can safely use the cheapest hard drives possible, which are currently found in Seagate Backup Plus Hub 8TB carried by Costco for only $129. (At the time I purchased, the last of their stock of the 6TB variant was being cleared for even cheaper.)

These drives are Seagate Baracuda ST8000DM005, which are an SMR drive. This technology, which has been used to great effect to increase the size of cheap consumer drives, essentially by overlapping the data on the platters, is only really suitable for write-once use and is known to be rather failure-prone. However, these have plenty of cache and perform just fine for reading, and adequately for writing, so are perfectly acceptable for my use-case.

Growing the target vDev was fairly straightforward. I had extra drive bays unused so simply shucked the drives from their plastic enclosures and proceeded one at a time. After formatting each disk for FreeNAS, I initiated the resilvering process. This took somewhere between 36–48 hours to resilver 1.7TB of data per drive. I found this performance rather poor, but was not able to locate an obvious bottleneck at the time. In hindsight, inadequate RAM was likely the cause. After resilvering I removed the old drive to make room for the next replacement. Although my drive bays are hot-swap (and this is supported by both my HBA and FreeNAS), I didn’t label the drive bays when I installed them initially and had some difficulty identifying the unused drives. The best solution I found was to leverage the per-disk activity light of the Rosewill hotswap cages.

A lovely sight.

With capacity to spare, I can finally test out some new backup strategies to support, such as Time Machine over SMB.

The Hyperocnverged Homelab—Configuration c.2018

Share Button

Although my original use-case included virtualizing a router/firewall, it was only beneficial for a couple months while I was still living in accommodation with a shared network. I ran OpenWRT for simplicity of configuration and had two separate vSwitches configured in ESXi, one for each NIC. This allowed me to connect to the shared network while retaining control over my own subnet and not leaking device access or mDNS. I had hoped to pass through the motherboard’s 802.11ac WiFi NIC (which worked fine), but was stymied by OpenWRT’s glacial upgrade cycle. They were running an absolutely ancient version of the Linux kernel which predated support for my WiFi chipset. I considered working around this by creating a virtual Access Point using a VM of Ubuntu Server or other lightweight Linux which would support the WiFi chipset, but it just wasn’t worth the trouble.

After spending a couple months abroad with the server powered down I returned home and found a new apartment. I was able to get CenturyLink’s symmetric Gigabit offering installed, and running their provided router eliminated the need for a virtual router appliance. The OpenWRT VM was quickly mothballed and replaced with an Ubuntu Server 18.04 VM to run Ubiquiti’s UniFi Controller.

The current (Dec. 2018) software configuration is fairly simple:

  • ESXi Server 6.5
    • FreeNAS 9.10
      • 12GB RAM, 4vCPU, 8GB boot disk
      • IBM M1015 IT Mode via PCIe passthrough
      • 2x RAIDZ1 vDevs of 3 disks (consumer 2 and 5TB drives)
      • Jails for utilities benefiting from direct pool access
    • Ubuntu Server 18.04
      • 2GB RAM, 2vCPU, 8GB boot disk
      • Ubiquiti UniFi Controller
      • DIY Linode dynamic dns

The Hyperconverged HomeLab—Introduction

Share Button

Now in its second relatively trouble-free year, it’s finally time to get some upgrades on my hyperconverged homelab. First, however, a long-overdue introduction!

The current case configuration: a modified Cooler Master Centurion 590 mid-tower case.

This project started out as a compact, low-power, ultra-quiet NAS build. However, I quickly decided that I wanted to virtualize and give myself more power and flexibility. At the very least, being able to run pfSense or another router/firewall appliance on the same device represented a significant benefit in terms of portability: the ability to plug into basically any network without making the NAS available on it was a huge potential benefit.

I decided to use a 35W Intel desktop processor and consumer motherboard. They’re economical and readily available, with plenty of products available for performance and cooling enhancement. At the time, Skylake (6th Gen.) was mature and Kaby Lake didn’t have an official release date, so I chose the i5-6500T. The $100 premium on MSRP and near total lack of single unit availability prevented me from choosing an i7-6700T.

For motherboard I chose Gigabyte’s GA-H87N-WIFI (rev. 2.0), a mini-ITX motherboard from their well-regarded UltraDurable line. The primary driver of this decision was the onboard dual 1GBase-T and M.2 802.11a/b/g/n plus Bluetooth 4.0 via M.2 card. Dual LAN was critical for the device’s potential use as a router, as virtualizing my NAS would require utilizing the single available PCIe slot for an HBA or RAID card.

RAM was sourced as 2x16GB G.Skill Aegis modules (still the cheapest DDR4-2133 2x16GB kit on the market), providing a solid starting point while leaving two DIMMs free for later expansion to the motherboard and processor’s max supported 64GB. I sourced a Seasonic SS460FL2 a 460W fanless modular PSU, a cheap SanDisk 240GB SSD for a boot drive, and Corsair’s H115i all-in-one liquid cooling loop.

At this time I was still case-less, and waffling on the purchase of a U-NAS NSC-800 hot-swap enclosure, when I discovered Rosewill’s 4-in-3 hot swap cages. I quickly located the Cooler Master Centurion 590 on local Craigslist, which represented a decent compromise on size and offered 9 5.25″ drive bays.

The final piece of the puzzle was the HBA, an IBM M1015 RAID card which I cross-flashed to LSI generic IT Mode firmware. See this other post for details. With that, the build was hardware-complete and went together (fairly) smoothly. Only minor case modification was required to fit the ridiculously over-sized water cooling radiator, which had to be mounted on the top of the case with the fans inside, since the case was not designed for water cooling and here was inadequate clearance above the motherboard.

I installed ESXi on the boot disk and then installed FreeNAS into a VM. (Yes, I should have drive redundancy for my VM datastore.) After flashing the M1015 everything was relatively plug-and-play, set-and-forget, with the only notable downside being that the motherboard refused to POST without detecting an attached display. That issue was solved when I discovered that an HDMI VGA adapter I purchased acted as a display simulator. This system served me well for the last couple years, but recently I’ve wanted to expand my capabilities. Having a single PCIe slot is somewhat limiting, especially since I didn’t end up buying a mini-ITX sized case…

Crossflash IBM M1015 to LSI 9220-8i IT Mode for FreeNAS

Share Button

The IBM M1015 is a widely available LSI SAS2008-based RAID controller card. It is an extremely popular choice for home and enthusiast server builders, especially among FreeNAS users, for its low price point (~$60 US secondhand on eBay) and excellent performance.

In essence, it’s hardware equivalent to the LSI 9211-8i; officially, it’s the 9220-8i, sold to OEMs to be rebadged. Two SFF-8087 mini-SAS quad-channel SAS2/SATA3 ports, no cache, no battery backup. Cross-flash it to LSI generic firmware in IT mode, they say, and you get an excellent SATA III HBA on the cheap. Turns out that’s easier said than done, especially if you’re working with a recent consumer motherboard.

The comprehensive, only slightly dated, instructions are here. Ironically, I only found them after I had pieced together the procedure for myself.

At this point, FreeNAS 9.10 is compatible with version P20 firmware. User Spearfoot on the FreeNAS forums has a package containing the utilities and firmware files. I’ve also attached it to this post: m1015.

Notes:

  • If your motherboard lacks an easily accessible EFI shell, use the one in rEFInd.
  • If you get the error “application not started from shell”, that’s an EFI shell version compatibility issue. Use the shell provided in the link.
  • “No LSI SAS adapters found!” from sas2flsh.exe indicates that likely IBM firmware is still present. Use megarec to erase it.
  • “ERROR: Failed to initialize PAL. Exiting Program.” means your motherboard is not compatible with the DOS sas2flsh. Use the EFI version.

Additional References:

  • GeekGoneOld on the FreeNAS forums has a quick guide: #18
  • And a useful reply in that same thread: #28
  • Redditor /u/PhyxsiusPrime describes the EFI shell compatibility workaround via rEFInd here.