All posts by Dakota

The Hyperconverged Homelab—Growing RAIDZ vDevs

Share Button

Quickly approaching 85% utilization of my pool, I found myself in need of more storage capacity. Since the first revision of this project’s hardware was scrounged together on a small budget and utilized some already-owned drives, one of my vDevs ended up being a RAIDZ1 vDev of only 3x2TB. Adding more vDevs to my pool would require either an additional HBA (not possible with my now-undersized motherboard’s single PCIe slot) or a SAS expander. In either case, I would need the drives themselves. I figured that this was a good opportunity to experience growing a ZFS pool by increasing the size of a vDev’s disks.

ZFS does not support “growing” of arrays by adding disks (yet!), unlike some other RAID and RAID-like products. The only way to increase the size of a pool (think of it as pooling the capacity of a bunch of individual RAID arrays) is to add vDevs (the individual RAID arrays in this example), or to replace every single disk in a vDev with a larger capacity. vDevs can be constructed out of mixed-size disks, but are limited to the maximum capacity of the smallest disk. For example, a ZFS vDev containing 2x 2TB and 1x 1TB disks has the same usable capacity as one containing 3x 1TB disks: the “extra” is ignored and unused. Replace the lone undersized disk, however, and ZFS can grow the vDev to the full available size.

Expanding vDevs is a replace-in-place strategy that essentially works the same as rebuilding (“resilvering”) after a disk failure. Recent versions of ZFS support manually replacing a disk without first failing it out of the vDev, which means that on single-parity (RAIDZ1) vDevs this process can be accomplished safely, without losing fault-tolerance. The FreeNAS documentation provides more information and instructions.

Growing by “too much” is not recommended and will result in poor performance, as some metadata will be an non-optimal size for the new disk size. As far as I have read (unfortunately I can’t find a link for this), it’s definitely “too much” around an order of magnitude, although aiming for no more than a factor of five is probably wise. For my case, as an example, we’re growing from 2TB disks to 6TB disks, which is only a factor of 3. This should be perfectly fine.

Speaking of 6TB drives… Hard drives may be cheap in historical terms, but there’s still value in being thrifty. For my use-case, which currently includes read-oriented archival storage, grown mostly write-only and used for backups and media storage, accessed by 1Gb network links, the performance requirements are rather low. The data is (mostly) replaceable, so single redundancy is adequate. This means that I can safely use the cheapest hard drives possible, which are currently found in Seagate Backup Plus Hub 8TB carried by Costco for only $129. (At the time I purchased, the last of their stock of the 6TB variant was being cleared for even cheaper.)

These drives are Seagate Baracuda ST8000DM005, which are an SMR drive. This technology, which has been used to great effect to increase the size of cheap consumer drives, essentially by overlapping the data on the platters, is only really suitable for write-once use and is known to be rather failure-prone. However, these have plenty of cache and perform just fine for reading, and adequately for writing, so are perfectly acceptable for my use-case.

Growing the target vDev was fairly straightforward. I had extra drive bays unused so simply shucked the drives from their plastic enclosures and proceeded one at a time. After formatting each disk for FreeNAS, I initiated the resilvering process. This took somewhere between 36–48 hours to resilver 1.7TB of data per drive. I found this performance rather poor, but was not able to locate an obvious bottleneck at the time. In hindsight, inadequate RAM was likely the cause. After resilvering I removed the old drive to make room for the next replacement. Although my drive bays are hot-swap (and this is supported by both my HBA and FreeNAS), I didn’t label the drive bays when I installed them initially and had some difficulty identifying the unused drives. The best solution I found was to leverage the per-disk activity light of the Rosewill hotswap cages.

A lovely sight.

With capacity to spare, I can finally test out some new backup strategies to support, such as Time Machine over SMB.

The Hyperocnverged Homelab—Configuration c.2018

Share Button

Although my original use-case included virtualizing a router/firewall, it was only beneficial for a couple months while I was still living in accommodation with a shared network. I ran OpenWRT for simplicity of configuration and had two separate vSwitches configured in ESXi, one for each NIC. This allowed me to connect to the shared network while retaining control over my own subnet and not leaking device access or mDNS. I had hoped to pass through the motherboard’s 802.11ac WiFi NIC (which worked fine), but was stymied by OpenWRT’s glacial upgrade cycle. They were running an absolutely ancient version of the Linux kernel which predated support for my WiFi chipset. I considered working around this by creating a virtual Access Point using a VM of Ubuntu Server or other lightweight Linux which would support the WiFi chipset, but it just wasn’t worth the trouble.

After spending a couple months abroad with the server powered down I returned home and found a new apartment. I was able to get CenturyLink’s symmetric Gigabit offering installed, and running their provided router eliminated the need for a virtual router appliance. The OpenWRT VM was quickly mothballed and replaced with an Ubuntu Server 18.04 VM to run Ubiquiti’s UniFi Controller.

The current (Dec. 2018) software configuration is fairly simple:

  • ESXi Server 6.5
    • FreeNAS 9.10
      • 12GB RAM, 4vCPU, 8GB boot disk
      • IBM M1015 IT Mode via PCIe passthrough
      • 2x RAIDZ1 vDevs of 3 disks (consumer 2 and 5TB drives)
      • Jails for utilities benefiting from direct pool access
    • Ubuntu Server 18.04
      • 2GB RAM, 2vCPU, 8GB boot disk
      • Ubiquiti UniFi Controller
      • DIY Linode dynamic dns

The Hyperconverged HomeLab—Introduction

Share Button

Now in its second relatively trouble-free year, it’s finally time to get some upgrades on my hyperconverged homelab. First, however, a long-overdue introduction!

The current case configuration: a modified Cooler Master Centurion 590 mid-tower case.

This project started out as a compact, low-power, ultra-quiet NAS build. However, I quickly decided that I wanted to virtualize and give myself more power and flexibility. At the very least, being able to run pfSense or another router/firewall appliance on the same device represented a significant benefit in terms of portability: the ability to plug into basically any network without making the NAS available on it was a huge potential benefit.

I decided to use a 35W Intel desktop processor and consumer motherboard. They’re economical and readily available, with plenty of products available for performance and cooling enhancement. At the time, Skylake (6th Gen.) was mature and Kaby Lake didn’t have an official release date, so I chose the i5-6500T. The $100 premium on MSRP and near total lack of single unit availability prevented me from choosing an i7-6700T.

For motherboard I chose Gigabyte’s GA-H87N-WIFI (rev. 2.0), a mini-ITX motherboard from their well-regarded UltraDurable line. The primary driver of this decision was the onboard dual 1GBase-T and M.2 802.11a/b/g/n plus Bluetooth 4.0 via M.2 card. Dual LAN was critical for the device’s potential use as a router, as virtualizing my NAS would require utilizing the single available PCIe slot for an HBA or RAID card.

RAM was sourced as 2x16GB G.Skill Aegis modules (still the cheapest DDR4-2133 2x16GB kit on the market), providing a solid starting point while leaving two DIMMs free for later expansion to the motherboard and processor’s max supported 64GB. I sourced a Seasonic SS460FL2 a 460W fanless modular PSU, a cheap SanDisk 240GB SSD for a boot drive, and Corsair’s H115i all-in-one liquid cooling loop.

At this time I was still case-less, and waffling on the purchase of a U-NAS NSC-800 hot-swap enclosure, when I discovered Rosewill’s 4-in-3 hot swap cages. I quickly located the Cooler Master Centurion 590 on local Craigslist, which represented a decent compromise on size and offered 9 5.25″ drive bays.

The final piece of the puzzle was the HBA, an IBM M1015 RAID card which I cross-flashed to LSI generic IT Mode firmware. See this other post for details. With that, the build was hardware-complete and went together (fairly) smoothly. Only minor case modification was required to fit the ridiculously over-sized water cooling radiator, which had to be mounted on the top of the case with the fans inside, since the case was not designed for water cooling and here was inadequate clearance above the motherboard.

I installed ESXi on the boot disk and then installed FreeNAS into a VM. (Yes, I should have drive redundancy for my VM datastore.) After flashing the M1015 everything was relatively plug-and-play, set-and-forget, with the only notable downside being that the motherboard refused to POST without detecting an attached display. That issue was solved when I discovered that an HDMI VGA adapter I purchased acted as a display simulator. This system served me well for the last couple years, but recently I’ve wanted to expand my capabilities. Having a single PCIe slot is somewhat limiting, especially since I didn’t end up buying a mini-ITX sized case…

Smart TVs Enable Creepy Ads That Follow You

Share Button

Since as early as 2013, the misleadingly-named, San Francisco-based Free Stream Media Corp. has touted smart TV software capable of detecting what you’re watching. Initially marketed as a social tool to drive viewer engagement, the software has morphed into an Orwellian advertising spy machine. Called “Samba TV” since its debut at CES in 2013, the software comes pre-installed on select Smart TV sets from a dozen manufacturers, including Sharp, Toshiba, Sony, and Philips. Claiming to provide consumers who opt in with “recommendations based on the content you love”, the software in fact monitors everything displayed on the TV to identify not only broadcast advertisements but also streaming services and even video games and internet videos.

This data is then distributed to advertisers in real time. The result: creepy targeted ads that know what you’re watching.

Christine DiLandro, a marketing director at Citi, joined Mr. Navin at an industry event at the end of 2015. In a video of the event, Ms. DiLandro described the ability to target people with digital ads after the company’s TV commercials aired as “a little magical.”

This accomplishment is a result of Samba’s “device map”, which appears to utilize a combination of local network exploration and mobile device fingerprinting to identify smartphones, tables, and other computers in the same household as an enabled Smart TV. This allows the company to target ads to other devices based on what’s on TV.

Presumably they’re also building a profile of your viewing habits to sell to advertisers as well. Yikes.

US Cell Phone Carriers Sell Your Location, Without Permission

Share Button

In May, the New York Times reported on a private company that purchased bulk user location data from US cellular carriers and then re-sold individual location data to law enforcement in a blatant violation of customer privacy and legal due process:

The service can find the whereabouts of almost any cellphone in the country within seconds. It does this by going through a system typically used by marketers and other companies to get location data from major cellphone carriers, including AT&T, Sprint, T-Mobile and Verizon, documents show.

US Sen. Ron Wyden (D-Ore.) took action the next day, calling on carriers to discontinue selling subscriber data to so-called “location aggregators”. So far AT&T, Verizon, Sprint, and T-Mobile have responded, issuing statements of intent to cut ties with location middlemen. Whether they will continue to share subscriber location data without explicit and affirmative consent remains to be seen. Congressional Republicans show no interest in preventing them:

“Chairman Pai’s total abandonment of his responsibility to protect Americans’ security shows that he can’t be trusted to oversee an investigation into the shady companies that he used to represent,” Wyden said. “If your location information falls into the wrong hands, you—or your children—can be vulnerable to predators, thieves, and a whole host of people who would use that knowledge to malicious ends.”

FCC Chairman Ajit Pai represented Securus in 2012. More information from ArsTechnica, who report that Obama-era regulations were blocked by Congress that would have prevented this kind of behavior.

Tapplock Is Basically Worthless

Share Button

Recently-kickstarted Tapplock touts a Bluetooth-enabled smart lock that uses a fingerprint sensor. The company came under fire from tech-savvy commentators when popular YouTuber JerryRigEverything completely disassembled and defeated in a matter of minutes using a screwdriver and adhesive pad. This attack appears to be related to a quality control problem with the specific unit he used; a spring-loaded shear pin is supposed to prevent the back from rotating. It’s unclear whether that pin can be easily snapped or retracted, for example with a string magnet, but it turns out that doesn’t matter. UK-based security researchers PenTestPartners:

The only thing we need to unlock the lock is to know the BLE MAC address. The BLE MAC address that is broadcast by the lock.

The security credentials used to control the lock are derived from the device’s publicly broadcast identifier. This means that every single lock is vulnerable to an attack that can be carried out with a smartphone app:

I scripted the attack up to scan for Tapplocks and unlock them. You can just walk up to any Tapplock and unlock it in under 2s. It requires no skill or knowledge to do this.

Can it get worse? Yes, it can. Responding to the researcher’s security disclosure, Tapplock reportedly said:

“Thanks for your note. We are well aware of these notes.”

Be wary of Internet of Things (IoT) “smart” security devices. The are neither smart nor secure.

Enable Searching of SMB Shares on Freenas under macOS

Share Button

One frustrating shortcoming of accessing SMB shares from macOS is the default failure of directory indexing for file searching. You simply can’t use the normal Finder “Search” field to do anything. This makes it particularly tedious to interact with large SMB shares when you don’t know exactly where the files you want are located.

The solution at the link is simple, if obscure: select the fruit object from the available VFS Objects under the Advanced configuration of the share in question. Thanks to Spiceworks user David_CSG for dropping a hint about vfs_fruit that led me to this solution.

Edit: turns out that this doesn’t actually work. The current state of enabling SMB server-side indexing under FreeBSD appears to involve running Gnome Tracker. These instructions apparently work under FreeBSD Jail with the addition of devel/dconf dependency. iXsystems development stance is currently “Nope”. I might take a look at this and see whether the installation can be pared down; with any luck it should be possible to exclude metadata indexing components with the largest dependency footprint.

FireWire Quibble

Share Button

I have a personal quibble: FireWire may be a dead product, but there are a lot of legacy devices out there (mostly in the audio world). The current-generation Thunderbolt–FireWire adapter is completely inadequate for these devices, for two reasons: 1) they’re an end-of-line device, meaning they don’t daisy chain, which makes them difficult to use with devices that have few TB ports and 2) they are limited by TB power delivery maximums to only 10W, which many FireWire legacy devices easily exceed when operating on bus power. As an example, I have a not-that-old FireWire audio interface that I’d like to run off bus power from my laptop, on the go. It draws 7.5W idle, but spikes over 10W during startup (charging capacitors, I’m sure). I can’t use it with the TB bus adapter, I need either DC power (dumb) or a second adapter (since like good FW devices this has two ports for daisy chaining). The DC power port went out a while back, so now I use an Original iPod FireWire charger on the second port to deliver enough power.

It would be nice if anyone offered a powered FireWire adapter that could deliver a lot of wattage for legacy devices.

InSecurity: Panera Bread Co.

Share Button

This is the first installment of a new segment titled InSecurity, covering consumer-relevant business and government security practices with an emphasis on their failures.


Each new week, it seems, brings a new corporate or government data breach or operational security failure to out awareness. This week is no exception. The failure this time, however, is particularly egregious: ever the course of eight months, Panera Bread Co. knowingly failed to protect sensitive customer data from unauthorized access. This data included, according to security researcher Dylan Houlihan who originally discovered the vulnerability, at least one web API endpoint which revealed “the full name, home address, email address, food/dietary preferences, username, phone number, birthday and last four digits of a saved credit card to be accessed in bulk for any user that had ever signed up for an account“. Equivalent vulnerabilities were eventually discovered across multiple Panera web properties.

Houlihan first reported the issue in August of 2017, reaching out to Panera’s Information Security Director, none other than Mike Gustavison. No stranger to “managing” wildly insecure corporate systems, Gustavison worked as “ISO – Sr. Director of Security Operations” at Equifax from 2009–2013. After eight months of repeated attempts to convince Panera of the severity of their security hole, Houlihan reached out to security industry expert Brain Krebs, whose influence was able to extract mainstream media coverage and a formal PR statement from Panera. Incredibly, and despite public statements to the contrary, Panera failed to secure the vulnerable API endpoints.

For a full explanation of the vulnerability and a timeline of events, reference the primary link.

On the Heritability of Intelligence

Share Button

In their 2013 article for Current Directions in Psychological Science, Tucker-Drob, Briley, and Harden propose a transactional model for the heritability of cognitive ability. The basis for this model is the well-documented biological phenomenon, the basis of phenotypic variation, the regulation of gene expression by environmental stimuli. The authors apply this finding to the characteristic of cognitive ability and support it using evidence from twin studies. They propose that, through self-selection of stimulating environments and experiences, individuals with inherited high cognitive ability cause the expression of more cognitive ability genes, which in turn feeds the selection of more stimulating environments. However, beyond this straightforward and reasonably supported model they also provide a guiding interpretation which places heavy emphasis on the significance of genetic heritability of high cognitive ability. They ignore shortcomings in the data and make claims which are both unnecessary and unsupported by the evidence. The most significant of these are presented, with rebuttal.

Having claimed that cognitive ability is undeniably a heritable trait (“no longer a question of serious scientific debate”) (Tucker-Drob, Briley, and Harden, 2013; 1), the authors propose that this trait is subject to the same regulatory forces of expression as affect everything from the production of digestive enzymes to the melanin content of the skin. Although undoubtedly a simplification, this claim seems to hold true. The evidence brought to support the heritability, on some level, of cognitive ability is reasonably robust. In fact, it follows naturally from the fundamental theoretical basis underpinning the whole diversity of individual characteristics: the regulation of gene expression. As a trait which varies between individuals, changes over the lifetime, and is subject to heritable factors, it is reasonable (if not inevitable) that cognitive ability should have some form of expression regulation mechanism in the genome. This proposed mechanism is sound and well founded. The authors then propose that heritability of cognitive ability is environmentally influenced, and further that high cognitive ability is more heritable than low cognitive ability.

However, the shortcomings of their specific model begin to show as early as the first example. The authors discuss the average educational attainment of Norwegians, which increased during the 20th century due to social and regulatory changes favouring education, in the context of measured heritability of educational attainment increasing during roughly the same period. The educational attainment data reports numbers for 1960 vs. 2000, while data on the heritability thereof reports “before 1940” and after, while itself being published in 1985. Ignoring the obvious flaw of comparing temporally un-matched samples of population data, the example provides no evidence of what the authors claim: educational attainment is not a measure of cognitive ability, but of the strength of social norms and reach of educational programs. Given Norway’s system of public education (10 years compulsory, 3 years ‘optional’ in name only, 3–8 years optional university level), the c. 2000 attainment of 11.86 years average demonstrates that 91% of individuals complete the first 13 years. Completion of compulsory public education is not a matter of individual aptitude or achievement, but of social norms and education system policy. Nine in ten completion of compulsory education is not indicative of high cognitive ability, and the data on heritability of cognitive ability are too narrow to offer a comparison.

The next section of the piece, which introduces the underlying framework of “gene-environment correlation” before explaining the authors’ proposed transactional model of cognitive ability specifically, serves as evidence against the very application which the authors make. They put it simply: “a broad array of presumably ‘environmental’ experiences—such as negative life events, relationships with parents, and experiences with peers—are themselves heritable” (Tucker-Drob, Briley, and Harden, 2013; 2). Clearly a broad range of experiences and qualities can be inherited. While the authors propose this to support the heritability of cognitive ability, it just as easily supports the heritability of academic inclination, curiosity, and even academic performance, factors which are often measured as indicators of cognitive ability. In fact, these very listed examples of heritable experiences are known to influence cognition themselves, as are other factors which fit the same category, such as parent attachment (Fryers & Brugha, 2013; 9). This gets into the base of the authors proposed model: that gene-environment correlation acts in concert with gene expression regulation to promote the achievement of cognitive ability potential in individuals with a genetic disposition for it. In particular, the authors posit that the natural outcome of this process is as observed: individuals with high cognitive ability are likely to have inherited it. In fact their evidence shows that, rather than cognitive ability itself being heritable, the indicators often measured for cognitive ability are heritable as are many influences on the development of cognitive ability.

Further complicating the authors’ model is their poor handling of shortcomings in their base evidence, twin studies. Due to restrictions on data retention and access, international adoption, research funding, and scope, the majority of twin studies showing heritability of cognitive ability sample from within the same country, with the same social norms, educational policy, and even similar socioeconomic context. The authors, however, insist that socioeconomic status is a predictor of both the heritability and level of cognitive ability. This is faulty: if socioeconomic status predicts cognitive ability and socioeconomic status is itself significantly heritable, then it will appear that socioeconomic status influences heritability of cognitive ability. This is demonstrated in the cited data, which break down outside of the US. In particular in social democracies such as Sweden (and Norway) where, the authors admit, strong social programs even the playing field and give all individuals of diverse backgrounds access to the same pool of potential environmental stimuli to select. In these states, it seems that the socioeconomic factors typically having a deleterious effect on  educational access and achievement (and thus, cognitive ability) are substantially reduced. As a result, the heritability of cognitive ability is low. This indicates that the authors’ model of positive experience selection for cognitive ability is fundamentally flawed: a socioeconomically- and gene-environment-linked deficit model better explains inheritance of cognitive ability. Instead of cognitive ability itself being inherited, it is a positive environment, with access to stimulating and diverse education and other experiences (among many factors), which is inherited. It is the absence of this environment and various opportunities of experience which reduces an individual’s achieved cognitive ability. This is clearly shown in the countries where their model does not hold, which have strong social programs.

The trouble with the authors’ model is not that it is based on a flawed mechanism, but that its predictions do not hold. There is no reason that high heritability of cognitive ability should correlate with high cognitive ability itself, absent the other factors. Their model provides no explanation for this claim, yet it is a clear component of their position. The authors themselves have set up evidence and arguments which can be used to make a completely different point: cognitive ability shows high heritability because environmental factors that influence it—such as social norms around education, parenting style, household stability, and academic aptitude and drive—are themselves heritable.

Although the authors identify and describe a compelling mechanism for the influence of environment on cognitive ability, they seem to mistake the causal direction of its operation in their interpretation. Rather than an abundance of stimulating experiences and environments improving an individual’s actualisation of their inherited potential, as is implied, a more parsimonious explanation is that a shortage of these influences suppresses achievement of genetic potential. Their base assumption is made clear on p.3: “The ‘end state’ of this transactional process—high levels of and high heritability of cognitive ability—is therefore expected to differ depending on the quality and availability of environmental experiences.” Emphasis here is on the the end state, particularly the high levels of cognitive ability. In other words, those who demonstrate the higher heritability of cognitive ability also demonstrate higher levels of that ability. High cognitive ability is more closely linked to parent performance than low cognitive ability. If this end state were high heritability of cognitive ability alone, then their argument would hold. However, the association of “high levels of” cognitive ability in this outcome does not follow from their model, and indicates that the observations could be better explained as a default outcome; Ockham’s Razor is well applied. This critical flaw in their proposal is laid bare in their only addressed counter-argument, that their model breaks down in countries with stronger education systems and social programs. Rather than individuals’ ability to self-select unique experiences reinforcing a pattern of gene expression and experience selection to cyclically maximise expression of inherited cognitive ability, it is a much simpler explanation that instead a paucity of these experiences suppresses the expression of inherited cognitive ability.

References

Fryers, T., & Brugha, T. (2013). Childhood Determinants of Adult Psychiatric Disorder. Clinical Practice and Epidemiology in Mental Health : CP & EMH, 9, 1–50. http://doi.org/10.2174/1745017901309010001

Tucker-Drob, E. M., Briley, D. A., & Harden, K. P. (2013). Genetic and Environmental Influences on Cognition Across Development and Context. Current Directions in Psychological Science, 22(5), 349–355. http://doi.org/10.1177/0963721413485087