Large Pools

You’ve guarded against disk failure this by adding redundancy, but was it enough? There’s a very mathy calculator at https://jro.io/r2c2/ that will allow you chart different parity configs. But a reasonable rule-of-thumb is to devote 20%, or 1 in 5 drives, to parity.

• RAIDZ1 - up to 5 Drives
• RAIDZ2 - up to 10 Drives
• RAIDZ3 - up to 15 Drives

Oracle however, recommends Virtual Devices when you go past 9 disks.

Pools and Virtual Devices

When you get past 15 drives, you can’t increase parity. You can however, create virtual devices. Best practice from Oracle says to do this even earlier as a VDev should be less than 9 disks ¹. So given 24 disks, you should have 3 VDevs of 8 each. Here’s an example with 2 parity. Slightly better than 1 in 5 and suitable for older disks.

### Build a 3-Wide RAIDZ2  across 24 disks
zpool create \
  pool01 \
  -m none \
  -f \
  raidz2 sdb sdc sdd sde sdf sdg sdh sdi \
  raidz2 sdj sdk sdl sdm sdn sdo sdp sdq \
  raidz2 sdr sds sdt sdu sdv sdw sdx sdy 

Using Disk IDs

Drive letters can be hard to trace back to a physical drive. A a better² way is the /dev/disk/by-id identifiers.

ls /dev/disk/by-id | grep ata | grep -v part

zpool create -m none -o ashift=12 -O compression=lz4 \
  pool04 \
    raidz2 \
      ata-ST4000NM0035-1V4107_ZC11AHH9 \
      ata-ST4000NM0035-1V4107_ZC116F11 \
      ata-ST4000NM0035-1V4107_ZC1195V5 \
      ata-ST4000NM0035-1V4107_ZC11CDMB \
      ata-ST4000NM0035-1V4107_ZC1195PR \
      ata-ST4000NM0024-1HT178_Z4F164WG \
      ata-ST4000NM0024-1HT178_Z4F17SJK \
      ata-ST4000NM0024-1HT178_Z4F17M6B \
      ata-ST4000NM0024-1HT178_Z4F18FZE \
      ata-ST4000NM0024-1HT178_Z4F18G35 

Hot and Distributed Spares

Spares vs Parity

You may not be reach a location quickly when a disk fails. In such a case, is it better to have a Z3 filesystem run in degraded performance mode (i.e. calculating parity the whole time) or a Z2 system that replaces the failed disk automatically?

It’s better to have a more parity until you go past the guidelines of 15 drives in a Z3 config. If you have 16 bays, add a hot spare.

Distributed vs Dedicated

A distributed spare is a newer feature that allows you reserve space on all of your disks, rather than just one. That allows resilvering to go much faster as you’re no longer limited by the speed of one disk. Here’s an example of such a pool that has 16 total devices.

# This pool has 3 parity, 12 data, 16 total count, with 1 spare
```bash
zpool create -f pool02 \
  draid3:12d:16c:1s \
    ata-ST4000NM000A-2HZ100_WJG04M27 \
    ata-ST4000NM000A-2HZ100_WJG09BH7 \
    ata-ST4000NM000A-2HZ100_WJG0QJ7X \
    ata-ST4000NM000A-2HZ100_WS20ECCD \
    ata-ST4000NM000A-2HZ100_WS20ECFH \
    ata-ST4000NM000A-2HZ100_WS20JXTA \
    ata-ST4000NM0024-1HT178_Z4F14K76 \
    ata-ST4000NM0024-1HT178_Z4F17SJK \
    ata-ST4000NM0024-1HT178_Z4F17YBP \
    ata-ST4000NM0024-1HT178_Z4F1BJR1 \
    ata-ST4000NM002A-2HZ101_WJG0GBXB \
    ata-ST4000NM002A-2HZ101_WJG11NGC \
    ata-ST4000NM0035-1V4107_ZC1168N3 \
    ata-ST4000NM0035-1V4107_ZC116F11 \
    ata-ST4000NM0035-1V4107_ZC116MSW \
    ata-ST4000NM0035-1V4107_ZC116NZM \

References

• https://oshogbo.vexillium.org/blog/66/https://oshogbo.vexillium.org/blog/66/
• https://docs.oracle.com/cd/E19253-01/819-5461/gaypm/index.html
• https://docs.oracle.com/cd/E19253-01/819-5461/gbchx/index.html
• https://docs.oracle.com/cd/E26505_01/html/E37384/zfspools-4.html#gentextid-11655