Disk Replacement

Locate the failed drive.

zpool status

It will show something like

	NAME                                        STATE     READ WRITE CKSUM
	pool01                                      DEGRADED     0     0     0
	  raidz3-0                                  ONLINE       0     0     0
	    44fca0d1-f343-48e6-9a43-c71463551aa4    ONLINE       0     0     0
	    7ca5e989-51a5-4f1b-a81e-982d9a05ac04    ONLINE       0     0     0
	    8fd249a0-c8c6-47bb-8787-3e246300c62d    ONLINE       0     0     0
	    573c1117-27d4-430c-b57c-858a75b4ca35    ONLINE       0     0     0
	    29b7c608-72ae-4ec2-830b-0e23925ac0b1    ONLINE       0     0     0
	    293acdbe-6be5-4fa7-945a-e9481b09c0fa    ONLINE       0     0     0
	    437bac45-433b-48e3-bc70-ae1c82e8155b    ONLINE       0     0     0
	    a5ca09a7-3f3f-4135-a2d9-71290fd79160    ONLINE       3     2     0
	  raidz3-1                                  DEGRADED     0     0     0
	    spare-0                                 DEGRADED     0     0     0
	      65f61699-e2fc-4a36-86dd-b0fa6a774798  FAULTED     53     0     0  too many errors
	      9d794dfd-2ef6-432d-8252-0c93e79509dc  ONLINE       0     0     0
	    e27f31e8-a1a4-47dc-ac01-4a6c99b6e5d0    ONLINE       0     0     0
	    aff60721-21ae-42bf-b077-1937aeafaab2    ONLINE       0     0     0
	    714da3e5-ca9c-43d0-a0f3-c0fa693a5b02    ONLINE       0     0     0
	    df89869a-4445-47f9-afa9-3b9cce3b1530    ONLINE       0     0     0
	    29748037-bbd5-4f2d-8878-4fa2b81d9ec3    ONLINE       0     0     0
	    1ff396ec-dec7-45dd-9172-de31e5f6fca7    ONLINE       0     0     0

Off-line the drive.

zpool offline pool01 65f61699-e2fc-4a36-86dd-b0fa6a77479

Get the serial number

hdparm -I /dev/disk/by-partuuid/65f61699-e2fc-4a36-86dd-b0fa6a774798 | grep Serial

The output will be something like

Serial Number:      ZC1168HE
Transport:          Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0

Identify the bay location

sas3ircu 0 display | grep -B 10 ZC1168HE                                          

The output will look like

  Device is a Hard disk
    Enclosure #                             : 2
    Slot #                                  : 17

Turn on the bay indicator

sas3ircu 0 locate 2:17 ON

Physically replace the disk

Check the logs for the new disk’s name

dmesg

The output will indicate the device id, such as ‘sdal’ in the below example

  [16325935.447081] sd 0:0:45:0: Power-on or device reset occurred
  [16325935.447962] sd 0:0:45:0: Attached scsi generic sg20 type 0
  [16325935.451271]  end_device-0:0:28: add: handle(0x001c), sas_addr(0x500304801810f321)
  [16325935.454768] sd 0:0:45:0: [sdal] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
  [16325935.477576] sd 0:0:45:0: [sdal] Write Protect is off
  [16325935.479913] sd 0:0:45:0: [sdal] Mode Sense: 9b 00 10 08
  [16325935.482100] sd 0:0:45:0: [sdal] Write cache: enabled, read cache: enabled, supports DPO and FUA
  [16325935.664995] sd 0:0:45:0: [sdal] Attached SCSI disk

Turn off the slot light

sas3ircu 0 locate 2:17 OFF

Use the GUI to replace the disk. (Use the GUI over the cmd lie to ensure it’s setup consistently with the other disks)

  Storage --> Pool Gear Icon (at right) --> Status

    (The removed disk  should be listed bu it's UUID)

  Disk Menu (three dots) --> Replace --> (disk from dmesg above) --> Force --> Replace Disk

After resilvering has finished, check the spare’s ID at the bottom and then detach it so it goes back to spare

zpool detach pool01 9d794dfd-2ef6-432d-8252-0c93e79509dc

Notes:

Note: The GUI takes several steps to prepare the disk and adds a partition to the pool, not the whole disk. It’s ‘strongly advised against’ using the CLI to replace the disk. Though if you must, you can recreate that process at the command line. as adapted from https://www.truenas.com/community/resources/creating-a-degraded-pool.100/

gpart and glable are not present on TrueNAS Scale, so you would have to adapt this to another tool

gpart create -s gpt /dev/da18
gpart add -i 1 -b 128 -t freebsd-swap -s 2g /dev/da18
gpart add -i 2 -t freebsd-zfs /dev/da18

zpool replace pool01 65f61699-e2fc-4a36-86dd-b0fa6a77479

To turn off all slot lights

for X in {0..23};do echo sas3ircu 0 locate 2:$X OFF;done
for X in {0..11};do sas3ircu 0 locate 3:$X OFF;done

Last modified November 8, 2024: Restructure (37c5bc6)