Turning a root on ZFS to a root on ZFS-mirror

Right, so from time to time I end up installing a FreeBSD system while there's only one disk for the os, sometimes by accident and sometimes on purpose.

Sometimes, it's just because the old mirror has a really bad drive it in and I want to freshen it up a bit while replacing the drive.

So I found this FreeBSD guide on Convert Single disk ZFS-On-Root to Mirror on the FreeBSD forum.

I followed it more or less to the letter and it worked like a charm.

First checking the status of the pool I want to modify.

box01# zpool status -v
  pool: zroot
 state: ONLINE
  scan: resilvered 11.3G in 0h16m with 0 errors on Sat Dec 23 04:59:14 2017
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0
            ada2p3  ONLINE       0     0     0

errors: No known data errors

The ada2 drive is showing some horrible read/write lag that drags the whole system down, so I attached a new drive between the two so it should be ada1.

box01# camcontrol devlist
<WDC WD20EARS-00S8B1 80.00A80>     at scbus0 target 0 lun 0 (ada0,pass0)
<WDC WD20EARS-00J2GB0 80.00A80>    at scbus1 target 0 lun 0 (ada1,pass1)
<WDC WD30EZRX-00MMMB0 80.00A80>    at scbus2 target 0 lun 0 (ada2,pass2)
<AHCI SGPIO Enclosure 1.00 0001>   at scbus6 target 0 lun 0 (ses0,pass3)

Wonder how long it'll be before 2T drives seems so small we wonder how we'd ever fit an OS on it, it definitely happened to the 40MB drive I had close to 20 years ago.

So next we have to do some gpart gymnastics to prepare ada1 for adding to the pool.

box01# gpart show ada1
=>        34  3907029101  ada1  GPT  (1.8T)
          34         222        - free -  (111K)
         256  3907012495     1  !6a898cc3-1dd2-11b2-99a6-080020736631  (1.8T)
  3907012751       16384     9  !6a945a3b-1dd2-11b2-99a6-080020736631  (8.0M)

box01# gpart destroy ada1
gpart: Device busy
box01# gpart destroy -F ada1
ada1 destroyed
box01# gpart create -s GPT ada1
ada1 created
box01# gpart show ada0
=>        40  3907029088  ada0  GPT  (1.8T)
          40        1024     1  freebsd-boot  (512K)
        1064         984        - free -  (492K)
        2048    16777216     2  freebsd-swap  (8.0G)
    16779264  3890249728     3  freebsd-zfs  (1.8T)
  3907028992         136        - free -  (68K)

box01# gpart add -t freebsd-boot -l boot2 -s 512K ada1
ada1p1 added
box01# gpart add -t freebsd-swap -l swap2 -s 8G ada1
ada1p2 added
box01# gpart add -t freebsd-zfs -l zfs2 ada1
ada1p3 added

Around here the guide tells me to run zdb to find the guid, I did run the zdb command and find the guid, but I didn't want to use it, so now is the time to check that it's been partitioned similar to the old disk and recheck the pool.

box01# gpart show ada0
=>        40  3907029088  ada0  GPT  (1.8T)
          40        1024     1  freebsd-boot  (512K)
        1064         984        - free -  (492K)
        2048    16777216     2  freebsd-swap  (8.0G)
    16779264  3890249728     3  freebsd-zfs  (1.8T)
  3907028992         136        - free -  (68K)

box01# gpart show ada1
=>        40  3907029088  ada1  GPT  (1.8T)
          40        1024     1  freebsd-boot  (512K)
        1064    16777216     2  freebsd-swap  (8.0G)
    16778280  3890250840     3  freebsd-zfs  (1.8T)
  3907029120           8        - free -  (4.0K)

box01# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zroot  1.80T  13.9G  1.78T         -     0%     0%  1.00x  ONLINE  -
box01# zpool status
  pool: zroot
 state: ONLINE
  scan: resilvered 11.3G in 0h16m with 0 errors on Sat Dec 23 04:59:14 2017
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0
            ada2p3  ONLINE       0     0     0

errors: No known data errors

Looks good and then we attach the disk.

box01# zpool attach zroot ada0p3 ada1p3
Make sure to wait until resilver is done before rebooting.

If you boot from pool 'zroot', you may need to update
boot code on newly attached disk 'ada1p3'.

Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:

        gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

box01# zpool status -v
  pool: zroot
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Dec 23 23:25:58 2017
        42.3M scanned out of 13.9G at 578K/s, 6h58m to go
        42.3M resilvered, 0.30% done
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0
            ada2p3  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0  (resilvering)

errors: No known data errors

.. 678K/s .. 6h58m to go, I nearly went to bed when I saw that, then I realized that ada2p3 was misbehaving badly.

box01# zpool detach zroot ada2p3
box01# zpool status -v
  pool: zroot
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Dec 23 23:25:58 2017
        640M scanned out of 13.9G at 4.85M/s, 0h46m to go
        640M resilvered, 4.50% done
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0  (resilvering)

errors: No known data errors

And it gained some speed after that.

root@box01:~ # zpool status zroot 
  pool: zroot
 state: ONLINE
  scan: resilvered 13.9G in 0h10m with 0 errors on Sat Dec 23 23:36:27 2017
config:

10 minutes to rebuild the root system, that's just fine for my needs.

Updating the bootcode is fairly straightforward, and there's a fair amount of manpage material. I went for solution that's mentioned on both [gptzfsboot(8)][man-8-gptzfsboot] and on gpart(8) that has a "protective MBR"

root@box01# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

And then it's time to look at swap.

The system had so far been idle and nothing much was started so I new there was nothing swapped out and a gmirror list | head -2 showed me the name of the old swap mirror was mirror/swap nice and predictable which is good!

So a few "quick" select commands to destroy the old and create a new just like it.

box01# swapinfo    
Device          1K-blocks     Used    Avail Capacity
/dev/mirror/swap.eli   8388604        0  8388604     0%
box01# swapoff /dev/mirror/swap.eli 
box01# gmirror status
       Name    Status  Components
mirror/swap  COMPLETE  ada0p2 (ACTIVE)
                       ada2p2 (ACTIVE)
box01# gmirror destroy swap       
box01# gmirror label -b prefer -F swap /dev/gpt/swap0 /dev/gpt/swap2

Editing the swapline in fstab to match and.

box01# swapon -a
swapon: adding /dev/mirror/swap as swap device
box01# swapinfo 
Device          1K-blocks     Used    Avail Capacity
/dev/mirror/swap.eli   8388604        0  8388604     0%

You might be wondering about the .eli at the end, that's a suffix you can put on the swap entry in fstab(5) to have it use geli(8) that makes it AES encrypted.

All in all a very smooth experience...

I still haven't had the courage to try and boot on the new drive though ;-)

Comments