skills/kvm-zvol-restore/SKILL.md

name	description
kvm-zvol-restore	Restore KVM/libvirt VM disks from borg backups onto ZFS zvols exported over iSCSI. Use when asked to restore, roll back, or recover a VM's drive to a previous date, or to inspect what VM disk backups are available. Covers the tama/kiki zvol+iSCSI setup backed up by kvm-backup-zvol-iscsi.sh.

Restoring KVM VM disks from borg to ZFS zvols

Topology

Two hosts, and it matters which one you run each command on:

Host	Name	Role
10.0.0.5	kiki	libvirt host (runs the VMs) and holds the borg repo at /media/Storage/Backup/kvm
10.0.0.6	tama	ZFS server: owns the zvols (tank/kvm/*), exports them via iSCSI (LIO/targetcli), runs the backup script

The backup script is /usr/local/bin/kvm-backup-zvol-iscsi.sh on tama (.6), not on kiki. It discovers domains via virsh locally and on REMOTE_HOSTS (kiki, gaming-pc), maps each iSCSI-backed disk to a local zvol, snapshots it, and dds the snapshot device into borg.

VMs reach their disks over the storage network 10.0.100.6:3260, not 10.0.0.6.

What the backups actually are

Raw whole-device images (dd of a ZFS snapshot block device) piped into borg as stdin, so each archive holds exactly one item named stdin. There is no filesystem structure inside — restoring means writing the stream back over the whole block device.

Archive naming:

<domain>-<target-dev>-<YYYY-MM-DDTHH:MM:SS>     disk image, e.g. centos7-vda-2026-08-14T07:38:41
<domain>-xml-<YYYY-MM-DDTHH:MM:SS>              virsh dumpxml output

Retention is --keep-daily 7 --keep-weekly 4 --keep-monthly 6, so anything older than a week only exists at weekly/monthly granularity. Check before promising a specific date.

Gotchas that have actually bitten

1. Archive names contain colons. centos7-vda-2026-08-14T07:38:41. Never use : as a field separator when building job lists in a shell script — it silently truncates the name to centos7-vda-2026-08-14T07 and borg reports "Archive does not exist". Use |.
2. borg list --format "{archive}" fails on an archive. {archive} is a repo-level key; using it while listing archive contents errors out and looks exactly like a missing archive. To test existence use borg info "::$ARCHIVE".
3. The device name is not always vda. centos6 is sdb; centos7/rocky9 are vda. Always read the device from the archive name or virsh domblklist.
4. virsh domblklist shows CDROMs with source -. Skip those; only -iscsi- paths are backed up.
5. Domain name vs zvol name is case-insensitive. IQN target names are lowercased (iqn...tama:centos7-2) while the zvol may be tank/kvm/CentOS7-2. There are also stale archives from an older capitalized Centos7 domain — don't confuse them with current centos7.
6. CentOS 6 ignores virsh shutdown (no/old acpid). It may need several minutes or a virsh destroy. Never start writing until virsh domstate says shut off.
7. Don't run during the backup window (~07:24–07:55 daily). The backup holds /tmp/backup-zvol-iscsi.pid on tama and would be snapshotting the same zvols.

Procedure

1. Shut down the VMs (on kiki, .5)

ssh root@10.0.0.5 'for d in vm1 vm2; do virsh shutdown "$d"; done'
# then poll until every one reports "shut off" — do not proceed otherwise
ssh root@10.0.0.5 'for i in $(seq 1 30); do s=$(virsh domstate vm1); [ "$s" = "shut off" ] && break; sleep 5; done; virsh domstate vm1'

If a guest won't go down gracefully, confirm with the user before virsh destroy.

2. Find the archives (from tama, .6)

ssh root@10.0.0.6 'export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes
  borg list --format "{archive}{TAB}{time}{NL}" root@10.0.0.5:/media/Storage/Backup/kvm' \
  | grep -Ei "^(vm1|vm2)-"

Then record each archive's exact byte size — it must equal the zvol's size:

borg info "::$ARCHIVE"          # "Original size" column, or borg list shows the stdin item size
blockdev --getsize64 /dev/zvol/tank/kvm/<zvol>

3. Map disks to zvols

ssh root@10.0.0.5 'virsh domblklist <domain>'      # dev -> /dev/disk/by-path/...-iscsi-iqn...:<target>-lun-0
ssh root@10.0.0.6 'zfs list -o name,volsize,volmode -r tank/kvm -d 1'

The zvol is the IQN target name after the last colon, matched case-insensitively under tank/kvm/.

4. Optional rollback point

A pre-restore zfs snapshot tank/kvm/<z>@pre-restore-<date> is cheap insurance, but ask first — it is not always wanted, and the daily backup snapshots (SNAPSHOTS_KEEP=2) may already cover it.

5. Restore (on tama, .6)

Run detached — 500 GiB takes ~20 min at ~475 MB/s, and an SSH drop must not kill it. Guard every device write behind a size check and an existence check.

#!/bin/bash
export BORG_REPO="root@10.0.0.5:/media/Storage/Backup/kvm"
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes
export BORG_RELOCATED_REPO_ACCESS_IS_OK=yes
set -o pipefail

# zvol|archive|expected_bytes   (pipe-delimited: archive names contain colons)
JOBS=(
  "centos7|centos7-vda-2026-08-14T07:38:41|536870912000"
)

for J in "${JOBS[@]}"; do
    IFS="|" read -r ZVOL ARCHIVE EXPECT <<<"$J"
    DEV="/dev/zvol/tank/kvm/$ZVOL"
    echo "=== $(date "+%F %T") restoring $ARCHIVE -> $DEV ($EXPECT bytes)"

    [[ -b "$DEV" ]] || { echo "FAIL: $DEV not a block device"; exit 1; }

    ACTUAL=$(blockdev --getsize64 "$DEV")
    [[ "$ACTUAL" == "$EXPECT" ]] || { echo "FAIL: $DEV is $ACTUAL, archive is $EXPECT"; exit 1; }

    borg info "::$ARCHIVE" >/dev/null 2>&1 || { echo "FAIL: archive $ARCHIVE not found"; exit 1; }

    borg extract --stdout "::$ARCHIVE" \
        | dd of="$DEV" bs=4M iflag=fullblock conv=fsync status=progress \
        || { echo "FAIL: restore of $ARCHIVE failed"; exit 1; }

    echo "=== $(date "+%F %T") completed $ZVOL"
done
echo "=== $(date "+%F %T") ALL RESTORES COMPLETE"

Launch and watch:

ssh root@10.0.0.6 'nohup setsid /root/restore.sh > /root/restore.log 2>&1 </dev/null & echo $!'
ssh root@10.0.0.6 'grep -aE "^===|FAIL:" /root/restore.log; tail -c 120 /root/restore.log'

iflag=fullblock matters: without it dd can do short reads from the pipe and write a short image. Writing to /dev/zvol/... while LIO exports it is safe only because the guest is powered off — LIO takes no exclusive lock, so a running VM would race and corrupt.

6. Flush stale initiator cache (on kiki, .5)

The zvol changed underneath the iSCSI initiator, so kiki's block-layer cache for that LUN is stale. Before booting, flush it:

ssh root@10.0.0.5 'blockdev --flushbufs /dev/disk/by-path/ip-10.0.100.6:3260-iscsi-iqn.2026-03.im.gec.tama:<target>-lun-0'

A iscsiadm -m node -T <iqn> --rescan, or logout/login of the session, is the heavier alternative.

7. Boot and verify

ssh root@10.0.0.5 'virsh start <domain>; virsh domstate <domain>'

Restoring the domain XML is a separate decision — the -xml- archives exist, but rolling back a disk rarely needs the definition rolled back too, and doing so can undo unrelated config. Ask first:

borg extract --stdout "::<domain>-xml-<ts>" > /tmp/<domain>.xml   # then virsh define