---
name: kvm-zvol-restore
description: Restore KVM/libvirt VM disks from borg backups onto ZFS zvols exported over iSCSI. Use when asked to restore, roll back, or recover a VM's drive to a previous date, or to inspect what VM disk backups are available. Covers the tama/kiki zvol+iSCSI setup backed up by kvm-backup-zvol-iscsi.sh.
---

# Restoring KVM VM disks from borg to ZFS zvols

## Topology

Two hosts, and it matters which one you run each command on:

| Host | Name | Role |
|---|---|---|
| `10.0.0.5` | `kiki` | libvirt host (runs the VMs) **and** holds the borg repo at `/media/Storage/Backup/kvm` |
| `10.0.0.6` | `tama` | ZFS server: owns the zvols (`tank/kvm/*`), exports them via iSCSI (LIO/targetcli), runs the backup script |

The backup script is `/usr/local/bin/kvm-backup-zvol-iscsi.sh` **on tama (.6)**, not on kiki.
It discovers domains via `virsh` locally and on `REMOTE_HOSTS` (`kiki`, `gaming-pc`), maps each
iSCSI-backed disk to a local zvol, snapshots it, and `dd`s the snapshot device into borg.

VMs reach their disks over the **storage network** `10.0.100.6:3260`, not `10.0.0.6`.

## What the backups actually are

Raw whole-device images (`dd` of a ZFS snapshot block device) piped into borg as **stdin**, so each
archive holds exactly one item named `stdin`. There is no filesystem structure inside — restoring
means writing the stream back over the whole block device.

Archive naming:

```
<domain>-<target-dev>-<YYYY-MM-DDTHH:MM:SS>     disk image, e.g. centos7-vda-2026-08-14T07:38:41
<domain>-xml-<YYYY-MM-DDTHH:MM:SS>              virsh dumpxml output
```

Retention is `--keep-daily 7 --keep-weekly 4 --keep-monthly 6`, so anything older than a week
only exists at weekly/monthly granularity. Check before promising a specific date.

## Gotchas that have actually bitten

1. **Archive names contain colons.** `centos7-vda-2026-08-14T07:38:41`. Never use `:` as a field
   separator when building job lists in a shell script — it silently truncates the name to
   `centos7-vda-2026-08-14T07` and borg reports "Archive does not exist". Use `|`.
2. **`borg list --format "{archive}"` fails on an archive.** `{archive}` is a repo-level key; using
   it while listing archive *contents* errors out and looks exactly like a missing archive.
   To test existence use `borg info "::$ARCHIVE"`.
3. **The device name is not always `vda`.** `centos6` is `sdb`; `centos7`/`rocky9` are `vda`.
   Always read the device from the archive name or `virsh domblklist`.
4. **`virsh domblklist` shows CDROMs with source `-`.** Skip those; only `-iscsi-` paths are backed up.
5. **Domain name vs zvol name is case-insensitive.** IQN target names are lowercased
   (`iqn...tama:centos7-2`) while the zvol may be `tank/kvm/CentOS7-2`. There are also stale archives
   from an older capitalized `Centos7` domain — don't confuse them with current `centos7`.
6. **CentOS 6 ignores `virsh shutdown`** (no/old acpid). It may need several minutes or a
   `virsh destroy`. Never start writing until `virsh domstate` says `shut off`.
7. **Don't run during the backup window** (~07:24–07:55 daily). The backup holds
   `/tmp/backup-zvol-iscsi.pid` on tama and would be snapshotting the same zvols.

## Procedure

### 1. Shut down the VMs (on kiki, .5)

```bash
ssh root@10.0.0.5 'for d in vm1 vm2; do virsh shutdown "$d"; done'
# then poll until every one reports "shut off" — do not proceed otherwise
ssh root@10.0.0.5 'for i in $(seq 1 30); do s=$(virsh domstate vm1); [ "$s" = "shut off" ] && break; sleep 5; done; virsh domstate vm1'
```

If a guest won't go down gracefully, confirm with the user before `virsh destroy`.

### 2. Find the archives (from tama, .6)

```bash
ssh root@10.0.0.6 'export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes
  borg list --format "{archive}{TAB}{time}{NL}" root@10.0.0.5:/media/Storage/Backup/kvm' \
  | grep -Ei "^(vm1|vm2)-"
```

Then record each archive's exact byte size — it must equal the zvol's size:

```bash
borg info "::$ARCHIVE"          # "Original size" column, or borg list shows the stdin item size
blockdev --getsize64 /dev/zvol/tank/kvm/<zvol>
```

### 3. Map disks to zvols

```bash
ssh root@10.0.0.5 'virsh domblklist <domain>'      # dev -> /dev/disk/by-path/...-iscsi-iqn...:<target>-lun-0
ssh root@10.0.0.6 'zfs list -o name,volsize,volmode -r tank/kvm -d 1'
```
The zvol is the IQN target name after the last colon, matched case-insensitively under `tank/kvm/`.

### 4. Optional rollback point

A pre-restore `zfs snapshot tank/kvm/<z>@pre-restore-<date>` is cheap insurance, **but ask first** —
it is not always wanted, and the daily backup snapshots (`SNAPSHOTS_KEEP=2`) may already cover it.

### 5. Restore (on tama, .6)

Run detached — 500 GiB takes ~20 min at ~475 MB/s, and an SSH drop must not kill it.
Guard every device write behind a size check and an existence check.

```bash
#!/bin/bash
export BORG_REPO="root@10.0.0.5:/media/Storage/Backup/kvm"
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes
export BORG_RELOCATED_REPO_ACCESS_IS_OK=yes
set -o pipefail

# zvol|archive|expected_bytes   (pipe-delimited: archive names contain colons)
JOBS=(
  "centos7|centos7-vda-2026-08-14T07:38:41|536870912000"
)

for J in "${JOBS[@]}"; do
    IFS="|" read -r ZVOL ARCHIVE EXPECT <<<"$J"
    DEV="/dev/zvol/tank/kvm/$ZVOL"
    echo "=== $(date "+%F %T") restoring $ARCHIVE -> $DEV ($EXPECT bytes)"

    [[ -b "$DEV" ]] || { echo "FAIL: $DEV not a block device"; exit 1; }

    ACTUAL=$(blockdev --getsize64 "$DEV")
    [[ "$ACTUAL" == "$EXPECT" ]] || { echo "FAIL: $DEV is $ACTUAL, archive is $EXPECT"; exit 1; }

    borg info "::$ARCHIVE" >/dev/null 2>&1 || { echo "FAIL: archive $ARCHIVE not found"; exit 1; }

    borg extract --stdout "::$ARCHIVE" \
        | dd of="$DEV" bs=4M iflag=fullblock conv=fsync status=progress \
        || { echo "FAIL: restore of $ARCHIVE failed"; exit 1; }

    echo "=== $(date "+%F %T") completed $ZVOL"
done
echo "=== $(date "+%F %T") ALL RESTORES COMPLETE"
```

Launch and watch:

```bash
ssh root@10.0.0.6 'nohup setsid /root/restore.sh > /root/restore.log 2>&1 </dev/null & echo $!'
ssh root@10.0.0.6 'grep -aE "^===|FAIL:" /root/restore.log; tail -c 120 /root/restore.log'
```

`iflag=fullblock` matters: without it `dd` can do short reads from the pipe and write a short image.
Writing to `/dev/zvol/...` while LIO exports it is safe **only** because the guest is powered off —
LIO takes no exclusive lock, so a running VM would race and corrupt.

### 6. Flush stale initiator cache (on kiki, .5)

The zvol changed underneath the iSCSI initiator, so kiki's block-layer cache for that LUN is stale.
Before booting, flush it:

```bash
ssh root@10.0.0.5 'blockdev --flushbufs /dev/disk/by-path/ip-10.0.100.6:3260-iscsi-iqn.2026-03.im.gec.tama:<target>-lun-0'
```

A `iscsiadm -m node -T <iqn> --rescan`, or logout/login of the session, is the heavier alternative.

### 7. Boot and verify

```bash
ssh root@10.0.0.5 'virsh start <domain>; virsh domstate <domain>'
```

Restoring the domain XML is a **separate** decision — the `-xml-` archives exist, but rolling back a
disk rarely needs the definition rolled back too, and doing so can undo unrelated config. Ask first:

```bash
borg extract --stdout "::<domain>-xml-<ts>" > /tmp/<domain>.xml   # then virsh define
```