169 lines
7.1 KiB
Markdown
169 lines
7.1 KiB
Markdown
---
|
||
name: kvm-zvol-restore
|
||
description: Restore KVM/libvirt VM disks from borg backups onto ZFS zvols exported over iSCSI. Use when asked to restore, roll back, or recover a VM's drive to a previous date, or to inspect what VM disk backups are available. Covers the tama/kiki zvol+iSCSI setup backed up by kvm-backup-zvol-iscsi.sh.
|
||
---
|
||
|
||
# Restoring KVM VM disks from borg to ZFS zvols
|
||
|
||
## Topology
|
||
|
||
Two hosts, and it matters which one you run each command on:
|
||
|
||
| Host | Name | Role |
|
||
|---|---|---|
|
||
| `10.0.0.5` | `kiki` | libvirt host (runs the VMs) **and** holds the borg repo at `/media/Storage/Backup/kvm` |
|
||
| `10.0.0.6` | `tama` | ZFS server: owns the zvols (`tank/kvm/*`), exports them via iSCSI (LIO/targetcli), runs the backup script |
|
||
|
||
The backup script is `/usr/local/bin/kvm-backup-zvol-iscsi.sh` **on tama (.6)**, not on kiki.
|
||
It discovers domains via `virsh` locally and on `REMOTE_HOSTS` (`kiki`, `gaming-pc`), maps each
|
||
iSCSI-backed disk to a local zvol, snapshots it, and `dd`s the snapshot device into borg.
|
||
|
||
VMs reach their disks over the **storage network** `10.0.100.6:3260`, not `10.0.0.6`.
|
||
|
||
## What the backups actually are
|
||
|
||
Raw whole-device images (`dd` of a ZFS snapshot block device) piped into borg as **stdin**, so each
|
||
archive holds exactly one item named `stdin`. There is no filesystem structure inside — restoring
|
||
means writing the stream back over the whole block device.
|
||
|
||
Archive naming:
|
||
|
||
```
|
||
<domain>-<target-dev>-<YYYY-MM-DDTHH:MM:SS> disk image, e.g. centos7-vda-2026-08-14T07:38:41
|
||
<domain>-xml-<YYYY-MM-DDTHH:MM:SS> virsh dumpxml output
|
||
```
|
||
|
||
Retention is `--keep-daily 7 --keep-weekly 4 --keep-monthly 6`, so anything older than a week
|
||
only exists at weekly/monthly granularity. Check before promising a specific date.
|
||
|
||
## Gotchas that have actually bitten
|
||
|
||
1. **Archive names contain colons.** `centos7-vda-2026-08-14T07:38:41`. Never use `:` as a field
|
||
separator when building job lists in a shell script — it silently truncates the name to
|
||
`centos7-vda-2026-08-14T07` and borg reports "Archive does not exist". Use `|`.
|
||
2. **`borg list --format "{archive}"` fails on an archive.** `{archive}` is a repo-level key; using
|
||
it while listing archive *contents* errors out and looks exactly like a missing archive.
|
||
To test existence use `borg info "::$ARCHIVE"`.
|
||
3. **The device name is not always `vda`.** `centos6` is `sdb`; `centos7`/`rocky9` are `vda`.
|
||
Always read the device from the archive name or `virsh domblklist`.
|
||
4. **`virsh domblklist` shows CDROMs with source `-`.** Skip those; only `-iscsi-` paths are backed up.
|
||
5. **Domain name vs zvol name is case-insensitive.** IQN target names are lowercased
|
||
(`iqn...tama:centos7-2`) while the zvol may be `tank/kvm/CentOS7-2`. There are also stale archives
|
||
from an older capitalized `Centos7` domain — don't confuse them with current `centos7`.
|
||
6. **CentOS 6 ignores `virsh shutdown`** (no/old acpid). It may need several minutes or a
|
||
`virsh destroy`. Never start writing until `virsh domstate` says `shut off`.
|
||
7. **Don't run during the backup window** (~07:24–07:55 daily). The backup holds
|
||
`/tmp/backup-zvol-iscsi.pid` on tama and would be snapshotting the same zvols.
|
||
|
||
## Procedure
|
||
|
||
### 1. Shut down the VMs (on kiki, .5)
|
||
|
||
```bash
|
||
ssh root@10.0.0.5 'for d in vm1 vm2; do virsh shutdown "$d"; done'
|
||
# then poll until every one reports "shut off" — do not proceed otherwise
|
||
ssh root@10.0.0.5 'for i in $(seq 1 30); do s=$(virsh domstate vm1); [ "$s" = "shut off" ] && break; sleep 5; done; virsh domstate vm1'
|
||
```
|
||
|
||
If a guest won't go down gracefully, confirm with the user before `virsh destroy`.
|
||
|
||
### 2. Find the archives (from tama, .6)
|
||
|
||
```bash
|
||
ssh root@10.0.0.6 'export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes
|
||
borg list --format "{archive}{TAB}{time}{NL}" root@10.0.0.5:/media/Storage/Backup/kvm' \
|
||
| grep -Ei "^(vm1|vm2)-"
|
||
```
|
||
|
||
Then record each archive's exact byte size — it must equal the zvol's size:
|
||
|
||
```bash
|
||
borg info "::$ARCHIVE" # "Original size" column, or borg list shows the stdin item size
|
||
blockdev --getsize64 /dev/zvol/tank/kvm/<zvol>
|
||
```
|
||
|
||
### 3. Map disks to zvols
|
||
|
||
```bash
|
||
ssh root@10.0.0.5 'virsh domblklist <domain>' # dev -> /dev/disk/by-path/...-iscsi-iqn...:<target>-lun-0
|
||
ssh root@10.0.0.6 'zfs list -o name,volsize,volmode -r tank/kvm -d 1'
|
||
```
|
||
The zvol is the IQN target name after the last colon, matched case-insensitively under `tank/kvm/`.
|
||
|
||
### 4. Optional rollback point
|
||
|
||
A pre-restore `zfs snapshot tank/kvm/<z>@pre-restore-<date>` is cheap insurance, **but ask first** —
|
||
it is not always wanted, and the daily backup snapshots (`SNAPSHOTS_KEEP=2`) may already cover it.
|
||
|
||
### 5. Restore (on tama, .6)
|
||
|
||
Run detached — 500 GiB takes ~20 min at ~475 MB/s, and an SSH drop must not kill it.
|
||
Guard every device write behind a size check and an existence check.
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
export BORG_REPO="root@10.0.0.5:/media/Storage/Backup/kvm"
|
||
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes
|
||
export BORG_RELOCATED_REPO_ACCESS_IS_OK=yes
|
||
set -o pipefail
|
||
|
||
# zvol|archive|expected_bytes (pipe-delimited: archive names contain colons)
|
||
JOBS=(
|
||
"centos7|centos7-vda-2026-08-14T07:38:41|536870912000"
|
||
)
|
||
|
||
for J in "${JOBS[@]}"; do
|
||
IFS="|" read -r ZVOL ARCHIVE EXPECT <<<"$J"
|
||
DEV="/dev/zvol/tank/kvm/$ZVOL"
|
||
echo "=== $(date "+%F %T") restoring $ARCHIVE -> $DEV ($EXPECT bytes)"
|
||
|
||
[[ -b "$DEV" ]] || { echo "FAIL: $DEV not a block device"; exit 1; }
|
||
|
||
ACTUAL=$(blockdev --getsize64 "$DEV")
|
||
[[ "$ACTUAL" == "$EXPECT" ]] || { echo "FAIL: $DEV is $ACTUAL, archive is $EXPECT"; exit 1; }
|
||
|
||
borg info "::$ARCHIVE" >/dev/null 2>&1 || { echo "FAIL: archive $ARCHIVE not found"; exit 1; }
|
||
|
||
borg extract --stdout "::$ARCHIVE" \
|
||
| dd of="$DEV" bs=4M iflag=fullblock conv=fsync status=progress \
|
||
|| { echo "FAIL: restore of $ARCHIVE failed"; exit 1; }
|
||
|
||
echo "=== $(date "+%F %T") completed $ZVOL"
|
||
done
|
||
echo "=== $(date "+%F %T") ALL RESTORES COMPLETE"
|
||
```
|
||
|
||
Launch and watch:
|
||
|
||
```bash
|
||
ssh root@10.0.0.6 'nohup setsid /root/restore.sh > /root/restore.log 2>&1 </dev/null & echo $!'
|
||
ssh root@10.0.0.6 'grep -aE "^===|FAIL:" /root/restore.log; tail -c 120 /root/restore.log'
|
||
```
|
||
|
||
`iflag=fullblock` matters: without it `dd` can do short reads from the pipe and write a short image.
|
||
Writing to `/dev/zvol/...` while LIO exports it is safe **only** because the guest is powered off —
|
||
LIO takes no exclusive lock, so a running VM would race and corrupt.
|
||
|
||
### 6. Flush stale initiator cache (on kiki, .5)
|
||
|
||
The zvol changed underneath the iSCSI initiator, so kiki's block-layer cache for that LUN is stale.
|
||
Before booting, flush it:
|
||
|
||
```bash
|
||
ssh root@10.0.0.5 'blockdev --flushbufs /dev/disk/by-path/ip-10.0.100.6:3260-iscsi-iqn.2026-03.im.gec.tama:<target>-lun-0'
|
||
```
|
||
|
||
A `iscsiadm -m node -T <iqn> --rescan`, or logout/login of the session, is the heavier alternative.
|
||
|
||
### 7. Boot and verify
|
||
|
||
```bash
|
||
ssh root@10.0.0.5 'virsh start <domain>; virsh domstate <domain>'
|
||
```
|
||
|
||
Restoring the domain XML is a **separate** decision — the `-xml-` archives exist, but rolling back a
|
||
disk rarely needs the definition rolled back too, and doing so can undo unrelated config. Ask first:
|
||
|
||
```bash
|
||
borg extract --stdout "::<domain>-xml-<ts>" > /tmp/<domain>.xml # then virsh define
|
||
```
|