Last night's adventure was redoing that command that I've been killing the MythTV machine with. My first suspect was the mirrored LV - something about mirrors came up in the messages when I did it. The second time, I ran pvmove with the -v (verbose) option, which showed the following bit of logging:
[root@media ~]# pvmove -v /dev/hdb /dev/hda3 Wiping cache of LVM-capable devices Finding volume group "vg_storage" Archiving volume group "vg_storage" metadata (seqno 80). Creating logical volume pvmove0 Moving 56559 extents of logical volume vg_storage/lv_storage Moving 86 extents of logical volume vg_storage/lv_swap Moving 0 extents of logical volume vg_storage/lv_backup_root Found volume group "vg_storage" Found volume group "vg_storage" Updating volume group metadata Creating volume group backup "/etc/lvm/backup/vg_storage" (seqno 81). Found volume group "vg_storage" Found volume group "vg_storage" Suspending vg_storage-lv_storage (253:0) Found volume group "vg_storage" Found volume group "vg_storage" Suspending vg_storage-lv_swap (253:1) Found volume group "vg_storage" Creating vg_storage-pvmove0 Loading vg_storage-pvmove0 table device-mapper: reload ioctl failed: Invalid argument ABORTING: Temporary mirror activation failed. Run pvmove --abort. Found volume group "vg_storage" Loading vg_storage-pvmove0 table device-mapper: reload ioctl failed: Invalid argument Loading vg_storage-lv_storage table device-mapper: reload ioctl failed: Invalid argument Found volume group "vg_storage" Loading vg_storage-pvmove0 table device-mapper: reload ioctl failed: Invalid argument Loading vg_storage-lv_swap table device-mapper: reload ioctl failed: No such device or address [root@media ~]# pvmove --abortAnd there it fails. The ioctl makes me think that moving a PV on a physical volume (rather than its own partition) must be what's causing it. Maybe the IO commands just don't like working on that part of the disk. So, test number three - try moving the data off the 160GB disk. Nothing wrong with that, the PV is all on a partition just fine.
Nup. Same problem again.
This time I'm quicker to restore the system to working order. I booted up off the System Rescue CD, but I couldn't issue the pvmove --abort command because the VG isn't complete (because I've unplugged one drive for the DVD). That's OK. I resize my root partition, create a new, small partition in the available space, copy the root filesystem off the System Rescue CD (i.e. off where it's uncompressed it - nifty trick, the partition image has a header that's a shell script that loads the cloop driver and mounts itself), copy the initrd and vmlinuz files off where the CD boots from, mangle up a new item in grub's config with those files, and, after changing the drives back so the LVM can initialise correctly, boot it.
It doesn't quite go perfectly, but it gets a long way further than I'd feared. I spend a bit of time trying to work out how this particular mangling of gentoo boots, but I don't get too far. So I just mount the uncompressed file system where it should be, run pvmove --abort and check the LVM state: everything comes up good. Reboot again and it's all working.
So now I'm making my own little equivalent to Dell's system rescue install - an unobtrusive partition that will give me all the tools to get the MythTV machine back on line without having to fiddle with cables and disable the LVM and so forth. Maybe I should sit down and make something like this - a version of the System Rescue CD, or a tool within its boot config, to install a version of the System Rescue CD on a partition, including (if possible) plumbing the correct options into grub to get it to come up as an option. It makes a lot of sense for those times when you want to get around a failure but can't put in a CD (or can't get in the box to attach one).
Maybe all the true Linux gurus just boot off their distro's rescue CD and go from there, though.
All posts licensed under the CC-BY-NC license. Author Paul Wayper.