Shrinking physical volumes in LVM on a Linux Guest in ESXi 5.0

  • The problem:

    Linux guest (OpenSuse 12.1), with multiple virtual disks attached.

    3 disks are in a logical volume, two of which are exactly 2TB.

    None of the disks are independent, and due to the backup software we use, cannot be independent.

    When the two 2TB virtual disks are "dependent", the snapshot fails stating that the file is too large for the datastore. When I put those two disks in independent mode, snapshots work fine (the other disk is 1.8TB).

    I have therefore concluded that even shrinking the two physical disks by 100GB should solve the problem, however I am having trouble conceptualizing how to go about getting those disks smaller without breaking the LVM entirely.

      The actual LV has 1.3TB free, so there is plenty of space to shrink with.

      What I need to accomplish:

      Deallocate 100GB from the two, 2TB virtual disks within the linux guest.

      Shrink the two virtual disks by 100GB within vsphere (not as complicated).

      Are there any vsphere/LVM gurus that can give me a clue?

      Edit:

      Fixing formatting:

      Something like this? e2fsk -f /dev/VGroup1

      resize2fs /dev/VGroup1 5922108040K (that is a 200GB shrink in KB)

      lvreduce -L 209715200K /dev/Vgroup1 pvresize /dev/sdb1(and sdc1) --

      setphysicalvolumesize 2042625023K Correct?

        Another thought occurred to me: Maybe to play on the safe side I should reduce 25G more than I plan on reducing the disks, to ensure that the physical volumes aren't smaller than the filesystem.

      Answers(25)

      • yes, you need to have the mapping from before deletion. – Hubert Kario Oct 12 '11 at 12:50

        • Oh it's multi-extent? hmmm, not a big fan of doing that but yeah, I can see how that changes things, still at least you'll be able to follow that tool chain I mentioned. – Chopper3 Apr 15 '12 at 12:40

        • There is 4TB free on the datastore, so I don't think that is the issue. – Stew Apr 15 '12 at 12:38

        • # lvextend -L +50GB /dev/VolGroup01/fileserver.home 
            Extending logical volume fileserver.home to 300.00 GB
            Logical volume fileserver.home successfully resized
          
          # e2fsck -f /dev/VolGroup01/fileserver.home
          e2fsck 1.39 (29-May-2006)
          /dev/VolGroup01/fileserver.home: recovering journal
          Pass 1: Checking inodes, blocks, and sizes
          Pass 2: Checking directory structure
          Pass 3: Checking directory connectivity
          Pass 4: Checking reference counts
          Pass 5: Checking group summary information
          
          
          # resize2fs /dev/VolGroup01/fileserver.home 300G
          resize2fs 1.39 (29-May-2006)
          Resizing the filesystem on /dev/VolGroup01/fileserver.home to 78643200 (4k) blocks.
          The filesystem on /dev/VolGroup01/fileserver.home is now 78643200 blocks long.
          

            done!

            • Welcome to Server Fault. Please don't post "me too" or "thank you" type message on Server Fault. This site is for Questions and the associated Answers. If you have any new questions please use the Ask Question button in the upper right corner of every page. – Chris S ♦ Jul 7 '10 at 14:58

            • to recover a deleted LVM volume I'd guess you'd need to go low level and re-create the volume using device mapper, not lvm. Also, you can use lvdisplay --maps to see LE to PE mapping (the exact and most important thing you need to recreate). – Hubert Kario Oct 9 '11 at 23:39

            • try inspecting the block device with sudo less -f /dev/sda5 which should show you all recent changes to lvm metadata. This may be more accurate than what vgcfgrestore finds in /etc/lvm especially when the disk is corrupted. Try extracting the right one by timestamp to a file and run vgcfgrestore from the file. – Martian Nov 19 '11 at 19:50

              • There aren't any really good recovery options, and there are no tools that support this to my knowledge. However, see the data recovery section of LVM dangers and caveats for some articles on manual recovery.

                Generally it's best to do a raw image backup of the broken volume(s) or underlying disks, and do the data recovery on the backup versions, so that you have a way of retrying the recovery.

                The above answer also has a section on resizing LVM volumes - expanding them is reasonably safe, and it's usually better to use lvresize instead of deleting and recreating.

                On a related point: since you are using VMware, you should also take care that hard disk write cache flushes (write barriers) propagate correctly from the guest Linux kernel through the hypervisor and any host OS. It's also important that write barriers are set up in the guest FS and guest kernel, which should be 2.6.33 or higher.

              • There is 4TB free on the datastore, so I don't think that is the issue. – Stew Apr 15 '12 at 12:38

              • I removed the comment and added it to an edit on the original question, better formatting. – Stew Apr 15 '12 at 13:03

              • In your XEN config, don't attach the LV to xvda, attach it to something like xvda1 etc. The xvda device in your domU won't exist, but your domU will still see /dev/xvda1 as a valid partition.

              • This isn't a VMWare issue really, the issue with the 2TB vmdk's is that there's no space left on the datastore to commit to a snapshot, as you say dropping the size of the vmdk will allow that to work.

                Now obviously you can use the usual chain of umount, e2fsck, resize2fs, lvreduce and pvresize then reduce the vmdk size within the vsclient, but there's another thought, if you have enough temporary space you could just convert them to thin disks. Obviously there can be a write penalty for this but it'd mean you'd not have to touch your guest filesystem.

              • Your problem here is that you can't resize ext3 partition with parted. you have to remove the journal (turning ext3 into ext2) and then resize.

                see this for more info

                http://www.hermann-uwe.de/blog/resizing-ext3-partitions-with-parted

              • I avoid RHEL like the plague, so I'd just use xen-tools to create the domU and avoid the RHEL installer altogether. – womble Sep 17 '09 at 20:29

                • Oh it's multi-extent? hmmm, not a big fan of doing that but yeah, I can see how that changes things, still at least you'll be able to follow that tool chain I mentioned. – Chopper3 Apr 15 '12 at 12:40

                • A thought: Perhaps the simplest way to achieve this would be: Add an additional 1.8TB disk, run pvmove on one of the 2TB disks, when all the data is moved off that disk to the new one, remove it from the vg and what virtual machine. Rinse repeat for the second disk. As a matter of fact I am going to try that today. – Stew Apr 15 '12 at 11:52

                • A thought: Perhaps the simplest way to achieve this would be: Add an additional 1.8TB disk, run pvmove on one of the 2TB disks, when all the data is moved off that disk to the new one, remove it from the vg and what virtual machine. Rinse repeat for the second disk. As a matter of fact I am going to try that today. – Stew Apr 15 '12 at 11:52

                • Thanks, I would like to use the LV directly, but the RHEL installer requires that I partition /dev/xvda to create /dev/xvda1 for the / partition. Is there a way around this? I'm not editing the partition table when the domU is running. – z0mbix Sep 17 '09 at 14:50

                • I removed the comment and added it to an edit on the original question, better formatting. – Stew Apr 15 '12 at 13:03

                • I would need to have run "lvdisplay --maps" before deleting the volumes, right? (When I run "lvdisplay --maps" on the new server, it shows the LE to PE are mapped the way I would expect, with the first logical volume starting at PE 0.) – Vegar Nilsen Oct 10 '11 at 9:05

                • Every time you perform an operation with LVM, by default, the previous metadata is archived in /etc/lvm/archive. You can use vgcfgrestore to restore it, or grab the extends by hand (harder, but lvcreate(8) should cover it).

                  Edit:

                  And to make it as easy as possible, I should add that you can find the last backup before your destructive operation by looking at descriptions:

                  # grep description /etc/lvm/archive/vg01_*
                  /etc/lvm/archive/vg01_00001.vg:description = "Created before executing 'lvremove -f /dev/vg01/foo'"
                  /etc/lvm/archive/vg01_00002.vg:description = "Created before executing 'lvremove -f /dev/vg01/bar'"
                  /etc/lvm/archive/vg01_00003.vg:description = "Created before executing 'lvremove -f /dev/vg01/baz'"
                  

                    Edit:

                    The normal allocation policy (default one) will allocate a stripe from the first free PE when there is enough room to do so. If you want to confirm where the LV was allocated, you can look in the archive files, those are perfectly readable by humans.

                    • This did not go well, however I am going to attempt it again when I can plan for a bit of downtime of that volume. Thanks for the answer, this is the route I am going to try and worse case scenario I can delete everything and recreate it from backup. Thanks! – Stew Apr 17 '12 at 7:02

                    • This did not go well, however I am going to attempt it again when I can plan for a bit of downtime of that volume. Thanks for the answer, this is the route I am going to try and worse case scenario I can delete everything and recreate it from backup. Thanks! – Stew Apr 17 '12 at 7:02

                      • Why are you partitioning the LV, instead of just using it directly? Also, if you are going to be manipulating the partition table, it's best to do it in the guest. Worst, it looks like you might be trying to fiddle with the partition table in the dom0 while the domU is still running... dangerous.

                        My simple recipe for resizing a domU disk, which I've done probably in excess of a hundred times by now, is to have the domUs with the LV as the full root partition (xvda1) and then running:

                        lvextend -L+NG -n domu-root vg
                        xm shutdown -w domu
                        xm create domu
                        ssh domu resize2fs /dev/xvda1
                        

                        And voila, all done. For non-root filesystems, you can just detach/reattach (useful for swap, in particular), but root needs the reboot.

                        • This isn't a VMWare issue really, the issue with the 2TB vmdk's is that there's no space left on the datastore to commit to a snapshot, as you say dropping the size of the vmdk will allow that to work.

                          Now obviously you can use the usual chain of umount, e2fsck, resize2fs, lvreduce and pvresize then reduce the vmdk size within the vsclient, but there's another thought, if you have enough temporary space you could just convert them to thin disks. Obviously there can be a write penalty for this but it'd mean you'd not have to touch your guest filesystem.