Shrinking physical volumes in LVM on a Linux Guest in ESXi 5.0

  • The problem:

    Linux guest (OpenSuse 12.1), with multiple virtual disks attached.

    3 disks are in a logical volume, two of which are exactly 2TB.

    None of the disks are independent, and due to the backup software we use, cannot be independent.

    When the two 2TB virtual disks are "dependent", the snapshot fails stating that the file is too large for the datastore. When I put those two disks in independent mode, snapshots work fine (the other disk is 1.8TB).

    I have therefore concluded that even shrinking the two physical disks by 100GB should solve the problem, however I am having trouble conceptualizing how to go about getting those disks smaller without breaking the LVM entirely.

    The actual LV has 1.3TB free, so there is plenty of space to shrink with.

    What I need to accomplish:

    Deallocate 100GB from the two, 2TB virtual disks within the linux guest.

    Shrink the two virtual disks by 100GB within vsphere (not as complicated).

    Are there any vsphere/LVM gurus that can give me a clue?

    Edit:

    Fixing formatting:

    Something like this? e2fsk -f /dev/VGroup1

    resize2fs /dev/VGroup1 5922108040K (that is a 200GB shrink in KB)

    lvreduce -L 209715200K /dev/Vgroup1 pvresize /dev/sdb1(and sdc1) --

    setphysicalvolumesize 2042625023K Correct?

    Another thought occurred to me: Maybe to play on the safe side I should reduce 25G more than I plan on reducing the disks, to ensure that the physical volumes aren't smaller than the filesystem.

Answers(84)

  • also, who is Jeff? – cnst Aug 24 at 20:33

  • I approve of this guy, despite his vanilla bean look and personality. – TheCleaner 2 days ago

  • It's a shame you weren't elected last year, your answers to the community questions were some of the best of the lot. Everyone who is even a little bit interested in the elections should read up on last years'. – Reaces Aug 25 at 13:36

    • @Wesley Originally meant "a field to the west" clearly an implied "wasteland to the west", which is terrorist rhetoric if I ever saw it! QED. (I like this game we should play it more often) – Reaces 2 days ago

    • This did not go well, however I am going to attempt it again when I can plan for a bit of downtime of that volume. Thanks for the answer, this is the route I am going to try and worse case scenario I can delete everything and recreate it from backup. Thanks! – Stew Apr 17 '12 at 7:02

    • @TheCleaner There's still some room for improvement! – EEAA 2 days ago

    • @PaulWhite Sometimes, telling people they are out of their depth is the best way to help them. That very much applies to the question you link to. – Iain Aug 25 at 6:58

      • Yes, they do. Large numbers of extremely poor questions dissuade people from participating in the site, and poor quality content on the site in general reduces the "Google value" of the site overall. ServerFault is not "ask a sysadmin-dot-com", it's a resource for professional sysadmins to help each other and find solutions to the problems they specifically face. The question you cite did nothing to further that goal. – womble Aug 25 at 2:15

        • A thought: Perhaps the simplest way to achieve this would be: Add an additional 1.8TB disk, run pvmove on one of the 2TB disks, when all the data is moved off that disk to the new one, remove it from the vg and what virtual machine. Rinse repeat for the second disk. As a matter of fact I am going to try that today. – Stew Apr 15 '12 at 11:52

        • How do you define the trolls? – peterh 2 days ago

        • @Reaces Your name has homolytic similarities to the Reese's candy company which makes preservative laden, fat filled candies sold to hundreds of millions of people, so clearly you are interested in the mass murder of the world population over long periods of time making you worse than Hitler. – Wesley 2 days ago

          • That's a good point. Thanks, @MarkHenderson. – womble Aug 25 at 0:31

          • This did not go well, however I am going to attempt it again when I can plan for a bit of downtime of that volume. Thanks for the answer, this is the route I am going to try and worse case scenario I can delete everything and recreate it from backup. Thanks! – Stew Apr 17 '12 at 7:02

          • Every time you perform an operation with LVM, by default, the previous metadata is archived in /etc/lvm/archive. You can use vgcfgrestore to restore it, or grab the extends by hand (harder, but lvcreate(8) should cover it).

            Edit:

            And to make it as easy as possible, I should add that you can find the last backup before your destructive operation by looking at descriptions:

            # grep description /etc/lvm/archive/vg01_*
            /etc/lvm/archive/vg01_00001.vg:description = "Created before executing 'lvremove -f /dev/vg01/foo'"
            /etc/lvm/archive/vg01_00002.vg:description = "Created before executing 'lvremove -f /dev/vg01/bar'"
            /etc/lvm/archive/vg01_00003.vg:description = "Created before executing 'lvremove -f /dev/vg01/baz'"
            

            Edit:

            The normal allocation policy (default one) will allocate a stripe from the first free PE when there is enough room to do so. If you want to confirm where the LV was allocated, you can look in the archive files, those are perfectly readable by humans.

          • @cnst you'd have to ask the junta. – Evan Carroll Aug 24 at 20:33

          • Oh it's multi-extent? hmmm, not a big fan of doing that but yeah, I can see how that changes things, still at least you'll be able to follow that tool chain I mentioned. – Chopper3 Apr 15 '12 at 12:40

          • I like your meta posts and I like that (at least I think what you believe) you feel the moderator is a role of helping users rather than making sure as many questions as possible get closed before someone accidentally answers one. – Jim B 2 days ago

          • Oh yeah? Well I've been unbanned for two years. Vote for me! – Doorknob Aug 24 at 20:28

            • There is 4TB free on the datastore, so I don't think that is the issue. – Stew Apr 15 '12 at 12:38

            • I removed the comment and added it to an edit on the original question, better formatting. – Stew Apr 15 '12 at 13:03

            • I removed the comment and added it to an edit on the original question, better formatting. – Stew Apr 15 '12 at 13:03

            • Thank you @KatherineVillyard – TheCleaner Aug 25 at 18:29

              • Your activity is suspicious … – mate64 Aug 26 at 14:50

              • Do your comments and actions on this question accurately reflect your "passion for helping people?". If so, why? – Paul White Aug 25 at 2:08

                • @NathanOsman, nuh-uh, I don't see a space in serverfault.com – Gene Aug 24 at 20:42

                • Another one of the strong contenders from last year, who unfortunately did not make it! Well I guess I can look into voting the exact same way this year the way things are panning out. – Reaces Aug 27 at 7:18

                • I like that you're patient, fair, and genuinely nice--all excellent qualities for a moderator. Your meta posts are also good. – Katherine Villyard 2 days ago

                • SF has a sophisticated spam filter, the few tries to advertise things are filtered out very fast by the reviewers, why would it need moderator intervention? – peterh 2 days ago

                  • I had an issue with your nomination, but couldn't put my finger on it. However thinking about it I don't understand I honestly don't plan to do all that much more . Why would someone ask for more responsibilities and the tools to be able to better serve the community, only to do just about the same things they were before? – Reaces 2 days ago

                  • As a recipient of your dope slaps, I approve this message. – Wesley 2 days ago

                  • I just mean I'm back and I will close questions doesn't give me much to go on. As compared to here's what I've done before being a mod and here's why you should elect me . Saying "well the other stuff was just so obvious I didn't say it" somehow rubs me the wrong way. – Reaces Aug 25 at 21:17

                    • I would say that you fulfill the "patient and fair," "leads by example," and "shows respect for fellow community members" qualifications. – Katherine Villyard Aug 25 at 16:44

                      • Candidate score of 39...slacker. ;) – TheCleaner 2 days ago

                      • As do I. I will vote for you again. – Jenny D Aug 27 at 14:50

                      • Did you call ServerFault a turd? If so, did you mean to do so? And if you did, why would you want to moderate a turd, even if it can be polished? – Thomas Owens Aug 24 at 20:31

                      • @Reaces thanks for the question. Probably because I never understood how it really worked. I have about 1500 reviews total I believe, and was in the top 20 all time for close vote reviews until recently. I never felt like there was much point in flagging something that was already in Review. I guess I don't quite understand still how the process should work. I always flagged something on Main that I saw as necessary to flag, but very rarely do it in Review since I figure its silly to flag something already there. Maybe that's the wrong way to do it? – TheCleaner Aug 25 at 19:05

                      • Oh it's multi-extent? hmmm, not a big fan of doing that but yeah, I can see how that changes things, still at least you'll be able to follow that tool chain I mentioned. – Chopper3 Apr 15 '12 at 12:40

                      • @Reaces no, I don't plan to only close/delete really bad questions. It seemed a little redundant to say, "I would do all the things that moderators usually do". I felt that should be a given for someone who nominates to be a moderator. – womble Aug 25 at 20:55

                      • A Grade guy right here. Great to see you run! – TheCleaner 2 days ago

                        • I agree with Reaces 100%. – Katherine Villyard Aug 25 at 16:44

                          • Allow me the honor of being the first one to say... Probably not. You have in the year since the first election only cast a single flag. How do you expect to be granted moderator responsibilities if you show no interest whatsoever in actually moderating the site? – Reaces Aug 25 at 18:38

                          • I don't plan to take on epic projects such as the bad tag burnation especially in light of the fact that among others the plesk , cpanel and whm tags that have recently been resurrected again. I have neither the inclination nor the time for that. – HBruijn 2 days ago

                          • @TheCleaner For me at this date in this election you are in my top list. Everyone got really good profile, but I like your tone in comment & how you deal with users generally. As some users are exasperated vs newbie on the site and I don't feel that from you when you comment. – yagmoth555 Aug 25 at 19:42

                          • A thought: Perhaps the simplest way to achieve this would be: Add an additional 1.8TB disk, run pvmove on one of the 2TB disks, when all the data is moved off that disk to the new one, remove it from the vg and what virtual machine. Rinse repeat for the second disk. As a matter of fact I am going to try that today. – Stew Apr 15 '12 at 11:52

                          • @peterh I define a troll as someone who's main purpose (on this site) is not to add or contribute anything to help the site, while at the same time attempting either cause tension or just plain spam the site. On SF from what I've seen it's mostly just people spamming the site. – BigHomie 2 days ago

                          • This isn't a VMWare issue really, the issue with the 2TB vmdk's is that there's no space left on the datastore to commit to a snapshot, as you say dropping the size of the vmdk will allow that to work.

                            Now obviously you can use the usual chain of umount, e2fsck, resize2fs, lvreduce and pvresize then reduce the vmdk size within the vsclient, but there's another thought, if you have enough temporary space you could just convert them to thin disks. Obviously there can be a write penalty for this but it'd mean you'd not have to touch your guest filesystem.

                          • SF took a left turn from the other SE communities a while back. Unless you've decided to take the complete opposite approach from last year you have almost no chance of being elected- have you? – Jim B 2 days ago

                          • There is 4TB free on the datastore, so I don't think that is the issue. – Stew Apr 15 '12 at 12:38

                          • Your answers to the community questions last election cycle were poorly received. Has your platform changed since last year? – Katherine Villyard Aug 25 at 19:17

                              • MH is a peaceful mod, while your nick shows you want "cleanup", i.e. destruct content. – peterh 2 days ago

                              • Would you do anything differently with the benefit of hindsite on the question @PaulWhite linked to? – Jim B 2 days ago

                              • @TheCleaner When something is in the first post, late answer or low quality queue it hasn't yet been flagged, it's just been picked up by the SO automated processes. So those should probably be flagged if inappropriate. – Reaces Aug 25 at 19:10

                              • This isn't a VMWare issue really, the issue with the 2TB vmdk's is that there's no space left on the datastore to commit to a snapshot, as you say dropping the size of the vmdk will allow that to work.

                                Now obviously you can use the usual chain of umount, e2fsck, resize2fs, lvreduce and pvresize then reduce the vmdk size within the vsclient, but there's another thought, if you have enough temporary space you could just convert them to thin disks. Obviously there can be a write penalty for this but it'd mean you'd not have to touch your guest filesystem.

                              • I didnt say it would need moderator intervention, I'm just saying if I came across it, I wouldn't let it fly, just as I don't now. I've also come across some pretty slick advertising disguised as answers as well. – BigHomie 2 days ago

                                • @peterh - you should click on my profile and the Youtube clip that shows where my nick comes from. It was given to me years ago for my ability to clean up another IT person's mess. – TheCleaner 2 days ago

                                • Womble is in the UTC+10 timezone. Currently I'm in this timezone, but only for another 3 weeks, when I move to a UTC-5 timezone. It would be good to have a set of hands to cover this zone. – Mark Henderson ♦ Aug 25 at 0:30

                                • I appreciate the honesty, but IMHO we can't have another mod that would perform actions in an opaque manner. – Jim B 2 days ago

                                • @JimB truthfully, I probably wouldn't have left the comment explaining why I voted to close it. There was practically no chance the question was going to be improved, because it was a migration, so there was little value in explaining why the question needed closing. – womble 2 days ago

                                • @yagmoth555 - thank you for the kind words – TheCleaner Aug 25 at 20:46

                                • While I think you're a great contributor to ServerFault, I have one remark. Out of the three serious candidates so far you have by far the least flags cast, while you've been a member as long as Womble and have been more active than Falcon Momot. Mind if I ask why? Do you not use the review queue that often? – Reaces Aug 25 at 18:47

                                • Well, I know who I'm voting for – dyasny Aug 25 at 15:27

                                • @Reaces Fair enough, I'll add some more info to my... not sure what to call it... perhaps "nomination speech"? – womble Aug 25 at 22:59

                                • @Gene look at the page title. Look at the page footer. Anywhere the name is properly capitalized, there is a space. – Nathan Osman Aug 24 at 20:43

                                  • who's the junta? – cnst Aug 24 at 20:34

                                  • You can't govern without a plan. Do you have a plan on how you're going to keep the home router questions out? I say we should have ID checks as people cross the border from SuperUser. I'm not so concerned about our friendly northern neighbors at SO. – LamonteCristo Aug 24 at 20:52

                                  • @Doorknob To be at same level than him it's not wrote, but you must insult SE in your profile bio too – yagmoth555 Aug 24 at 20:33

                                  • Works for me thanks for nominating. Being in Europe can only be a help too with most of the other ♦ and candidates being stacked into USdian timezones. – Iain Aug 27 at 11:03

                                  • yes, you need to have the mapping from before deletion. – Hubert Kario Oct 12 '11 at 12:50

                                  • There aren't any really good recovery options, and there are no tools that support this to my knowledge. However, see the data recovery section of LVM dangers and caveats for some articles on manual recovery.

                                    Generally it's best to do a raw image backup of the broken volume(s) or underlying disks, and do the data recovery on the backup versions, so that you have a way of retrying the recovery.

                                    The above answer also has a section on resizing LVM volumes - expanding them is reasonably safe, and it's usually better to use lvresize instead of deleting and recreating.

                                    On a related point: since you are using VMware, you should also take care that hard disk write cache flushes (write barriers) propagate correctly from the guest Linux kernel through the hypervisor and any host OS. It's also important that write barriers are set up in the guest FS and guest kernel, which should be 2.6.33 or higher.

                                  • I would need to have run "lvdisplay --maps" before deleting the volumes, right? (When I run "lvdisplay --maps" on the new server, it shows the LE to PE are mapped the way I would expect, with the first logical volume starting at PE 0.) – Vegar Nilsen Oct 10 '11 at 9:05

                                  • Judging by your contributions the last few months I'd very much like to vote for you during the next part of the election. However so far your nomination can be summed up in one sentence. I plan to close/delete really bad questions . Is this the only thing you expect to have to do as a moderator? Or more likely is this the reason you're running? I'd very much appreciate a bit more meat to your candidacy! – Reaces Aug 25 at 8:27

                                  • @PaulWhite I agree with you as it was migrated and I seen worst question on SF. BUT, womble did commented why he flagged a close vote. It's rare that people told why, we might agree or disagree, but at least he told is point and by doing so he give the OP the chance to answer back. (Myselft I found it offtopic too, server configuration is for me a talk with a CSE&vendor, as they can simulate the load and suggest proper raid controller and such, a talk a lot over SF's can have. A bad suggestion can backstab the OP) – yagmoth555 Aug 25 at 12:41

                                  • try inspecting the block device with sudo less -f /dev/sda5 which should show you all recent changes to lvm metadata. This may be more accurate than what vgcfgrestore finds in /etc/lvm especially when the disk is corrupted. Try extracting the right one by timestamp to a file and run vgcfgrestore from the file. – Martian Nov 19 '11 at 19:50

                                  • why were you banned? – cnst Aug 24 at 20:31

                                  • @Reaces thanks, that's good to know. Something I didn't know. I wish I could say I understand after all these years the way the site operates in the background (like Hampton and others seem to pick up so easily). I'm definitely not the top choices here, I love Falcon and Womble, but if for some reason you need a 3rd and it comes down to me and Evan...well... :) I have reached over ~2M users according to my profile and tend to be active on Meta and Slack for the site as well. – TheCleaner Aug 25 at 19:28

                                  • For the 100 millionth time... there's a space between "server" and "fault" . – Nathan Osman Aug 24 at 20:35

                                    • As a mod on another site, and someone who's known falcon for ages (and I suspect introduced him to SF), I believe he's make a great moderator. – Journeyman Geek 2 days ago

                                    • Thank heavens. Three decent candidates :) – Journeyman Geek Aug 26 at 5:24

                                        • @TheCleaner Also the street name for someone who makes crime scenes vanish before the authorities show up, so you are clearly here to murder and pillage. QED. – Wesley 2 days ago

                                          • @cnst Jeff and crew. People undemocratically employed by StackExchange. next question. – Evan Carroll Aug 24 at 20:34

                                          • @ThomasOwens because I can make our network great again. next question. – Evan Carroll Aug 24 at 20:33

                                            • @NathanOsman We don't have time to be politically correct. We have real problems. You call it what you want. I'll call it what I want. We know what we're talking about, right? next question. – Evan Carroll Aug 24 at 20:36

                                            • to recover a deleted LVM volume I'd guess you'd need to go low level and re-create the volume using device mapper, not lvm. Also, you can use lvdisplay --maps to see LE to PE mapping (the exact and most important thing you need to recreate). – Hubert Kario Oct 9 '11 at 23:39

                                            • I bet you also want drones to keep the undesirables out, huh? – LamonteCristo Aug 24 at 20:54