[TriLUG] pilot/trilug.org reboot - guess what I learned?
porter at trilug.org
Tue Sep 15 22:54:58 EDT 2009
Hey, guess what I learned about LVM today?
A few months ago, I made a complete backup of /home on pilot. I did
this following the guidelines that have been told to me by smart people:
- make a snapshot LV of /home (called /home-snap)
- make a new empty partition (called /home-backup)
- 'dd' from /home-snap to /home-backup
- remove the snapshot LV
Guess what I learned today? The ext3 UUID of /home-backup is the same
as /home. You can check this with the "blkid" command.
So when pilot rebooted, it mounted the WRONG copy of /home. It looked
in /etc/fstab to see what to mount, read the UUID, and started looking
for that filesystem. Unfortunately, it found 'home-backup' before it
found of 'home', and so our users took a step back in time for the
evening. All of the files in our home directories looked like they used
to in May.
So far, no harm, no foul.
We fixed the problem at about 10pm tonight. I just did a "tune2fs -U
random /dev/mapper/vg01-home.bak.20090517", and that gave the backup
filesystem a new UUID.
But that means that any files that you created between 6pm and 10pm were
written to the 'home-backup' partition and not to your real home
directory. Most people were not logged on at that time. But if you
received any email in that time period, it might have gotten dumped into
(home-backup)/trilug/username/Maildir instead of where it SHOULD have put.
Gosh, this is a lot like "Back to the Future", when Marty's mom tries to
kiss him. Except the characters involved here are not as good-looking.
If you are concerned about those four hours worth of emails, or about
the 18 minutes of missing Watergate recordings, you might want to take a
look in /mnt/home.bak.20090517/trilug/YOURUSERNAME and see if they are
hiding in there.
If I weren't such a klutz, I would write a shell script that looks for
files in that May 2009 partition that have September 2009 dates. But I
have done enough damage for one night.
Let me know if I can help clarify what happened and how to look for any
More information about the TriLUG