OOOPS.... Now what?

Lord.DragonFly.of.Dawn · Posted: Mon Jul 27, 2025 2:12 am Post subject: OOOPS.... Now what?

so we are all familiar with the following command, and it fills us with dread to our very bones...

lynch · Posted: Mon Jul 27, 2025 8:54 am Post subject:

Recover everything? I doubt it.
At least you can congratulate yourself for doing backups.
What was the equivalent command?

_________________
Mandriva 2025 Spring -2.6.31.12-server-2mnb
PCLinuxOS 2025 -2.6.26.8.tex3

nukes · Linux Guru Joined: 29 Aug 2025 Posts: 4558

It depends on the filesystem, assuming you immediately unmounted it, and haven't replayed the journal (e.g. mount read-only even) it may be possible.

One tool to look at is ext3grep, which is quite complicated to use. You need to discover the inode of the relevant files/directories, then the block that contained them - which requires you to pick through the journal backwards. Not easy, but sometimes possible.
http://code.google.com/p/ext3grep

If it's reiserfs, you might try --rebuild-tree, sometimes that can help, but more often than not it produces unintended results.

This all depends on the actual command you did, if it was something like cat /dev/zero > /dev/sda or whatever, you're not going to get the data back.

Backups are all well and good, but you need a procedure for restoring them (too many places get caught up with the idea of backups, but not the act of restoration, which is the main point). at least you've got them Wink

_________________
Gentoo x86-64 2.6.29.1
FreeBSD 7-CURRENT
Arch x86 2.6.30

Lord.DragonFly.of.Dawn · Posted: Mon Jul 27, 2025 5:34 pm Post subject:

the equivalent command was actually run by a script.

mushroom · Posted: Tue Jul 28, 2025 4:54 am Post subject:

nukes · Linux Guru Joined: 29 Aug 2025 Posts: 4558

Lord.DragonFly.of.Dawn · Posted: Wed Jul 29, 2025 3:05 am Post subject:

I mayhap have been using the same script....

or i should really not write shell scripts at 3AM after a night at the bar.....

_________________
ArchLinux x86_64 - Custom Built Desktop
ArchLinux x86_64 - Compaq CQ50 Laptop
ArchLinux i686 - Acer Aspire One Netbook
ArchLinux i686 - Dell Presario ze2000 (w/ shattered LCD)

PuppyLinux, CloneZilla, PartedMagic, DBAN - rescue thumbdrives
Windows 7 (x86_64 desktop alternate boot)

masinick · Posted: Wed Jul 29, 2025 4:02 pm Post subject: A variant: rm -fre /

One lesson for the wise - ANY script that does an rm -fr of anything should have checks in it - and at least a echo or two to help diagnose it, but if it has variables and their values are empty or NULL, that should raise a flag and force termination of the script - but with a few diagnostic messages before termination to help deduce what is going on.

Some of the better Bourne, Korn, and Bash scripts put ${var} in place and evaluate ${var} before even placing it into any kind of expression. Failing to do that, as we can see - and as I have also experienced, can lead to some VERY interesting results, to say the least!

This reminds me of my own story. In March 1998 I left Digital Equipment Corporation after a very nice 13 1/2 year career there. My last job was in the Digital UNIX Engineering organization, specifically in the DUDE - Digital UNIX Development Environment, working on Internationalization (I18N) and Localization (L10N) --- see why we abbreviate? Those are LONG words!

Anyway, I always used to kid about using rm -fr /, (cd / && rm -fr *), and similar variations. Well, it just so happened that the Digital UNIX implementation of the rm command also happened to offer an e option to list errors and output. So a great way to watch a system self destruct is the command rm -fre /.

Since our organization produced and frequently tested nightly builds, plus we had every production release available, the fact that it took all of 20-30 minutes to build a new system from scratch, and the fact that whoever inhereted the systems that I was leaving would almost certainly rebuild them to suit their own needs, I thought, "What a great way to see the system - themas - go down in flames as I got ready to leave!" Only problem was that there was SO MUCH output that it took a long time to SEE all of the files getting removed as the system removed itself! Smile

I finally gave up watching it around 4:30 PM and turned in my badge as I left. It was interesting watching it tear itself apart! Smile

_________________
Brian Masinick
Distros: SimplyMEPIS
sidux - no CAPS!, antiX, Debian

Lord.DragonFly.of.Dawn · Posted: Wed Jul 29, 2025 5:26 pm Post subject:

mkfs is quicker, but not nearly so satisfying.....

_________________
ArchLinux x86_64 - Custom Built Desktop
ArchLinux x86_64 - Compaq CQ50 Laptop
ArchLinux i686 - Acer Aspire One Netbook
ArchLinux i686 - Dell Presario ze2000 (w/ shattered LCD)

PuppyLinux, CloneZilla, PartedMagic, DBAN - rescue thumbdrives
Windows 7 (x86_64 desktop alternate boot)

masinick · Posted: Wed Jul 29, 2025 5:44 pm Post subject:

nukes · Linux Guru Joined: 29 Aug 2025 Posts: 4558

well, mkfs you could potentially recover from, only one directory is created (the root).
rm-ing everything, especially on journalled filesystems generally destroys the directory descriptors for each and every one.

wrt after nights at the bar - when i first went to uni there were a number of times I woke up and my computer was hosed - had to use a livecd to get it back up and running. I had vague memories of trying to reconfigure the whole system to do some silly thing or work in some ridiculous way but only getting half way through or getting stuck etc.

WRT self destruction, my best experience was on FreeBSD 6 - I was trying to rebuild the whole system with GCC4, and fix any compilation problems along the way. I was logged into X at the time, and the system literally broke around me - various features stopped working, and by the end of it I couldn't launch any programs (even xterm or csh) and all i had were xchat and a few terminals already running.

I didn't even bother trying to boot back into it after that Smile

_________________
Gentoo x86-64 2.6.29.1
FreeBSD 7-CURRENT
Arch x86 2.6.30

Lord.DragonFly.of.Dawn · Posted: Wed Jul 29, 2025 9:36 pm Post subject:

or how about this fun one?

masinick · Posted: Wed Jul 29, 2025 9:38 pm Post subject:

That looks like you can wipe out the entire drive with that one!

_________________
Brian Masinick
Distros: SimplyMEPIS
sidux - no CAPS!, antiX, Debian

Lord.DragonFly.of.Dawn · Posted: Thu Jul 30, 2025 12:42 am Post subject:

yep... and you see on the screen whats getting written to the drive!

_________________
ArchLinux x86_64 - Custom Built Desktop
ArchLinux x86_64 - Compaq CQ50 Laptop
ArchLinux i686 - Acer Aspire One Netbook
ArchLinux i686 - Dell Presario ze2000 (w/ shattered LCD)

PuppyLinux, CloneZilla, PartedMagic, DBAN - rescue thumbdrives
Windows 7 (x86_64 desktop alternate boot)

masinick · Posted: Thu Jul 30, 2025 12:56 am Post subject:

	USA Linux Users Group Forum Index » Servers and Server Administration	All times are GMT Goto page 1, 2 Next
Page 1 of 2