Source
r/sysadmin•
DMatty

Soo..... Just found the 2nd most scary linux command...

I was working with a colleague today with validating a linux failover cluster and needed a way of testing a lockup/system failure.

We thought -- why not generate a kernel panic? You can force a blue screen in Windows, there's gotta be a way in linux. So, with a quick lookup we found this:

echo c > /proc/sysrq-trigger as found here: http://unix.stackexchange.com/questions/66197/how-to-cause-kernel-panic-with-a-single-command

Well, it works. And that really begs the question, how the hell could you stop that? Because seriously... that's scary. a simple echo redirected at a system file and it instantly locks up a linux machine -- it doesn't even log the command to history so it's not even auditable.

If you have sudo enabled without a root password, any privileged user could stop a box. And, god forbid some kiddy puts a script to run this in /etc/init.d to run it on boot or in cron.

And there you have it, behind rm -rf /, the second most scary linux command.

...Wow

Posted Jul 12, 2016, 2:41 PM UTC

3299
unix_hereticGentleman BOFH•

"Unix gives you just enough rope to hang yourself -- and then a couple of more feet, just to be sure."

-- Eric Allman

79
Zaphod_Bchown -R us ~/.base•

For those who have not read this book, enjoy

3
tubezninjaIt's not a Big Truck•

I'm laughing at the foreword:

As for me? I switched to the Mac. No more grep, no more piping, no more SED scripts. Just a simple, elegant life: “Your application has unexpectedly quit due to error number –1. OK?”

Donald A. Norman

Apple Fellow

Apple Computer, Inc.

Then, just five years later, and there's UNIX on Macs and we get the seed for variants of UNIX running on smartphones and even watches.

2
Zaphod_Bchown -R us ~/.base•

That book is pretty awesome. The Anti-Foreword was also really good, and written by Dennis Ritchie himself.

1
deeseearrSysadmin•

How the hell would you stop that?

[user@host:~] ls -l /proc/sysrq-trigger 
--w------- 1 root root 0 Jul 12 11:08 /proc/sysrq-trigger

That way.

If you're looking for ways to prevent the root user from shutting down your system, then I'm afraid you're doing it wrong.

46
randomguy186DOS 6.22 sysadmin•

While we're at it, I need to find a way to prevent a user with physical access to my server from wrecking it. We installed metal detectors at the building entrances, so they can't bring shotguns to the server room anymore, but last week someone banged up a file server with a 2x4.

19
deeseearrSysadmin•

As the old saying goes, "The only secure computer is one that's unplugged, locked in a safe, and buried 20 feet under the ground in a secret location... and I'm not even too sure about that one."

Have you considered a secret underwater data centre? Or perhaps encasing the whole thing in several feet of concrete? Putting everything on a remote desert island inside of a volcano just encourages secret agents to come by and blow it all up, so don't make that mistake.

13
Stealthy_WolfJack of All Trades•

in a sealed room filled with mineral oil,

3
cookiemonsterwave•

So you're saying Bond villans had the most secure servers?

1
eccitazeStudent•

No, no, no, weren't you listening? Bond villains with their servers in a remote volcanic island had the least secure servers. Might as well have used a 8-year old unpatched Linux server with iptables disabled, no root password, and port 22 open to the Internet.

3
Hillsy21•

The bond villains had the same sysadmins as Hilary Clinton

0
gex80Wannabe VMware Admin•

so they can't bring shotguns to the server room anymore,

Anymore?

6
angryadminDont go away angry, just go away.•

I read that in my head with Archer's voice.

3
randomguy186DOS 6.22 sysadmin•

Joke.

Your head.

Any questions?

-7
Jeoh•

A lock?

3
dlyk•

If you don't have armed guards everywhere in your company, y'all doing something wrong. All those Blackwater (sorry, Academi) boys need jobs now that there's no more war.

2
[deleted]•

[deleted]

1
Zaphod_Bchown -R us ~/.base•

chflags schg /proc/sysrq-trigger

I have not tested this, but that basically makes that file immutable to the root user. You will not be able to write to it, until you remove the flags.

1
walrus01•

Unix - The Hole Hawg THE HOLE HAWG OF OPERATING SYSTEMS

by Neal Stephenson

Unix has always lurked provocatively in the background of the operating system wars, like the Russian Army. Most people know it only by reputation, and its reputation, as the Dilbert cartoon suggests, is mixed. But everyone seems to agree that if it could only get its act together and stop surrendering vast tracts of rich agricultural land and hundreds of thousands of prisoners of war to the onrushing invaders, it could stomp them (and all other opposition) flat.

It is difficult to explain how Unix has earned this respect without going into mind-smashing technical detail. Perhaps the gist of it can be explained by telling a story about drills.

The Hole Hawg is a drill made by the Milwaukee Tool Company. If you look in a typical hardware store you may find smaller Milwaukee drills but not the Hole Hawg, which is too powerful and too expensive for homeowners. The Hole Hawg does not have the pistol-like design of a cheap homeowner's drill. It is a cube of solid metal with a handle sticking out of one face and a chuck mounted in another. The cube contains a disconcertingly potent electric motor. You can hold the handle and operate the trigger with your index finger, but unless you are exceptionally strong you cannot control the weight of the Hole Hawg with one hand; it is a two-hander all the way. In order to fight off the counter-torque of the Hole Hawg you use a separate handle (provided), which you screw into one side of the iron cube or the other depending on whether you are using your left or right hand to operate the trigger. This handle is not a sleek, ergonomically designed item as it would be in a homeowner's drill. It is simply a foot-long chunk of regular galvanized pipe, threaded on one end, with a black rubber handle on the other. If you lose it, you just go to the local plumbing supply store and buy another chunk of pipe.

During the Eighties I did some construction work. One day, another worker leaned a ladder against the outside of the building that we were putting up, climbed up to the second-story level, and used the Hole Hawg to drill a hole through the exterior wall. At some point, the drill bit caught in the wall. The Hole Hawg, following its one and only imperative, kept going. It spun the worker's body around like a rag doll, causing him to knock his own ladder down. Fortunately he kept his grip on the Hole Hawg, which remained lodged in the wall, and he simply dangled from it and shouted for help until someone came along and reinstated the ladder.

I myself used a Hole Hawg to drill many holes through studs, which it did as a blender chops cabbage. I also used it to cut a few six-inch-diameter holes through an old lath-and-plaster ceiling. I chucked in a new hole saw, went up to the second story, reached down between the newly installed floor joists, and began to cut through the first-floor ceiling below. Where my homeowner's drill had labored and whined to spin the huge bit around, and had stalled at the slightest obstruction, the Hole Hawg rotated with the stupid consistency of a spinning planet. When the hole saw seized up, the Hole Hawg spun itself and me around, and crushed one of my hands between the steel pipe handle and a joist, producing a few lacerations, each surrounded by a wide corona of deeply bruised flesh. It also bent the hole saw itself, though not so badly that I couldn't use it. After a few such run-ins, when I got ready to use the Hole Hawg my heart actually began to pound with atavistic terror.

But I never blamed the Hole Hawg; I blamed myself. The Hole Hawg is dangerous because it does exactly what you tell it to. It is not bound by the physical limitations that are inherent in a cheap drill, and neither is it limited by safety interlocks that might be built into a homeowner's product by a liability-conscious manufacturer. The danger lies not in the machine itself but in the user's failure to envision the full consequences of the instructions he gives to it.

A smaller tool is dangerous too, but for a completely different reason: it tries to do what you tell it to, and fails in some way that is unpredictable and almost always undesirable. But the Hole Hawg is like the genie of the ancient fairy tales, who carries out his master's instructions literally and precisely and with unlimited power, often with disastrous, unforeseen consequences.

Pre-Hole Hawg, I used to examine the drill selection in hardware stores with what I thought was a judicious eye, scorning the smaller low-end models and hefting the big expensive ones appreciatively, wishing I could afford one of them babies. Now I view them all with such contempt that I do not even consider them to be real drills--merely scaled-up toys designed to exploit the self-delusional tendencies of soft-handed homeowners who want to believe that they have purchased an actual tool. Their plastic casings, carefully designed and focus-group-tested to convey a feeling of solidity and power, seem disgustingly flimsy and cheap to me, and I am ashamed that I was ever bamboozled into buying such knicknacks.

It is not hard to imagine what the world would look like to someone who had been raised by contractors and who had never used any drill other than a Hole Hawg. Such a person, presented with the best and most expensive hardware-store drill, would not even recognize it as such. He might instead misidentify it as a child's toy, or some kind of motorized screwdriver. If a salesperson or a deluded homeowner referred to it as a drill, he would laugh and tell them that they were mistaken--they simply had their terminology wrong. His interlocutor would go away irritated, and probably feeling rather defensive about his basement full of cheap, dangerous, flashy, colorful tools.

Unix is the Hole Hawg of operating systems, and Unix hackers, like Doug Barnes and the guy in the Dilbert cartoon and many of the other people who populate Silicon Valley, are like contractor's sons who grew up using only Hole Hawgs. They might use Apple/Microsoft OSes to write letters, play video games, or balance their checkbooks, but they cannot really bring themselves to take these operating systems seriously.

24
catwieselSysadmin in extended training•

Brilliant, simply brilliant.thank you for sharing

3
payl0adDigital fire dept.•

but they cannot really bring themselves to take these operating systems seriously.

This is exactly how I feel about any system that is primarily GUI-driven. For the exact reasons your story brought forth. Get out of my head, damn you!

3
XANi_Senior Cat Herder•

And that really begs the question, how the hell could you stop that?

Literally first answer in your link answers that

Do. Not. Do. That.

that file saved my ass more than once.

one of more useful ones is Sync, Umount reBoot.

If everything is fucked up you can still close the system without having to fsck on next boot, do

echo s >/proc/sysrq-trigger 
echo u >/proc/sysrq-trigger

and wait for Emergency Sync complete and (device): re-mounted. then trigger reboot via

echo b >/proc/sysrq-trigger

there are also few other useful commands: https://en.wikipedia.org/wiki/Magic_SysRq_key

19
poooglesmay contain nuts.•

Reboot Even If System Utterly Broken

Is the one that I remember.

3
XANi_Senior Cat Herder•

R redundant if you are going shut down anyway

1
HildartheDorfMore Ops then Dev•

The R is to remap the keyboard iirc, so it bypasses the dodgy misconfigured unstable X server if you are at the actual console (not a remote session).

2
XANi_Senior Cat Herder•

Yes but you dont need that if you are rebooting the machine (and you probably want to kill all processes on current tty if you want to "get back" your access

1
glasspelican•

bash: echo: command not found

-1
perskes•

Debian 8.5 gives me that whenever I forget sudo.

1
XANi_Senior Cat Herder•

echo is bash builtin... same with most other shells

1
digimerOSS HA/Clustering•

In an HA environment, as OP said s/he was in, the node has next to no value. A node in an HA environment should always be easily replaceable.

The goal of the 'echo c' test is to uncleanly crash a system. Another test done in HA is to remove all power to a node without warning. If your HA system (specifically, the fencing part) can't handle these types of failures, then you have bigger problems.

3
XANi_Senior Cat Herder•

In an HA environment, as OP said s/he was in, the node has next to no value

Well OP seemed to be awfully scared by that command

2
IUseRhetoric•

how the hell could you stop that?

With the appropriate user privileges, and by setting appropriate masks in /proc/sys/kernel/sysrq

If you have sudo enabled without a root password, any privileged user could stop a box.

Any privileged user can stop a box period. Don't need to use a sysrq crash for that.

For what it's worth, the sysrq documentation mentions why this exists:

crash(c) can be used to manually trigger a crashdump when the system is hung. Note that this just triggers a crash if there is no dump mechanism available.

15
gachimuchi4•

I'd say

chown -R nobody /

is much worse than yours and is worthy of being the second worst command.

11
frnxt•

I actually saw a new colleague completely break his new OS with a recursive chmod on /. It took us less time to reinstall it entirely than to troubleshoot everything that complained about wrong permissions.

3
perskes•

Did that too when I was youger. "why does my user not own everything, its my PC! And I dont use root anyway, what could happen?"

1
NadiarJack of All Trades/IaaS•

Reboot to single user,

for I in $(rpm -qa); do rpm --setperms $I; rpm --setugids $I; done
3
LarryUnderwood84•

Not bad, but you'r operating under a number of assumptions (1 - fedora based distro; 2 - all of your services/progs have been installed via rpms; 3 - you're able to get your RFS mounted and get a shell on the box that will let you run those command). Also this doesn't fix built-ins. You're going to have to reinstall the OS or revert to a backup image in all likelyhood.

2
NadiarJack of All Trades/IaaS•

debian: apt-get clean && apt-get install --reinstall $(dpkg --get-selections | grep -w 'install$' | cut -f 1)

I've done a challenge lab before where the problem was someone had broken the built-ins, including chmod, and you had to fix it due to "no backup". It's a bit of work, and really involved a bit of trial and error.

But really, this is why you have backups for everything and can just revert back. The point is that rm -rf / is the most dangerous, because you lost everything from last backup to current. Permissions fuckups are annoying, but shouldn't be much more work. As I understood it, OP's point was that someone could randomly cause crashes to random systems, and it would be almost impossible to track down, causing huge numbers of man hours to be wasted troubleshooting the wrong problem.

"mail-1 crashed. Well, after a reboot it seems like its fine. Don't see anything in the logs, I guess I'll have to keep an eye on it" "Oh, now dns-2 crashed... well thats weird... seems like its okay now" "WTF, now nginx-4 crashed, wtf is going on?!!? WHAT CHANGED? Kernel Bug?"

3
LarryUnderwood84•

side note: I love that kind of shit. One of the senior engineers I worked with once walked over, noticed I was logged into one of my lab boxes:

him - "Do you care about that VM?"

me - "meh.. not really."

him - "want to learn something?"

me - "shoot."

He proceeds to run:

sudo chmod -x /usr/bin/chmod

That's a fun exercise to fix :)

5
LarryUnderwood84•

I love this. I would love to run through that lab too. TBH when I ran into that problem on a customer's prod box (had a user fumbling around trying to install java, you can see in the bash history where it goes south, make the decision to chown -R user:user /etc). The ticket read "lost ssh connection" lol. I really wanted to try to rescue the box, but revert was quicker and less impacting (since it was a static content web-node) so I had to go with that option. I still want to go back and try to rescue a system with borked permissions. Definitely a good learning opportunity!

1
debee1jp•

This won't fix everything. Libraries will still remain owned by the other user and things will still be broken.

It might get the server booted, but it will not fix everything.

EDIT:

Found an old ticket of where somebody ran chown -R apache /* and somebody tried to restore with a method similar to this. There were still some issues on the server so I did a find / -user apache and here are a few files that were still owned by that user:

/lib/modules/2.6.32-504.8.1.el6.x86_64
/lib/modules/2.6.32-504.8.1.el6.x86_64/extra

Funny thing is, this was a huge customer that I am sure plenty of people in r/sysadmin deal with on a daily basis. If they knew who it was it would almost surely change their opinion of them.

1
LarryUnderwood84•

This will do the trick by subbing pretty much any user. It turns out the kernel really likes certain things to run with UID 0...

Recursive chown's and chmod's scare me... Stick those in a 1-liner and you have a recipe for reverting to last nights backup image.

1
mlts22•

I can top those. On a SSD based system:

blkdiscard -v /dev/sda

A few milliseconds later, you have a completely blank drive.

I actually do this as a matter of prepping a new machine. It marks any data on there as free for garbage collection to go destroy, and it clears out all pointers.

11
RedShift9•

Ooh I didn't even knew this was possible. I always wonder if garbage collection and trimming still works as it should if you format it, this command solves those worries :-D . And at the right time too, I was just about to upgrade my laptop.

1
mlts22•

You might be able to use the "-s" option with blkdiscard as well. This guarantees that the data is gone by telling the SSD controller to shred all free pages. Once that is done, the data is history.

2
RedShift9•

Tried it with "-s" but it reported the command was not supported. Crucial MX100 SSD.

1
verysadverylonely•

And that really begs the question,

  Agh, no it doesn't. begthequestion.info

7
fp4•

And that really begs the question, how the hell could you stop that?

Not giving complete unrestricted access to the box in question is a good start.

6
LarryUnderwood84•

boom. +1

1
TargrenLinux Admin•

I guess that makes "visudo" the 1st most scary linux command?

If you have sudo enabled without a root password, any privileged user could stop a box.

They could do that with or without sysrq - it's one of those "root" things. Don't hand it out like Halloween candy.

6
LarryUnderwood84•

Sudo echo "some shit" >> /etc/sudoers

scares me more than visudo. At least someone using visudo has the courtesy to have something check their syntax before they leave the box completely borked...

0
TargrenLinux Admin•

Sudo chokes on straight redirection like that. :)

1
LarryUnderwood84•

lol which is why we have visudo :p

Also, I have a case error at the beginning anyways so that train was derailed before leaving the station...

1
Nye•

Sudo chokes on straight redirection like that.

Sudo has nothing to do with it. It was told to execute the command 'echo "some shit"' as UID 0, and it does so, job done.

Once it's finished, you then try to redirect the output of that command to a file that your standard user account can't write to, and of course you can't.

A fundamental point of pipelines is that each section is its own command, independent of any others to which it it plumbed. This allows you to do things like

spam | sudo eggs | ham

or

spam | ssh foo eggs | ham

and having sudo or ssh in one section of the pipeline has no effect on any other sections - it can safely be used transparently.

0
TargrenLinux Admin•

So you're the guy who comes across a bunch of other guys busting each others' chops, and has to jump into a lecture about just why it is, in fact, biologically impossible for a human male to be the offspring of a canine female, right?

1
Bonn93•

Installing java is pretty bad...

6
[deleted]•

[deleted]

4
debee1jp•

This is how I program.

2
icebox1701•

For me, one of the worst was:

[oracle@centos7 ~]$ /stage/database/runInstaller  
Starting Oracle Universal Installer...
4
demosthenexIndependent Systems Integrator•

Even better.

mount /dev/sdX1 /mnt
dd if=/dev/zero of=/dev/sdX1

Linux doesn't implement device locking. So you can zero out your storage while it's mounted. Funny that the biggest complaint I found on this wasn't that it's possible, but that desktop environments would disrupt ongoing CDROM burns trying to read the drive while it was being written and causing a coaster.

3
Enxer•

My favorite was in the 90's if you sent more data than the CD could hold (aka overburning) Linux|Unix would append the data to the root of the mount point - so if /mnt/ was in /'s partition you'd start overwriting files. :/

So many OSs & CDs fubarred with overburning...

3
XANi_Senior Cat Herder•

Linux doesn't implement device locking.

That is incorrect. normal filesystem commands just dont care about the locks

1
demosthenexIndependent Systems Integrator•

Care to elaborate? Also the file system commands do have a locking, mkfs will check for mounted and refuse. That is a protection built into those tools, not the device.

1
XANi_Senior Cat Herder•

By "do not care" I've meant that normal file-level tools (dd, mv, rm etc) do not check if file or device behind the file is locked in any way.

That way (and the way linux fs layer works) you can delete opened file (as opposed to windows where that will give an error) but app using it will still be able to use it and it will only be deallocated once every app closes it (some apps use that for temporary files)

Linux itself has device locking, for example try to remove mounted CD/DVD... the "eject" button on hardware will literally stop working. Or when you "remove" mounted device (like pendrive), its drive letter will stay in system but unlocked (not mounted anywhere) one will disappear (which also allows to queue requests and wait for say iSCSI to reconnect instead of dropping data). And the fs-level tools just check for that

1
demosthenexIndependent Systems Integrator•

Regarding "do not care", they don't care because Linux doesn't lock devices.

Deleting an open file is a common thing on all POSIX platforms. I've worked with many applications that do that for temporary files to make cleanup easy. This has nothing to do with device locking.

The last example you provide is not a device lock. The eject is the driver telling the button to be disabled, but it doesn't prevent multiple processes reading/writing the DVD. If you remove a mounted pendrive, the driver letters etc are still in use and in an error state. They are still reserved, but this is also not a lock.

The FS level tools do some of their own checking and safeguards which is admirable, but is only effective for filesystems.

Let's be clear about the definition. A kernel layer device lock means that once a process is using a device it has exclusive access, even root processes aren't allowed to bypass it. With a lock any other process would receive an error if it tried to read or write an already opened device. This is a safety feature in other platforms and I've only pointed out that it isn't available on Linux and that's scary.

I've met admins who were trying to format a USB key, or setup a new partition and wiped something already mounted. Sure, it's a typo. Maybe they were confused because of the really poor device naming scheme. But a device lock like other platforms provide would have prevented the problem in the first place.

1
goozbachinfrastructure consultant•

to disable: http://tldp.org/HOWTO/Remote-Serial-Console-HOWTO/security-sysrq.html

3
kurinalias ok="git commit -a -m stuff"•

Well, it works. And that really begs the question, how the hell could you stop that?

Well, that file is only writable by root, so.

3
davethebarbLinux Admin•

I'd say that root kits and anything sneaking data out of your system is much, much worse. If someone looking to do something malcious is already on your server with root, or sudo, then you're already done.

3
tcpip4lyfeNetwork Engineer•

puts a script to run this in /etc/init.d to run it on boot or in cron.

Mental note for when my coworkers piss me off.

2
Flakmaster92•

Booting to single user would stop it, as would remapping init to bash

1
planiverse•

The thing that makes rm -rf / scary isn't what it does. It's how easy it is to fat finger.

**rm -rf /**home/user1

If you accidentally hit enter after the bold, it's over. And it's not bad practice to use absolute paths. rm -rf ./*? Well . . . You were on the wrong server session, and accidentally removed /var/www/html. Oops.

Who would write files in /proc via bash for routine maintenance?

2
icebox1701•

I make it a habbit of first typing the path and then returning to add -rf and/or sudo exactly because of this;

Stopping and taking a look at your PS1 (which I always have set as user@machine path ) before pressing enter is also a habbit I acquired after I shutdown a production server while typing poweroff in the wrong terminal (wanting to shut down my laptop and go home and thus ensuring I stay at work a bit more :) )

1
Zaphod_Bchown -R us ~/.base•

rm -rfi /path/to/whatever will prompt you, and really if you need to actively delete sub directories and files there are better ways to do it.

1
Nye•

-f and -i mean literally the opposite thing. Passing them both at the same time is a pretty good warning sign that the user should probably spend more time learning this powerful tool before using it.

2
Zaphod_Bchown -R us ~/.base•

I just ran rm -rfi and it prompted me. The -f is force and the -i is interactive. You can totally run them in the same command, guess I am not following what you mean by total opposite thing?

1
Nye•

By using '-i', you have disabled '-f'^(1). That is, you've given it the order "don't prompt" followed by "prompt". '-i' is the default, and '-f' turns it off, only to have it turned straight back on again.

^(1) For the sake of technical correctness, there is a slight exception in that if you specify a file by name and that file doesn't exist, then '-f' will prevent rm from showing an error about it; aside from that, 'rm -rfi' is simply 'rm -r'.

^(Edit: actually, I've just tested it, and even this part of '-f' is disabled by '-i', so using '-i' does literally disable '-f' entirely.)

1
lowermiddleclass•

Here's a cute one I recently discovered:

fsfreeze -f /some_mountpoint_here

Stops all I/O to that filesystem dead in its tracks.

2
brontideHere's a nickle kid, get yourself a real computer•

Great for taking snapshots, just make triple sure to undo it!

2