A few days ago, while I was doing an Oracle RMAN duplicate operation to set up another database on a Oracle Linux 6.10 system running on VMWare, the sytem went into a kernel panic state with the following messages:

kernel:[217869.366586] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 223s! [sadc:32354]
kernel:[217869.366598] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 223s! [master:3354]
kernel:[217869.366626] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 223s! [vami_login:3427]
kernel:[217899.737240] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [swapper/6:0]

I lost my SSH connection and when I sshed back, nothing really worked. I am getting Input/output error for most of commands I issued. I could not check the system logs.

-bash-4.1# uptime
-bash: /usr/bin/uptime: Input/output error
-bash-4.1# cat /var/log/messages
cat: /var/log/messages: Input/output error
-bash-4.1# reboot
-bash: /usr/bin/reboot: Input/output error
-bash-4.1# which reboot
Segmentation fault
-bash-4.1# shutdown -r now
-bash: /sbin/shutdown: Input/output error

So no matter what the issue was — like kernel panic caused disk failure, I needed to reboot the system. The only way would be using the “magic SysRq key” which sends commands directly to the kernel through the /proc filesystem. It is enabled via a kernel compile time option, CONFIG_MAGIC_SYSRQ, which is likely available on most Linux distributions. First activate the magic SysRq option with the following command:

echo 1 > /proc/sys/kernel/sysrq

reboot the machine simply run the following:

echo b > /proc/sysrq-trigger

Note this does not attempt to unmount or sync filesystems, so it should be used only when the system is in the situation as I ran into.

You can also shutdown a system with the following command:

echo o > /proc/sysrq-trigger

• Linux