Rebooting with systemd wedged

[Bob Proulx]

Recently emacsconfmedia0p reported system package upgrade errors.
Looking into the problem the root cause seemed to be systemd having
gotten wedged.

    root@emacsconfmedia0p:~# systemctl status
    Failed to read server status: Failed to activate service 
'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)

The system was not responsive.  I suspect it was accumulating zombies
if systemd is stuck.

    root@emacsconfmedia0p:~# systemctl reboot
    Failed to read server status: Failed to activate service 
'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)

And of course shutdown and reboot both call systemctl now.  Could not
reboot it normally it since systemd is in charge of the reboot and
systemd is not responding.  Isn't systemd just such a wonderful
system?

To work around this I used the Linux Magic SysRq system to sync the
system and boot the kernel directly.  I have been looking for ways to
do this since systemd has a habit of getting stuck on the down side of
a reboot.  And if the system does not do the shut down part then it
can't do the boot up part.  The SysRq interface worked swimmingly.

    root@emacsconfmedia0p:~# echo 1 >/proc/sys/kernel/sysrq
    root@emacsconfmedia0p:~# echo s >/proc/sysrq-trigger
    Apr 29 12:39:00 emacsconfmedia0p kernel: sysrq: Emergency Sync
    root@emacsconfmedia0p:~# echo s >/proc/sysrq-trigger
    Apr 29 12:39:57 emacsconfmedia0p kernel: sysrq: Emergency Sync
    root@emacsconfmedia0p:~# echo b >/proc/sysrq-trigger
    [2593154.257181] sysrq: Resetting
    ...

The first enables the interface.  Because system usually disables most
of the features.  (I don't know why.)  Writing a 1 to sysrq enables
all of the features all at once.  Other values as a bitmask will
individually enable individual features.  But here it is simpler to
enable everything.

Then I triggered the file system sync.  Traditionally one does this
three times before rebooting.  Once is probably enough.  I did two out
of paranoia and a nod to tradition.

Then the last triggers the kernel to reboot immediately.  This avoids
the systemd shutdown entirely.  Worked perfectly.

Then having shutdown the system booted normally.  This is definitely
going to be my new go-to action to reach for when I want to boot
misbehaving systemd systems from now moving forward.  Maybe I will
install it as a script reboot-sysrq for ease of use.

    #!/bin/sh
    echo 1 >/proc/sys/kernel/sysrq
    echo s >/proc/sysrq-trigger
    echo b >/proc/sysrq-trigger

Bob