This is something that seems to happen randomly to my little RaspberryPi 4B homeserver, like once every few months maybe.

What happens is that out of nowhere (meaning I didn’t make any change to config or installation) I cannot reach any services I host on it (i.e. Jellyfin, Anchor notes…). I cannot access it via SSH at all, connection keeps timing out.

Ping however gives a signal. I can see/hear its running and doing something, since I hear the external HDD working. It is running headless and I have no monitor and/or keyboard I could connect.

So far my only way out is to just switch its power off, which is obviously not good for the hard drive. It has happened now like half a dozen times over the past few years, so I am wondering what a better way is to handle this. This was on RPI OS and now its happening on DietPi, so seems not related to the OS version.

I tried sending a shutdown signal via SSH but that also just times out, so how do I get it to shutdown properly?

Secondly, what kind of logs could I look at or start collecting to make sense of this?

  • farcaller@fstab.shEnglish
    1·
    3 hours ago

    systemd will try to save any kernel panics in pstore via systemd-pstore, if it’s enabled in your kernel. I’d check /var/lib/systemd/pstore and see if anything in there on the next boot.

    Can you elaborate on ping “running”? Do you get actual icmp replies coming back? Because there’s no code path I can imagine where a ping would cause hdd activity (on a normally running system).

    If ssh times out (and you don’t do anything fancy with the firewall), then it’s not sshd dead, it’s sshd not being able to respond. Grab a tcpdump for dst port 22 from your local machine while RPi is stuck and see if you get any replies whatsoever or it’s just retransmits going into the void.

    My first rough suspicion would be ram abuse. Something eats up all ram and the system locks up and semi-dies. Pstore would have OOMs. You could run a local script for telemetry recording too to see if ram use spikes up before the system gets unresponsive.

  • confusedpuppy@lemmy.dbzer0.comEnglish
    1·
    9 hours ago

    It sounds to me like ssdh may have stopped working. That may explain why you can’t ssh into your server but pings still respond. I have a Raspberry Pi4 and a Pi5 and have had similar issues in the past.

    I would probably approach this issue by writing a small script that checks every so often if the process sshd is still alive and if not restart sshd. Maybe SystemD can so something similar but I am not familiar with SystemD.

    Edit: A quick and simple script looks like this

    #!/bin/sh

# Check if `sshd` process is running; If not running, `pgrep` returns
# an exit status of '1' and restarts `sshd`
pgrep 'sshd' > /dev/null || systemctl restart sshd

    Make the script executable with chmod +x /home/user_name/sshd_check

    Add the following line to /etc/crontabs/root to run the script every 15 minutes

     */15    *       *       *       *       /home/user_name/sshd-check

    I don’t use SystemD but I am pretty sure systemctl restart sshd is correct, otherwise it can be changed to whatever your operating system uses to control services

    • gedfromgont@piefed.caOPEnglish
      1·
      5 hours ago

      Thanks, I’ll try this. What speaks against this is that all my services turn unreachable, and those do not rely on ssh access. But still worth a try.