DigitalOcean droplet: Repair broken Ubuntu with recovery ISO

Recently, I needed to repair a Ubuntu droplet server that was stuck in a kernel panic. The server maintainer decided to upgrade the OS from Ubuntu 16.04 to 18.04. Everything went well until the final restart after the upgrade when the dorplet was stuck during boot. The maintainer could not SSH into the system and the website was down as well. In the DO control centre, CPU usage of the droplet was showing 100% and the console was showing the kernel panic state and the system calls that caused it.

I searched around and found this post that had a similar problem and solved it. Basically, these are the steps:

1. Boot into the recovery ISO
2. Mount and chroot into the existing filesystem
3. Reinstall the kernel

From the console commands, it was obvious that the solution was for REHL based distributions and I will need to use 'apt' instead of 'yum' for Ubuntu. I found the appropriate command after a bit of digging:

apt-get install --reinstall linux-image-generic linux-image

However, when I entered the command during Step 3, I got an error from /usr/sbin/grub-probe that it "failed to get canonical path of overlay".

I used the 'df' command to list mounted filesystems and realised that, the live CD ISO mounts overlay at the root i.e. '/', but it was not mounted when I chroot into the broken system.

After a few more searches, I found some usefule info in  here and here that says I need to bind mount '/dev', '/proc' and '/sys' before chroot, so that they can be found by applications. I did that and it all worked out! This time, the reinstall was completed successfully without any error (though there were a few warnings and fallbacks).

In summary, the following steps were performed:

1. Power off the droplet and boot from the recovery ISO
2. Choose 6 for interactive shell (don't mount or chroot)
3. In the interactive shell,
   1. Mount the filesystem: "mount /dev/vda1 /mnt"
   2. Bind mount /dev: "mount --bind /dev /mnt/dev
   3. Bind mount /proc: "mount --bind /proc /mnt/proc"
   4. Bind mount /sys: "mount --bind /sys /mnt/sys"
   5. Chroot: "chroot /mnt"
   6. Update and upgrade: "apt update" and "apt upgrade"
4. Exit the interactive shell and restart the droplet to boot from main harddisk

Note that when using "apt update", previously broken installation automatically restarted.