Skip to main content

DigitalOcean droplet: Repair broken Ubuntu with recovery ISO

Recently, I needed to repair a Ubuntu droplet server that was stuck in a kernel panic. The server maintainer decided to upgrade the OS from Ubuntu 16.04 to 18.04. Everything went well until the final restart after the upgrade when the dorplet was stuck during boot. The maintainer could not SSH into the system and the website was down as well. In the DO control centre, CPU usage of the droplet was showing 100% and the console was showing the kernel panic state and the system calls that caused it.

I searched around and found this post that had a similar problem and solved it. Basically, these are the steps:
  1. Boot into the recovery ISO
  2. Mount and chroot into the existing filesystem
  3. Reinstall the kernel
From the console commands, it was obvious that the solution was for REHL based distributions and I will need to use 'apt' instead of 'yum' for Ubuntu. I found the appropriate command after a bit of digging:

apt-get install --reinstall linux-image-generic linux-image

However, when I entered the command during Step 3, I got an error from /usr/sbin/grub-probe that it "failed to get canonical path of overlay".

I used the 'df' command to list mounted filesystems and realised that, the live CD ISO mounts overlay at the root i.e. '/', but it was not mounted when I chroot into the broken system.

After a few more searches, I found some usefule info in  here and here that says I need to bind mount '/dev', '/proc' and '/sys' before chroot, so that they can be found by applications. I did that and it all worked out! This time, the reinstall was completed successfully without any error (though there were a few warnings and fallbacks).

In summary, the following steps were performed:
  1. Power off the droplet and boot from the recovery ISO
  2. Choose 6 for interactive shell (don't mount or chroot)
  3. In the interactive shell,
    1. Mount the filesystem: "mount /dev/vda1 /mnt"
    2. Bind mount /dev: "mount --bind /dev /mnt/dev
    3. Bind mount /proc: "mount --bind /proc /mnt/proc"
    4. Bind mount /sys: "mount --bind /sys /mnt/sys"
    5. Chroot: "chroot /mnt"
    6. Update and upgrade: "apt update" and "apt upgrade"
  4. Exit the interactive shell and restart the droplet to boot from main harddisk
Note that when using "apt update", previously broken installation automatically restarted.

Comments

  1. Paint Drinking Pete6 December 2019 at 15:09

    THANK YOU!

    ReplyDelete
  2. elle25 March 2020 at 07:36

    Can you reprint your article

    ReplyDelete
  3. alfizo28 June 2020 at 21:31

    Thank you very much i could kiss you... but we social distancing. I had an issue with apt update it wouldnt resolve host names, I had to exit chroot and run "sudo mount -o bind /run /mnt/run" then get back to chroot, and it worked like a bomb... thanks.

    ReplyDelete
  4. newbie1513 December 2020 at 15:49

    thank you so much. let me spred this article on my facebook.

    ReplyDelete
  5. ruiz31 January 2021 at 23:27

    if the system cannot resolve any hostname, make sure to edit /etc/resolv.conf properly by adding "nameserver 8.8.8.8". Likely will face the issue when "sudo apt update". Thanks.

    ReplyDelete
  6. CripyIce21 February 2021 at 23:43

    Thank you so much!!!
    You save my life with this!

    ReplyDelete
  7. Mojmir17 October 2022 at 00:20

    It's 1:15 AM and your post just saved me. I can't thank you enough.

    ReplyDelete

Post a Comment

Popular posts from this blog

Set up Zotero Attachment File Synchronisation with Google Drive

Background I had a ton of downloaded PDF papers. These files were unorganised and kept in a folder in my Google Drive. I learned about Zotero from a colleague and started using it. I was very impressed at the begining for its convenience and ease of use. Just drag a PDF file into Zotero , and select an option to retreive metadata from the PDF and viola! A proper citation record is ready! I can export it to bibtex format in order to use the citation in Latex, or include it in MS Word in my preferred citation style. The most exciting feature of Zotero for me was the ability to take notes on a paper and keep them organised. I can search for a paper/topic, and open a relevant paper PDF from zotero. I can write comments on that PDF or take notes and everything will be synced to the server. However, soon I began to notice some issues. Zotero didn't use the downloaded PDF files, but included a copy of them instead. These copies were being synced to the Zotero server and soon filled my 3...
Post a Comment
Read more

Finding Optimal PDF Paper Size for Kobo Clara HD Ereader

tl;dr: Optimal page layout for Kobo Clara HD ereader: Width: 9cm, Height: 11.1cm, margins: 0.5cm. **** I have recently bought a Kobo Clara HD ereader. I used to own a Glo HD ereader by the same company, until I accidentally pushed it off the table causing its display to crack. Though I prefer EPUB format for reading on an ereader, there are many cases when a PDF format might be more convenient. However, regular PDF files are difficult to read on the ereader for their large page sizes. The PDF viewer of the device scales a page to fit its display causing tiny texts. I can zoom in a section to read, however I need to swipe and drag around to view other areas which is really not convenient due to slow response time of e-ink display. Worse is when I accidentlly tap the display, causing it  to immediately turn the next page. I need to go back to previous page and go through all the zooming and dragging before I can resume reading. Therefore I always wanted to generate a PDF optim...
1 comment
Read more