August 31, 2023

Fedora 38 -- Bug! Won't boot, initrd-switch-root

I am now running the 6.4.10 kernel. I have received updates with the 6.4.11 and 6.4.12 kernels. Both have the bug and won't boot.

I get dropped into emergency mode. The message tells me that initrd-switch-root.service has failed and that /sysroot is missing (or something like that). It also coaches me to copy /run/initramfs/rdsoreport.txt to /boot and submit it with a bug report.

To do that you must mount (in my case) /dev/sda1, then copy it there, so I did, and I filed a bug report as directed.

I first saw this with the 6.4.11 kernel and hoped that a subsequent kernel would fix it. Now 6.4.12 has appeared and has the same issue. So I boot back to 6.4.10 and am now running it while I type this and research the problem.

One recommended fix I find online is to run: 

# cp /boot/initramfs-*.x86_64.img /root
dracut --force --regenerate-all

This page discusses the issue in some detail (in the context of RHEL7). It recommends the above as a fix, but also recommends backing up the ramfs image as shown in the comment above.

Let's try it

Since Fedora keeps only 3 kernels, I may lose my working 6.4.10 kernel when another update rolls in. So while I have a working system, I figure I will test fly some of the above. 
su
cd /root
mkdir initramfs
cd initramfs
cp /boot/initramfs* .
dracut --force --regenerate-all
The regenerate step takes a very long time (with no encouraging console output). And it is not clear to me if running it while running the 6.4.10 kernel will fix things for the 6.4.12 kernel, but away we go.

Looking at /boot, I see initramfs appearing for all kinds of ancient kernels from 5.0.6 and on -- don't ask me what is triggering that mess. Also note that "dracut" is in /bin and is a bash script.

It finally finishes and indeed the initramfs for the 6.4.* kernels now have new timestamps. Time to reboot and see if 6.4.12 will boot now.

This does not fix a thing. I still get the same problem booting 6.4.12. And it is not possible to run dracut from emergency mode (there is no dracut -- there is a collection of dracut-* scripts but trying dracut-cmdline did not do the job).

I type "dnf update" to ensure that my system is up to date, and it is. There was some talk about versions of systemd being at the root of this problem.

Go into a holding pattern

The 6.4.10 kernel works just fine, so I can run that more or less indefinitely. So what I would like to do is either disable kernel updates while I wait for the Fedora gang to wake up and fix this mess, or allow more than just 3 updates to pile up.

Thinking of more than 3 updates gets me to wondering about how much space I have in /boot. Lo and behold my /boot is all full -- no doubt due to dracut generating stupid initramfs images for every kernel known to mankind. I do this: 

su
cd /boot
rm iniframfs-5*
After this I see my /boot partition is 28 percent full. So I could probably accomodate at least 8 kernels

Edit /etc/dnf/dnf.conf and change this line: 

installonly_limit=8
Note that this pertains to all packages, not just kernels. A person could set 0 to have an unlimited number, but this seems unwise.

Fedora 39

This is due in a month or two (sometime in October, 2023) and I have hope (perhaps ill founded) that this bug will be fixed there. We shall see.
Have any comments? Questions? Drop me a line!

Adventures in Computing / tom@mmto.org