Networking not coming up during boot

[Beat Kuster]

Dear Fedora CoreOS Community

I did have FCOS installed on multiple Intel NUCs for a long time and was always very happy. Being on board since CentOS Atomic Host, Fedora CoreOS really is an outstandig product, thank you so much for making this possible.

Now to the issue:
My current system (NUC8i5BEK) has a Zincati maintenance window defined and after updates reboots automatically during this time window. Unfortunately, since release 35.20220116.3.0, it seems that networking does not come up reliably after a reboot. The boot process gets stuck and the system does not request an IP from the DHCP. (Somehow sounded a bit like https://github.com/coreos/fedora-coreos-tracker/issues/1066 but my network driver "e1000e" is different)

I identified two workarounds:
- physically access the system and select at boot loader menu the last working release (~rollback)
- reboot multiple times, sometimes networking magically comes up

Last working release: 35.20220103.3.0 -> please note that this release always boots the first time even as of today
Not working releases: 35.20220116.3.0, 35.20220131.3.0, 35.20220213.3.0

Could you please share any guidance how I could proceed? Provide more technical details here in the mailing list? Or opening a bug at https://github.com/coreos/fedora-coreos-tracker?

Thanks in advance for any feedback,
Beat

[Dusty Mabe]

On 3/8/22 13:05, Beat Kuster wrote:
> Dear Fedora CoreOS Community

Hi Beat!
> I did have FCOS installed on multiple Intel NUCs for a long time and was always very happy. Being on board since CentOS Atomic Host, Fedora CoreOS really is an outstandig product, thank you so much for making this possible.

Thank you! We love to hear stories from happy users.
> Now to the issue:
> My current system (NUC8i5BEK) has a Zincati maintenance window defined and after updates reboots automatically during this time window. Unfortunately, since release 35.20220116.3.0, it seems that networking does not come up reliably after a reboot. The boot process gets stuck and the system does not request an IP from the DHCP. (Somehow sounded a bit like https://github.com/coreos/fedora-coreos-tracker/issues/1066 but my network driver "e1000e" is different)
>
> I identified two workarounds:
> physically access the system and select at boot loader menu the last working release (~rollback)
> reboot multiple times, sometimes networking magically comes up
>
> Last working release: 35.20220103.3.0 -> please note that this release always boots the first time even as of today
> Not working releases: 35.20220116.3.0, 35.20220131.3.0, 35.20220213.3.0
>
> Could you please share any guidance how I could proceed? Provide more technical details here in the mailing list? Or opening a bug at https://github.com/coreos/fedora-coreos-tracker?

Interesting. I'd definitely say open a bug over in the issue tracker. If we could get the logs from a failed boot that would be ideal. Being that you're not getting the network that makes things hard. Could you maybe set a password so you can log in on the console of the machine and for a failed boot you can at least run `sudo journalctl -b0 > failed-boot-journal.txt` and then share that back with us.
> Thanks in advance for any feedback,
> Beat

Thank you!

[Beat Kuster]

Hi Dusty

Thanks for your fast reply!

I'm not sure if I understand your proposition correctly. I do have physical access to the system but when the boot process stalls, it does not reach the login prompt. So even if I would set a password for my account, I never get the chance to login.

However, I just reproduced the boot process stall now (Wed 2022-03-09 around 20:53 UTC) and then forced physical poweroff after 5min of hanging there. After booting the working release (35.20220103.3.0) I checked the logged boots with $ journalctl --list-boots:
-5 05ea07dd8f6d437a966c7296de773fe3 Mon 2022-02-21 20:42:08 UTC—Tue 2022-02-22 02:00:24 UTC
-4 94ec3894d8b54af98ab677bc14b62d77 Tue 2022-02-22 17:37:04 UTC—Tue 2022-02-22 17:44:57 UTC
-3 c653f3e3815a470485af6d0fbc1ab728 Tue 2022-02-22 18:06:06 UTC—Tue 2022-02-22 18:07:08 UTC
-2 90c02bf939964260ba78960196601484 Tue 2022-02-22 18:08:24 UTC—Tue 2022-03-08 02:00:59 UTC
-1 caa89f6a944d4b4b9d85839ff1664796 Tue 2022-03-08 17:18:50 UTC—Wed 2022-03-09 20:47:01 UTC
 0 143ab2c95037459b8458b8bca259132b Wed 2022-03-09 21:00:59 UTC—Wed 2022-03-09 21:02:07 UTC

Unfortunately, as you can see, the stalled boot process around 20:53 does not show up. And as expected, checking boot 0 does not show anything strange because it brought up the system correctly and I could SSH in. Maybe the physical poweroff is preventing journalctl to persist the logs?

Any other suggestions on how to generate any meaningful logs for a bug report?

Thanks in advance,
Beat

[Dusty Mabe]

On 3/9/22 16:20, Beat Kuster wrote:
> Hi Dusty
>
> Thanks for your fast reply!
>
> I'm not sure if I understand your proposition correctly. I do have physical access to the system but when the boot process stalls, it does not reach the login prompt. So even if I would set a password for my account, I never get the chance to login.

Ahh. I thought the machine was just not getting networking, but otherwise was OK, which means you would have been able to log in on the console.
> However, I just reproduced the boot process stall now (Wed 2022-03-09 around 20:53 UTC) and then forced physical poweroff after 5min of hanging there. After booting the working release (35.20220103.3.0) I checked the logged boots with $ journalctl --list-boots:
> -5 05ea07dd8f6d437a966c7296de773fe3 Mon 2022-02-21 20:42:08 UTC—Tue 2022-02-22 02:00:24 UTC
> -4 94ec3894d8b54af98ab677bc14b62d77 Tue 2022-02-22 17:37:04 UTC—Tue 2022-02-22 17:44:57 UTC
> -3 c653f3e3815a470485af6d0fbc1ab728 Tue 2022-02-22 18:06:06 UTC—Tue 2022-02-22 18:07:08 UTC
> -2 90c02bf939964260ba78960196601484 Tue 2022-02-22 18:08:24 UTC—Tue 2022-03-08 02:00:59 UTC
> -1 caa89f6a944d4b4b9d85839ff1664796 Tue 2022-03-08 17:18:50 UTC—Wed 2022-03-09 20:47:01 UTC
>  0 143ab2c95037459b8458b8bca259132b Wed 2022-03-09 21:00:59 UTC—Wed 2022-03-09 21:02:07 UTC
>
> Unfortunately, as you can see, the stalled boot process around 20:53 does not show up. And as expected, checking boot 0 does not show anything strange because it brought up the system correctly and I could SSH in. Maybe the physical poweroff is preventing journalctl to persist the logs?

It depends on how far along in the boot process the system got before it hangs.
> Any other suggestions on how to generate any meaningful logs for a bug report?

Unfortunately without setting up a serial console on the machine so you can scrollback and copy/paste your only real options is to stare at what scrolls across the screen and try to derive any clues that way. Maybe recording a video on of the screen would help?

Dusty

[Beat Kuster]

Hi Dusty

I already made a series of pictures during such a stalled boot. Unfortunately, all of the last messages are concerning the graphics driver which I suspect are only there when I attach my screen to "debug". As the boot process stalls without screen as well, I think the culprit must be something else i.e. earlier messages.

In order to rule out anything related to this specific piece of hardware, I ordered a new Intel NUC 11 (NUC11TNKv5). As soon as I receive the new device, I'll deploy my current Ignition config and let you know if I'm able to reproduce this behaviour.

Additional benefit of the new device is Intel vPro functionality which might give me console access without FCOS bringing networking up, not sure though...

Many thanks for your thoughts so far, much appreciated.
Beat

[Beat Kuster]

Hi Dusty

I received the new NUC hardware (NUC11TNKv5) and started my tests, deploying the Ignition config from the other NUC, etc.

Interestingly enough, the older NUC (NUC8i5BEK) started to work again normally with release 35.20220227.3.0. I first thought it was just lucky as the stalled boots happened not in 100% of the cases but also the following release (and it's subsequent Zincati triggered automatic reboot) did work without hiccups.

To sum up, this leaves me with following updated working/non-working table:
Working release: everything before 35.20220103.3.0
Not working releases: 35.20220116.3.0, 35.20220131.3.0, 35.20220213.3.0
Releases working again: 35.20220227.3.0, 35.20220313.3.1

I hope it stays this way and therefore consider this case closed. On the bright side; the new NUC hardware really did came with Intel AMT and Serial-over-Lan, so I can give more detailed error descriptions next time...

Thanks again for your thoughts,
Cheers Beat

[Dusty Mabe]

On 4/10/22 14:42, Beat Kuster wrote:
> Hi Dusty
>
> I received the new NUC hardware (NUC11TNKv5) and started my tests, deploying the Ignition config from the other NUC, etc.
>
> Interestingly enough, the older NUC (NUC8i5BEK) started to work again normally with release 35.20220227.3.0. I first thought it was just lucky as the stalled boots happened not in 100% of the cases but also the following release (and it's subsequent Zincati triggered automatic reboot) did work without hiccups.
>
> To sum up, this leaves me with following updated working/non-working table:
> Working release: everything before 35.20220103.3.0
> Not working releases: 35.20220116.3.0, 35.20220131.3.0, 35.20220213.3.0
> Releases working again: 35.20220227.3.0, 35.20220313.3.1
>
> I hope it stays this way and therefore consider this case closed. On the bright side; the new NUC hardware really did came with Intel AMT and Serial-over-Lan, so I can give more detailed error descriptions next time...

Hmm. Maybe some kernel issue that regressed and then got fixed. I hope it stays fixed too!

Thanks for the update!
> Thanks again for your thoughts,
> Cheers Beat