f34 on J1900D2Y (quad-core Celeron) mobo: kernel <= 5.12.15, OK; kernel >= 5.12.17, 5.13.4, slow boot (>> 660 secs) until FAIL/hang ?

[PGNet Dev]

My servers run Fedora 34, with latest kernel.

On just one server , an old ASRockRack J1900D2Y (quad-core Celeron) motherboard build,

hwinfo --bios
    	01: None 00.0: 10105 BIOS
    	  [Created at bios.186]
    	  Unique ID: rdCR.lZF+r4EgHp4
    	  Hardware Class: bios
    	  BIOS Keyboard LED Status:
    	    Scroll Lock: off
    	    Num Lock: off
    	    Caps Lock: off
    	  Serial Port 0: 0x3f8
    	  Base Memory: 626 kB
    	  PnP BIOS: @@@0000
    	  MP spec rev 1.4 info:
    	    OEM id: "A M I"
    	    Product id: "ALASKA"
    	    4 CPUs (0 disabled)
    	  SMBIOS Version: 2.8
    	  BIOS Info: #0
    	    Vendor: "American Megatrends Inc."
    	    Version: "P1.10"
    	    Date: "01/23/2015"
    	    Start Address: 0xf0000
    	    ROM Size: 8192 kB
    	    Features: 0x0d03000000013f8b9880
    	      PCI supported
    	      BIOS flashable
    	      BIOS shadowing allowed
    	      CD boot supported
    	      Selectable boot supported
    	      BIOS ROM socketed
    	      EDD spec supported
    	      1.2MB Floppy supported
    	      720kB Floppy supported
    	      2.88MB Floppy supported
    	      Print Screen supported
    	      8042 Keyboard Services supported
    	      Serial Services supported
    	      Printer Services supported
    	      ACPI supported
    	      USB Legacy supported
    	      BIOS Boot Spec supported
    	  System Info: #1
    	    Manufacturer: "To Be Filled By O.E.M."
    	    Product: "To Be Filled By O.E.M."
    	    Version: "To Be Filled By O.E.M."
    	    Serial: "To Be Filled By O.E.M."
    	    UUID: 03000200-0400-0500-0006-000700080009
    	    Wake-up: 0x06 (Power Switch)
    	  Board Info: #2
    	    Manufacturer: "ASRock"
    	    Product: "J1900D2Y"
    	    Type: 0x0a (Motherboard)
    	    Features: 0x09
    	      Hosting Board
    	      Replaceable
    	    Chassis: #3
    	...

with Fedora34 Kernel <=

Fedora (5.12.15-300.fc34.x86_64) 34 (Thirty Four)

the box boots/runs OK,

[    0.000000] microcode: microcode updated early to revision 0x838, date = 2019-04-22
    [    0.000000] Linux version 5.12.15-300.fc34.x86_64 ([e-mail]) (gcc (GCC) 11.1.1 20210531 (Red Hat 11.1.1-3), GNU ld version 2.35.1-41.fc34) #1 SMP Wed Jul 7 19:46:50 UTC 2021
    [    0.000000] Command line: BOOT_IMAGE=(mduuid/5687b25f8f25671a243a83cb661f7841)/vmlinuz-5.12.15-300.fc34.x86_64 root=/dev/mapper/VG0-LV_ROOT ro i915.modeset=1 vconsole.keymap=us vconsole.font=eurlatgr vconsole.font_map=trivial domdadm_
    [    0.000000] x86/fpu: x87 FPU will use FXSAVE
    [    0.000000] BIOS-provided physical RAM map:
    [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
    [    0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009efff] reserved
    [    0.000000] BIOS-e820: [mem 0x000000000009f000-0x000000000009ffff] usable
    [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000001fffffff] usable
    [    0.000000] BIOS-e820: [mem 0x0000000020000000-0x00000000200fffff] reserved
    [    0.000000] BIOS-e820: [mem 0x0000000020100000-0x000000006d66cfff] usable
    [    0.000000] BIOS-e820: [mem 0x000000006d66d000-0x000000006d69cfff] reserved
    [    0.000000] BIOS-e820: [mem 0x000000006d69d000-0x000000006d6acfff] ACPI data
    [    0.000000] BIOS-e820: [mem 0x000000006d6ad000-0x000000006d7f1fff] ACPI NVS
    [    0.000000] BIOS-e820: [mem 0x000000006d7f2000-0x000000006db80fff] reserved
    [    0.000000] BIOS-e820: [mem 0x000000006db81000-0x000000006db81fff] usable
    [    0.000000] BIOS-e820: [mem 0x000000006db82000-0x000000006dbc3fff] reserved
    [    0.000000] BIOS-e820: [mem 0x000000006dbc4000-0x000000006dd31fff] usable
    [    0.000000] BIOS-e820: [mem 0x000000006dd32000-0x000000006dff9fff] reserved
    [    0.000000] BIOS-e820: [mem 0x000000006dffa000-0x000000006dffffff] usable
    [    0.000000] BIOS-e820: [mem 0x00000000e00f8000-0x00000000e00f8fff] reserved
    [    0.000000] BIOS-e820: [mem 0x00000000fed01000-0x00000000fed01fff] reserved
    [    0.000000] BIOS-e820: [mem 0x00000000ffb00000-0x00000000ffffffff] reserved
    [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000047fffffff] usable
    [    0.000000] printk: console [earlyser0] enabled
    Memory KASLR using RDRAND RDTSC...
    Poking KASLR using RDRAND RDTSC...

Welcome to Fedora 34 (Thirty Four) dracut-055-3.fc34 (Initramfs)!

[  OK  ] Started Dispatch Password …ts to Console Directory Watch.
    [  OK  ] Reached target Local File Systems.
    [  OK  ] Reached target Path Units.
    ...
    Fedora 34 (Thirty Four)
    Kernel 5.12.15-300.fc34.x86_64 on an x86_64 (ttyS0)

srv01 login:
    ...
    lsb_release -rd
    	Description:    Fedora release 34 (Thirty Four)
    	Release:        34
    uname -rm
    	5.12.15-300.fc34.x86_64 x86_64

after simply updating to either of

Fedora (5.12.17-300.fc34.x86_64) 34 (Thirty Four)
    Fedora (5.13.4-200.fc34.x86_64) 34 (Thirty Four)

and re-gen'ing initrd, system boot hangs early,

[    0.000000] microcode: microcode updated early to revision 0x838, date = 2019-04-22
    [    0.000000] Linux version 5.13.4-200.fc34.x86_64 ([e-mail]) (gcc (GCC) 11.1.1 20210531 (Red Hat 11.1.1-3), GNU ld version 2.35.1-41.fc34) #1 SMP Tue Jul 20 20:27:29 UTC 2021
    [    0.000000] Command line: BOOT_IMAGE=(mduuid/5687b25f8f25671a243a83cb661f7841)/vmlinuz-5.13.4-200.fc34.x86_64 root=/dev/mapper/VG0-LV_ROOT ro i915.modeset=1 vconsole.keymap=us vconsole.font=eurlatgr vconsole.font_map=trivial domdadm b
    [    0.000000] x86/fpu: x87 FPU will use FXSAVE
    [    0.000000] BIOS-provided physical RAM map:
    [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
    [    0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009efff] reserved
    [    0.000000] BIOS-e820: [mem 0x000000000009f000-0x000000000009ffff] usable
    [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000001fffffff] usable
    [    0.000000] BIOS-e820: [mem 0x0000000020000000-0x00000000200fffff] reserved
    [    0.000000] BIOS-e820: [mem 0x0000000020100000-0x000000006d66cfff] usable
    [    0.000000] BIOS-e820: [mem 0x000000006d66d000-0x000000006d69cfff] reserved
    [    0.000000] BIOS-e820: [mem 0x000000006d69d000-0x000000006d6acfff] ACPI data
    [    0.000000] BIOS-e820: [mem 0x000000006d6ad000-0x000000006d7f1fff] ACPI NVS
    [    0.000000] BIOS-e820: [mem 0x000000006d7f2000-0x000000006db80fff] reserved
    [    0.000000] BIOS-e820: [mem 0x000000006db81000-0x000000006db81fff] usable
    [    0.000000] BIOS-e820: [mem 0x000000006db82000-0x000000006dbc3fff] reserved
    [    0.000000] BIOS-e820: [mem 0x000000006dbc4000-0x000000006dd31fff] usable
    [    0.000000] BIOS-e820: [mem 0x000000006dd32000-0x000000006dff9fff] reserved
    [    0.000000] BIOS-e820: [mem 0x000000006dffa000-0x000000006dffffff] usable
    [    0.000000] BIOS-e820: [mem 0x00000000e00f8000-0x00000000e00f8fff] reserved
    [    0.000000] BIOS-e820: [mem 0x00000000fed01000-0x00000000fed01fff] reserved
    [    0.000000] BIOS-e820: [mem 0x00000000ffb00000-0x00000000ffffffff] reserved
    [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000047fffffff] usable
    [    0.000000] printk: console [earlyser0] enabled
    Memory KASLR using RDRAND RDTSC...
    Poking KASLR using RDRAND RDTSC...

, and appears to proceed no further.

checking, drop-back to

Fedora (5.12.15-300.fc34.x86_64) 34 (Thirty Four)

boots, again, with no issues.

for the

Fedora (5.13.4-200.fc34.x86_64) 34 (Thirty Four)

case, with logging dialed up, logs actually show a sloooooow boot process ( 700+ seconds and counting ),
with eventual, complete hang @

"Starting Apply Kernel Variables..."

--> https://pastebin.com/dP6Lm84J

So far, this occurs on this J1900 box only; other F34 installs with kernel >= 5.12.17 are booting fine.

AND, for _this_ J1900, all kernels <= 5.12.15 boot OK.

not clear to me if this is a Kernel/upstream, or Kernel/Fedora issue.  Atm, these are Fedora's kernel rpms, not vanilla from upstream.  So starting here.

is there's specific, additional logging that'd be informative, I can attach.

[PGNet Dev]

Found 'vanilla kernels' for Fedora,

https://fedoraproject.org/wiki/Kernel_Vanilla_Repositories
    https://repos.fedorapeople.org/repos/thl/kernel-vanilla-stable/fedora-34/x86...

Installed from repo

kernel-5.13.4-250.vanilla.1.fc34.x86_64.rpm
    kernel-devel-5.13.4-250.vanilla.1.fc34.x86_64.rpm
    kernel-modules-extra-5.13.4-250.vanilla.1.fc34.x86_64.rpm
    kernel-core-5.13.4-250.vanilla.1.fc34.x86_64.rpm
    kernel-modules-5.13.4-250.vanilla.1.fc34.x86_64.rpm

then, re-gen'd init & re-booted

Same issue as above ... long/slow boot, eventual hang.

Again, drop back to 5.12.15, all's good.

[PGNet Dev]

mv'd -> https://lkml.org/lkml/2021/7/22/553

[Justin Forbes]

On Thu, Jul 22, 2021 at 10:00 AM PGNet Dev  wrote:
> Found 'vanilla kernels' for Fedora,
>     https://fedoraproject.org/wiki/Kernel_Vanilla_Repositories
>     https://repos.fedorapeople.org/repos/thl/kernel-vanilla-stable/fedora-34/x86_64/
>
> Installed from repo
>     kernel-5.13.4-250.vanilla.1.fc34.x86_64.rpm
>     kernel-devel-5.13.4-250.vanilla.1.fc34.x86_64.rpm
>     kernel-modules-extra-5.13.4-250.vanilla.1.fc34.x86_64.rpm
>     kernel-core-5.13.4-250.vanilla.1.fc34.x86_64.rpm
>     kernel-modules-5.13.4-250.vanilla.1.fc34.x86_64.rpm
>
> then, re-gen'd init & re-booted
>
> Same issue as above ... long/slow boot, eventual hang.
>
> Again, drop back to 5.12.15, all's good.

Are you sure this isn't another case of the dracut/dbus bug that has
been hitting everyone for a few weeks now? 5.12.16+ was roughly the
timeline that it hit.
https://bugzilla.redhat.com/show_bug.cgi?id=1976653

Justin

[PGNet Dev]

On 7/24/21 11:51 AM, Justin Forbes wrote:
> On Thu, Jul 22, 2021 at 10:00 AM PGNet Dev  wrote:
> > Found 'vanilla kernels' for Fedora,
> >      https://fedoraproject.org/wiki/Kernel_Vanilla_Repositories
> >      https://repos.fedorapeople.org/repos/thl/kernel-vanilla-stable/fedora-34/x86_64/
> >
> > Installed from repo
> >      kernel-5.13.4-250.vanilla.1.fc34.x86_64.rpm
> >      kernel-devel-5.13.4-250.vanilla.1.fc34.x86_64.rpm
> >      kernel-modules-extra-5.13.4-250.vanilla.1.fc34.x86_64.rpm
> >      kernel-core-5.13.4-250.vanilla.1.fc34.x86_64.rpm
> >      kernel-modules-5.13.4-250.vanilla.1.fc34.x86_64.rpm
> >
> > then, re-gen'd init & re-booted
> >
> > Same issue as above ... long/slow boot, eventual hang.
> >
> > Again, drop back to 5.12.15, all's good.
>
> Are you sure this isn't another case of the dracut/dbus bug that has
> been hitting everyone for a few weeks now? 5.12.16+ was roughly the
> timeline that it hit.
> https://bugzilla.redhat.com/show_bug.cgi?id=1976653
>
> Justin

hi,

a bisect found

https://lore.kernel.org/regressions/5b5d1b7f-7327-52a2-5221-de39206a07a3@gma...

the errant commit

96b15a0b45182f1c3da5a861196da27000da2e3c is the first bad commit

which appears (?) unrelated to the #1976653 dbus issue.