Some heads-ups for Rawhide users: python3-ptyprocess and systemd issues

[Adam Williamson]

Hey folks! Just wanted to send out a heads-up for Rawhide users about
some issues that have shown up in openQA testing.

First, you may have found that trying to update to today's Rawhide
fails like this:

Error: Transaction test error:
  file /usr/lib/python3.11/site-packages/ptyprocess-0.7.0-py3.11.egg-info from install of
python3-ptyprocess-0.7.0-2.fc38.noarch conflicts with file from package
python3-ptyprocess-0.7.0-1.fc38.noarch

That's https://bugzilla.redhat.com/show_bug.cgi?id=2164207 , we figured
out the problem (thanks to Panu for getting to the root cause of it)
and it is fixed in python-ptyprocess-0.7.0-3.fc38 , which should be in
the next Rawhide compose. If you really can't wait you can update
directly to that version
from https://koji.fedoraproject.org/koji/buildinfo?buildID=2138649 ,
but you'd better watch out for issue #2...

The second issue is that systems that update to systemd-253~rc1-1.fc38
seem to get stuck on boot. With Plymouth enabled you just see the
splash screen. With it disabled (or by pressing esc) it seems to be
stuck at "Stopped initrd-switch-root.service - Switch Root.". I'm still
looking into this one, but it's happened to a lot of openQA tests and I
was able to confirm it first try in a local VM, by installing from the
20230123.n.0 compose then updating systemd and rebooting. Fresh
installs with the newer systemd seem to be OK, at least most openQA
tests for the new compose passed - it seems to be only updating an
existing install that has the problem, at least so far.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Adam Williamson]

On Wed, 2023-01-25 at 10:30 -0800, Adam Williamson wrote:
>  
>  The second issue is that systems that update to systemd-253~rc1-1.fc38
>  seem to get stuck on boot. With Plymouth enabled you just see the
>  splash screen. With it disabled (or by pressing esc) it seems to be
>  stuck at "Stopped initrd-switch-root.service - Switch Root.". I'm still
>  looking into this one, but it's happened to a lot of openQA tests and I
>  was able to confirm it first try in a local VM, by installing from the
>  20230123.n.0 compose then updating systemd and rebooting. Fresh
>  installs with the newer systemd seem to be OK, at least most openQA
>  tests for the new compose passed - it seems to be only updating an
>  existing install that has the problem, at least so far. 
I've now filed this as
https://bugzilla.redhat.com/show_bug.cgi?id=2164594 .
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Vít Ondruch]

Dne 25. 01. 23 v 19:53 Adam Williamson napsal(a):
>  On Wed, 2023-01-25 at 10:30 -0800, Adam Williamson wrote:
> > The second issue is that systems that update to systemd-253~rc1-1.fc38
> > seem to get stuck on boot. With Plymouth enabled you just see the
> > splash screen. With it disabled (or by pressing esc) it seems to be
> > stuck at "Stopped initrd-switch-root.service - Switch Root.". I'm
> still
> > looking into this one, but it's happened to a lot of openQA tests and I
> > was able to confirm it first try in a local VM, by installing from the
> > 20230123.n.0 compose then updating systemd and rebooting. Fresh
> > installs with the newer systemd seem to be OK, at least most openQA
> > tests for the new compose passed - it seems to be only updating an
> > existing install that has the problem, at least so far.
>  I've now filed this as
>  https://bugzilla.redhat.com/show_bug.cgi?id=2164594 .
>  

Just FTR, this is very likely duplicate of:

https://bugzilla.redhat.com/show_bug.cgi?id=2164404


Vít

[Zbigniew Jędrzejewski-Szmek]

On Thu, Jan 26, 2023 at 09:29:22AM +0100, Vít Ondruch wrote:
>  Dne 25. 01. 23 v 19:53 Adam Williamson napsal(a):
>  > On Wed, 2023-01-25 at 10:30 -0800, Adam Williamson wrote:
>  > > The second issue is that systems that update to systemd-253~rc1-1.fc38
>  > > seem to get stuck on boot. With Plymouth enabled you just see the
>  > > splash screen. With it disabled (or by pressing esc) it seems to be
>  > > stuck at "Stopped initrd-switch-root.service - Switch Root.". I'm
> still
>  > > looking into this one, but it's happened to a lot of openQA tests and I
>  > > was able to confirm it first try in a local VM, by installing from the
>  > > 20230123.n.0 compose then updating systemd and rebooting. Fresh
>  > > installs with the newer systemd seem to be OK, at least most openQA
>  > > tests for the new compose passed - it seems to be only updating an
>  > > existing install that has the problem, at least so far.
>  > I've now filed this as
>  > https://bugzilla.redhat.com/show_bug.cgi?id=2164594 . 
Quoting the commit in dist-git:
    The socket exists and is enabled in the initrd. After switch-root, the system
    goes into an infinite loop trying to stop the socket while incoming audit
    messages trigger start jobs for the socket. This is a bug in the transaction
    logic, that'll need to be fixed separately.

    We need to preset the socket after the upgrade so that it remains enabled
    by default. This should fix the boot issue, though it's not a complete fix,
    because we actually want to allow people to disable the socket.

    On initial install, the socket is covered by preset-all and gets enabled.

This explains why the issue was observed on upgraded systems, but not
on fresh installations.

The issue should now be resolved in rawhide because the socket will be
enabled again. But the issue with pid1 not handling this is a real
problem that we should solve regardless, see
https://github.com/systemd/systemd/issues/26216.

>  Just FTR, this is very likely duplicate of:
>
>  https://bugzilla.redhat.com/show_bug.cgi?id=2164404 
I *think* that those are separate issues. I'm having trouble with
a combination of systemd-253~rc1 and the 6.2.0.rc4 kernel.
udev is unhappy.

Zbyszek

[Adam Williamson]

On Wed, 2023-01-25 at 10:30 -0800, Adam Williamson wrote:
>  Hey folks! Just wanted to send out a heads-up for Rawhide users
> about
>  some issues that have shown up in openQA testing.
>  
>  First, you may have found that trying to update to today's Rawhide
>  fails like this:
>  
>  Error: Transaction test error:
>    file /usr/lib/python3.11/site-packages/ptyprocess-0.7.0-py3.11.egg-info from install of
> python3-ptyprocess-0.7.0-2.fc38.noarch conflicts with file from package
> python3-ptyprocess-0.7.0-1.fc38.noarch
>  
>  That's https://bugzilla.redhat.com/show_bug.cgi?id=2164207 , we figured
>  out the problem (thanks to Panu for getting to the root cause of it)
>  and it is fixed in python-ptyprocess-0.7.0-3.fc38 , which should be in
>  the next Rawhide compose. If you really can't wait you can update
>  directly to that version
>  from https://koji.fedoraproject.org/koji/buildinfo?buildID=2138649 ,
>  but you'd better watch out for issue #2...
>  
>  The second issue is that systems that update to systemd-253~rc1-1.fc38
>  seem to get stuck on boot. With Plymouth enabled you just see the
>  splash screen. With it disabled (or by pressing esc) it seems to be
>  stuck at "Stopped initrd-switch-root.service - Switch Root.". I'm still
>  looking into this one, but it's happened to a lot of openQA tests and I
>  was able to confirm it first try in a local VM, by installing from the
>  20230123.n.0 compose then updating systemd and rebooting. Fresh
>  installs with the newer systemd seem to be OK, at least most openQA
>  tests for the new compose passed - it seems to be only updating an
>  existing install that has the problem, at least so far. 
And to add to the fun, I think either a mesa update or a glibc update
has broken GNOME and KDE: in current openQA tests, with an updated base
image to avoid the systemd and ptyprocess issues, GNOME and KDE tests
boot fine initially, run an update that includes glibc and mesa, then
reboot and fail to reach the desktop. I'm gonna look into this in more
detail when I get home, but here's the X traceback:

Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) open /dev/fb0: Permission
denied
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (WW) VGA arbiter: cannot open
kernel arbiter, no multi-card support
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (II) modeset(0): Creating default
Display subsection in Screen section
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]:         "Default Screen
Section" for depth/fbbpp 24/32
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (==) modeset(0): Depth 24, (==)
framebuffer bpp 32
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (==) modeset(0): RGB weight 888
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (==) modeset(0): Default visual
is TrueColor
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (II) Loading sub module
"glamoregl"
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (II) LoadModule:
"glamoregl"
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (II) Loading
/usr/lib64/xorg/modules/libglamoregl.so
Jan 25 13:38:47 fedora abrtd[622]: Skipping
/var/spool/abrt/ccpp-2023-01-25-16:38:47.526247-1012.new: directory locked. Is a backtrace
being generated?
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (II) Module glamoregl:
vendor="X.Org Foundation"
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]:         compiled for 1.20.14,
module version = 1.0.1
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]:         ABI class: X.Org ANSI C
Emulation, version 0.4
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: free(): double free detected in
tcache 2
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE)
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) Backtrace:
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]:         ABI class: X.Org ANSI C
Emulation, version 0.4
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: free(): double free detected in
tcache 2
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE)
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) Backtrace:
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 0: /usr/libexec/Xorg
(ErrorFSigSafe+0xcc) [0x556438ff496c]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 1: /lib64/libc.so.6
(__sigaction+0x50) [0x7fbae3c3bba0]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 2: /lib64/libc.so.6
(__pthread_kill_implementation+0x114) [0x7fbae3c8cad4]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 3: /lib64/libc.so.6
(gsignal+0x1e) [0x7fbae3c3baee]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 4: /lib64/libc.so.6
(abort+0xdf) [0x7fbae3c2487f]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 5: /lib64/libc.so.6
(__libc_message.cold+0x5) [0x7fbae3c2560f]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 6: /lib64/libc.so.6
(malloc_printerr+0x15) [0x7fbae3c96a05]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 7: /lib64/libc.so.6
(_int_free+0x9e5) [0x7fbae3c98de5]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 8: /lib64/libc.so.6
(__libc_free+0x7e) [0x7fbae3c9b42e]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 9:
/usr/lib64/dri/zink_dri.so (__driDriverGetExtensions_zink+0x9e70) [0x7fbad82b8180]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 10: /lib64/libgbm.so.1
(gbm_format_get_name+0xe81) [0x7fbae3229361]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 11: /lib64/libgbm.so.1
(gbm_format_get_name+0x1018) [0x7fbae32294f8]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 12: /lib64/libgbm.so.1
(gbm_format_get_name+0x121a) [0x7fbae32296fa]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) unw_get_proc_name failed: no
unwind info found [-10]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 13: /lib64/libgbm.so.1
(?+0x0) [0x7fbae3227c5c]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 14: /lib64/libgbm.so.1
(gbm_create_device+0x4e) [0x7fbae3227dae]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 15:
/usr/lib64/xorg/modules/libglamoregl.so (glamor_egl_init+0x58) [0x7fbae324f818]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) unw_get_proc_name failed: no
unwind info found [-10]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 16:
/usr/lib64/xorg/modules/drivers/modesetting_drv.so (?+0x0) [0x7fbae328b65b]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 17: /usr/libexec/Xorg
(InitOutput+0x121b) [0x556438ed2e4b]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 18: /usr/libexec/Xorg
(miPutImage+0xb6d) [0x556438e7449c]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 19: /lib64/libc.so.6
(__libc_start_call_main+0x7a) [0x7fbae3c25b4a]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 20: /lib64/libc.so.6
(__libc_start_main+0x8b) [0x7fbae3c25c0b]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 21: /usr/libexec/Xorg
(_start+0x25) [0x556438e752a5]
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE)
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE)
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: Fatal server error:
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) Caught signal 6 (Aborted).
Server aborting
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE)
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE)
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: Please consult the The X.Org
Foundation support
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]:          at http://wiki.x.org
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]:  for help.
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) Please also check the log
file at "/var/lib/gdm/.local/share/xorg/Xorg.0.log" for additional information.
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE)
Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) Server terminated with error
(1). Closing log file.

so, yeah, today is a great day to go outside, enjoy the
sunshine/snow/rain (delete as appropriate), and *not* update your
Rawhide installation. :D
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Florian Weimer]

* Adam Williamson:

>  6 (__libc_message.cold+0x5) [0x7fbae3c2560f]
>  Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 6: /lib64/libc.so.6
> (malloc_printerr+0x15) [0x7fbae3c96a05]
>  Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 7: /lib64/libc.so.6
> (_int_free+0x9e5) [0x7fbae3c98de5]
>  Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 8: /lib64/libc.so.6
> (__libc_free+0x7e) [0x7fbae3c9b42e]
>  Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 9:
> /usr/lib64/dri/zink_dri.so (__driDriverGetExtensions_zink+0x9e70) [0x7fbad82b8180]
>  Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 10: /lib64/libgbm.so.1
> (gbm_format_get_name+0xe81) [0x7fbae3229361]
>  Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 11: /lib64/libgbm.so.1
> (gbm_format_get_name+0x1018) [0x7fbae32294f8]
>  Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 12: /lib64/libgbm.so.1
> (gbm_format_get_name+0x121a) [0x7fbae32296fa] 
I saw this during the latest glibc update, but given that the update
wasn't tested in isolation, I waived it through because the update
addresses the known sprintf issue.

I'm not aware of anything in glibc this might have caused this.  The
crashes related to -D_FORTIFY_SOURCE=3 look different (they try to crash
before causing heap corruption!), and it's not the sprintf assertion
failure in __printf_buffer_as_file_commit, either.

Thanks,
Florian

[Adam Williamson]

On Thu, 2023-01-26 at 12:41 +0100, Florian Weimer wrote:
>  * Adam Williamson:
>  
>  > 6 (__libc_message.cold+0x5) [0x7fbae3c2560f]
>  > Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 6: /lib64/libc.so.6
> (malloc_printerr+0x15) [0x7fbae3c96a05]
>  > Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 7: /lib64/libc.so.6
> (_int_free+0x9e5) [0x7fbae3c98de5]
>  > Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 8: /lib64/libc.so.6
> (__libc_free+0x7e) [0x7fbae3c9b42e]
>  > Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 9:
> /usr/lib64/dri/zink_dri.so (__driDriverGetExtensions_zink+0x9e70) [0x7fbad82b8180]
>  > Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 10: /lib64/libgbm.so.1
> (gbm_format_get_name+0xe81) [0x7fbae3229361]
>  > Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 11: /lib64/libgbm.so.1
> (gbm_format_get_name+0x1018) [0x7fbae32294f8]
>  > Jan 25 13:38:47 fedora /usr/libexec/gdm-x-session[1040]: (EE) 12: /lib64/libgbm.so.1
> (gbm_format_get_name+0x121a) [0x7fbae32296fa]
>  
>  I saw this during the latest glibc update, but given that the update
>  wasn't tested in isolation, I waived it through because the update
>  addresses the known sprintf issue.
>  
>  I'm not aware of anything in glibc this might have caused this.  The
>  crashes related to -D_FORTIFY_SOURCE=3 look different (they try to crash
>  before causing heap corruption!), and it's not the sprintf assertion
>  failure in __printf_buffer_as_file_commit, either.
>   
No, it's definitely mesa that causes it - I verified that with manual
local testing, updating only mesa causes the bug to start happening,
downgrading it makes it go away. This is now being tracked/investigated
as https://bugzilla.redhat.com/show_bug.cgi?id=2164667 .
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net