libGL mess with nvidia drivers

[Amadeus W.M.]

Just swapped nouveau out for the nvidia driver from rpmfusion following 
https://rpmfusion.org/Howto/NVIDIA. Basically I did

dnf install xorg-x11-drv-nvidia-340xx akmod-nvidia-340xx "kernel-devel-
uname-r == $(uname -r)"
dnf update -y

Reboot and startx and only one (of 2) monitor was working, the other was 
blank. I ran

nvidia-config

and this created a /etc/X11/xorg.conf, then restarted the X server and 
both monitors are up and running and I'm running the nvidia drivers:

28) root:~> lsmod | grep nvidia
nvidia              10563584  24
drm                   348160  4 nvidia
29) root:~> lsmod | grep nouveau
30) root:~>

Now the problems:

30) root:~> glxinfo
name of display: :0.0
X Error of failed request:  BadWindow (invalid Window parameter)
  Major opcode of failed request:  154 (NV-GLX)
  Minor opcode of failed request:  4 ()
  Resource id in failed request:  0x4a00009
  Serial number of failed request:  56
  Current serial number in output stream:  56
31) root:~> glxgears
X Error of failed request:  BadWindow (invalid Window parameter)
  Major opcode of failed request:  154 (NV-GLX)
  Minor opcode of failed request:  4 ()
  Resource id in failed request:  0x4a00002
  Serial number of failed request:  38
  Current serial number in output stream:  38
32) root:~> xdriinfo 
libGL is too old.
33) root:~>

I think all this has to do with libGL.

34) root:~> ldd /usr/bin/glxgears | grep libGL
    libGL.so.1 => /usr/lib64/nvidia-340xx/libGL.so.1 
(0x00007f55a0e8c000)

so that points to the nvidia libGL. But I have

35) root:~> ls -l /usr/lib64/libGL.so*
lrwxrwxrwx. 1 root root     14 Jul  6 04:12 /usr/lib64/libGL.so -> 
libGL.so.1.0.0
lrwxrwxrwx. 1 root root     14 Jul  6 04:12 /usr/lib64/libGL.so.1 -> 
libGL.so.1.0.0
-rwxr-xr-x. 1 root root 581840 Jul  6 04:12 /usr/lib64/libGL.so.1.0.0

and

36) root:~> rpm -qf /usr/lib64/libGL.so.1.0.0 
libglvnd-glx-0.2.999-19.20170620gitd850cdd.fc26.x86_64

Of course, nvidia provides its own libGL:

37) root:~> rpm -qf /usr/lib64/nvidia-340xx/libGL.so.*
xorg-x11-drv-nvidia-340xx-libs-340.102-1.fc26.x86_64
xorg-x11-drv-nvidia-340xx-libs-340.102-1.fc26.x86_64

Finally, the mess:

38) root:~> rpm -e --test libglvnd-glx
error: Failed dependencies:
    libGL is needed by (installed) python2-pyglet-1.2.4-3.fc26.noarch
    libGL.so.1()(64bit) is needed by (installed) qt-
x11-1:4.8.7-28.fc26.x86_64
    libGL.so.1()(64bit) is needed by (installed) mesa-
libGLU-9.0.0-11.fc26.x86_64
    libGL.so.1()(64bit) is needed by (installed) xorg-x11-server-
Xorg-1.19.3-4.fc26.x86_64
    libGL.so.1()(64bit) is needed by (installed) 
Coin3-3.1.3-19.fc26.x86_64
    libGL.so.1()(64bit) is needed by (installed) 
fltk-1.3.4-1.fc26.x86_64
    libGL.so.1()(64bit) is needed by (installed) 
fox-1.6.54-1.fc26.x86_64
    libGL.so.1()(64bit) is needed by (installed) glx-
utils-8.3.0-6.fc26.x86_64

and MANY more. So I can't uninstall the libGL provided by fedora, it would 
take out too many packages with it, including the X server.

Can I choose which libGL to use with alternatives somehow? How do switch 
everything to use the nvidia libGL?

I assume there are people there who are using the nvidia driver right now, 
some maybe using the rpmfusion rpms. Can someone post which libGL they 
have installed on their system and how they got the whole thing to work?

Thanks!

[Tom Horsley]

On Sun, 6 Aug 2017 02:59:25 +0000 (UTC)
Amadeus W.M. wrote:
> I assume there are people there who are using the nvidia driver right now, 
> some maybe using the rpmfusion rpms. Can someone post which libGL they 
> have installed on their system and how they got the whole thing to work?

I got it to work by simply installing the rpmfusion akmod package.

It makes a file in /etc/ld.so.conf.d/ named nvidia-lib64.conf
which points at /usr/lib64/nvidia and /usr/lib64/libglvnd
which should override the "default" path where the mesa
versions live.

BUT: I once discovered a curious thing. If some other totally
unrelated package happens to install another file in
/etc/ld.so.conf.d which explicitly names the default
library locations such as /lib64, then it depends entirely
on which way ldconfig happens to hash things as to which
library it will find first.

So the first step is to see if any other file in /etc/ld.so.conf.d
has a line that just says "/lib64" or "/usr/lib64", and if it
does complain to whatever package installed that file (which you
can find with rpm -q -f full-filename).

[Amadeus W.M.]

On Sat, 05 Aug 2017 23:20:53 -0400, Tom Horsley wrote:
> On Sun, 6 Aug 2017 02:59:25 +0000 (UTC)
> Amadeus W.M. wrote:
> > I assume there are people there who are using the nvidia driver right
> > now,
> > some maybe using the rpmfusion rpms. Can someone post which libGL they
> > have installed on their system and how they got the whole thing to
> > work?
>
> I got it to work by simply installing the rpmfusion akmod package.
>
> It makes a file in /etc/ld.so.conf.d/ named nvidia-lib64.conf which
> points at /usr/lib64/nvidia and /usr/lib64/libglvnd which should
> override the "default" path where the mesa versions live.
>
> BUT: I once discovered a curious thing. If some other totally unrelated
> package happens to install another file in /etc/ld.so.conf.d which
> explicitly names the default library locations such as /lib64, then it
> depends entirely on which way ldconfig happens to hash things as to
> which library it will find first.
>
> So the first step is to see if any other file in /etc/ld.so.conf.d has a
> line that just says "/lib64" or "/usr/lib64", and if it does complain to
> whatever package installed that file (which you can find with rpm -q -f
> full-filename). _______________________________________________
> users mailing list -- [e-mail] To unsubscribe send
> an email to [e-mail]

Thanks for the quick reply! I do have an nvidia-lib64.conf and this is 
what's in it:

8) root:/etc/ld.so.conf.d> cat /etc/ld.so.conf.d/nvidia-lib64.conf 
/usr/lib64/nvidia-340xx

There's no /usr/lib64/libglvnd in it nor do I have a /usr/lib64/libglvnd 
directory on my system:

ls /usr/lib64/libglvnd*
ls: cannot access '/usr/lib64/libglvnd*': No such file or directory

This is what I have /etc/ld.so.conf.d:

21) root:/etc/ld.so.conf.d> grep lib64 *
atlas-x86_64.conf:/usr/lib64/atlas
bind99-x86_64.conf:/usr/lib64/bind99
libiscsi-x86_64.conf:/usr/lib64/iscsi
mariadb-x86_64.conf:/usr/lib64/mysql
nvidia-lib64.conf:/usr/lib64/nvidia-340xx
nx-libs-x86_64.conf:/usr/lib64/nx
octave-x86_64.conf:/usr/lib64/octave/4.2.1
qt-x86_64.conf:/usr/lib64/qt-3.3/lib
tix-x86_64.conf:/usr/lib64/tcl8.6

No /lib64 nor /usr/lib64, only /usr/lib64/<something>.

From /usr/share/doc/libglvnd/README.md it looks like libglvnd is supposed 
to manage opengl calls between multiple vendors. Maybe it's not working 
right?

Do you have other libGL installed on your system? This is what I have:

28) root:~> locate libGL.so
/data/MATLAB/R2016a/sys/opengl/lib/glnxa64/libGL.so.1
/data/MATLAB/R2016a/sys/opengl/lib/glnxa64/libGL.so.1.6.0
/usr/lib64/libGL.so
/usr/lib64/libGL.so.1
/usr/lib64/libGL.so.1.0.0
/usr/lib64/nvidia-340xx/libGL.so.1
/usr/lib64/nvidia-340xx/libGL.so.340.102
/var/lib/docker/overlay2/
c9451d99d6ef2a10b3dc8426ba20a88879c5c921b8dcc371db330502c08f7226/diff/usr/
lib64/libGL.so.1
/var/lib/docker/overlay2/
c9451d99d6ef2a10b3dc8426ba20a88879c5c921b8dcc371db330502c08f7226/diff/usr/
lib64/libGL.so.1.0.0

Incidentally,

uname -a
Linux alpha 4.11.11-300.fc26.x86_64 #1 SMP Mon Jul 17 16:32:11 UTC 2017 
x86_64 x86_64 x86_64 GNU/Linux

[Tom Horsley]

On Sun, 6 Aug 2017 03:53:05 +0000 (UTC)
Amadeus W.M. wrote:
> Linux alpha 4.11.11-300.fc26.x86_64 #1 SMP Mon Jul 17 16:32:11 UTC 2017 
> x86_64 x86_64 x86_64 GNU/Linux

I'm on fedora 24 at the moment, so the libraries may be
different. Actually, things are apparently radically
different. On my fedora 26 system at work, there
is no nvidia file under /etc/ld.so.conf.d, so things
must work in some completely different way on f26.

Looking at the libraries loaded by a GL program on my
f26 system at work, I see a new one: /lib64/libGLdispatch.so.0
which is part of the libglvnd package that says:

libglvnd is an implementation of the vendor-neutral dispatch layer for
arbitrating OpenGL API calls between multiple vendors on a per-screen basis.

So maybe your aren't supposed to have any ldconfig overrides any
longer. Could the one you see be leftover from an upgrade versus
a fresh install?

I see the rpm at work:

xorg-x11-drv-nvidia-libs-375.66-7.fc26.x86_64

contains things that look like they might be
nvidia specific versions of stuff ligvlgnd might
be dispatching:

/usr/lib64/libEGL_nvidia.so.0
/usr/lib64/libEGL_nvidia.so.375.66
/usr/lib64/libGLESv1_CM_nvidia.so.1
/usr/lib64/libGLESv1_CM_nvidia.so.375.66
/usr/lib64/libGLESv2_nvidia.so.2
/usr/lib64/libGLESv2_nvidia.so.375.66
/usr/lib64/libGLX_indirect.so.0
/usr/lib64/libGLX_nvidia.so.0
/usr/lib64/libGLX_nvidia.so.375.66
/usr/lib64/libnvidia-cfg.so.1
/usr/lib64/libnvidia-cfg.so.375.66
/usr/lib64/libnvidia-eglcore.so.375.66
/usr/lib64/libnvidia-fbc.so.1
/usr/lib64/libnvidia-fbc.so.375.66
/usr/lib64/libnvidia-glcore.so.375.66
/usr/lib64/libnvidia-glsi.so.375.66
/usr/lib64/libnvidia-ifr.so.1
/usr/lib64/libnvidia-ifr.so.375.66
/usr/lib64/libnvidia-tls.so.375.66
/usr/lib64/nvidia
/usr/lib64/vdpau/libvdpau_nvidia.so.1
/usr/lib64/vdpau/libvdpau_nvidia.so.375.66

[Ed Greshko]

On 08/06/2017 11:53 AM, Amadeus W.M. wrote:
> On Sat, 05 Aug 2017 23:20:53 -0400, Tom Horsley wrote:
> > On Sun, 6 Aug 2017 02:59:25 +0000 (UTC)
> > Amadeus W.M. wrote:
> > > I assume there are people there who are using the nvidia driver right
> > > now,
> > > some maybe using the rpmfusion rpms. Can someone post which libGL they
> > > have installed on their system and how they got the whole thing to
> > > work?

Oh, and the other thing I would check is to make sure you're loading the correct glx
module.....

[egreshko@acer log]$ grep -i glx /var/log/Xorg.0.log
[    94.014] (II) "glx" will be loaded by default.
[    94.014] (II) LoadModule: "glx"
[    94.015] (II) Loading /usr/lib64/nvidia-340xx/xorg/libglx.so
[    94.162] (II) Module glx: vendor="NVIDIA Corporation"
[    94.163] (II) NVIDIA GLX Module  340.102  Mon Jan 16 12:37:38 PST 2017
[    97.130] (II) Initializing extension GLX
[    97.130] (II) Indirect GLX disabled.(II) config/udev: Adding input device Power
Button (/dev/input/event3)\

In a previous post by you I saw...

Major opcode of failed request:  154 (NV-GLX)

Which seems odd....

[Amadeus W.M.]

On Sun, 06 Aug 2017 12:51:12 +0800, Ed Greshko wrote:
> On 08/06/2017 11:53 AM, Amadeus W.M. wrote:
> > On Sat, 05 Aug 2017 23:20:53 -0400, Tom Horsley wrote:
> > > On Sun, 6 Aug 2017 02:59:25 +0000 (UTC)
> > > Amadeus W.M. wrote:
> > > > I assume there are people there who are using the nvidia driver right
> > > > now,
> > > > some maybe using the rpmfusion rpms. Can someone post which libGL
> > > > they have installed on their system and how they got the whole thing
> > > > to work?
>
> Oh, and the other thing I would check is to make sure you're loading the
> correct glx module.....
>
> [egreshko@acer log]$ grep -i glx /var/log/Xorg.0.log [    94.014] (II)
> "glx" will be loaded by default.
> [    94.014] (II) LoadModule: "glx"
> [    94.015] (II) Loading /usr/lib64/nvidia-340xx/xorg/libglx.so [   
> 94.162] (II) Module glx: vendor="NVIDIA Corporation"
> [    94.163] (II) NVIDIA GLX Module  340.102  Mon Jan 16 12:37:38 PST
> 2017 [    97.130] (II) Initializing extension GLX [    97.130] (II)
> Indirect GLX disabled.(II) config/udev: Adding input device Power Button
> (/dev/input/event3)\
>
> In a previous post by you I saw...
>
> Major opcode of failed request:  154 (NV-GLX)
>
> Which seems odd....

Spot on! See my post with the possible solution. The X server was trying 
to load /usr/lib64/xorg/modules/extensions/libglx.so
which fails with the nvidia driver. I fixed this - getto-style - by 
linking the xorg libglx.so to the nvidia libglx.so (which is in 
/usr/lib64/nvidia-340xx) and, knock on wood, all seems well now.

I don't know why it was trying to load the xorg glx because

cat /etc/X11/xorg.conf.d/99-nvidia.conf 
#This file is provided by xorg-x11-drv-nvidia-340xx
#Do not edit

Section "Files"
    ModulePath   "/usr/lib64/nvidia-340xx/xorg"
    ModulePath   "/usr/lib64/xorg/modules"
EndSection

So it stands to reason that if it searches the paths in the above order it 
should find the nvidia libglx.so.

[Ed Greshko]

On 08/06/2017 01:37 PM, Amadeus W.M. wrote:
> On Sun, 06 Aug 2017 12:51:12 +0800, Ed Greshko wrote:
> > On 08/06/2017 11:53 AM, Amadeus W.M. wrote:
> > > On Sat, 05 Aug 2017 23:20:53 -0400, Tom Horsley wrote:
> > > > On Sun, 6 Aug 2017 02:59:25 +0000 (UTC)
> > > > Amadeus W.M. wrote:
> > > > > I assume there are people there who are using the nvidia driver right
> > > > > now,
> > > > > some maybe using the rpmfusion rpms. Can someone post which libGL
> > > > > they have installed on their system and how they got the whole thing
> > > > > to work?
> >
> > Oh, and the other thing I would check is to make sure you're loading the
> > correct glx module.....
> >
> > [egreshko@acer log]$ grep -i glx /var/log/Xorg.0.log [    94.014] (II)
> > "glx" will be loaded by default.
> > [    94.014] (II) LoadModule: "glx"
> > [    94.015] (II) Loading /usr/lib64/nvidia-340xx/xorg/libglx.so [   
> > 94.162] (II) Module glx: vendor="NVIDIA Corporation"
> > [    94.163] (II) NVIDIA GLX Module  340.102  Mon Jan 16 12:37:38 PST
> > 2017 [    97.130] (II) Initializing extension GLX [    97.130] (II)
> > Indirect GLX disabled.(II) config/udev: Adding input device Power Button
> > (/dev/input/event3)\
> >
> > In a previous post by you I saw...
> >
> > Major opcode of failed request:  154 (NV-GLX)
> >
> > Which seems odd....
>
> Spot on! See my post with the possible solution. The X server was trying 
> to load /usr/lib64/xorg/modules/extensions/libglx.so
> which fails with the nvidia driver. I fixed this - getto-style - by 
> linking the xorg libglx.so to the nvidia libglx.so (which is in 
> /usr/lib64/nvidia-340xx) and, knock on wood, all seems well now.
>
> I don't know why it was trying to load the xorg glx because
>
> cat /etc/X11/xorg.conf.d/99-nvidia.conf 
> #This file is provided by xorg-x11-drv-nvidia-340xx
> #Do not edit
>
> Section "Files"
>    ModulePath   "/usr/lib64/nvidia-340xx/xorg"
>    ModulePath   "/usr/lib64/xorg/modules"
> EndSection
>
> So it stands to reason that if it searches the paths in the above order it 
> should find the nvidia libglx.so.

Well, you've created a symbolic link to "workaround" an issue.  However, it should
not have been necessary and wasn't necessary in my case.  I would be concerned that
some update in the future may end up removing your link or overwriting the file to
which it is linked to and put you back in confusion mode.

I ran the nvidia-config script and wrote the resulting file to a temp location.  I
noticed that it has the line

Load           "glx"

which may override the information in 99-nvidia.conf  or be parsed before that.....

So, if you still have the xorg.conf I would try to do away with that and get the
correct lib loaded without a link....as you may find yourself in this situation at a
later date and forgot what you did to "fix" it.   :-)

[Ed Greshko]

On 08/06/2017 10:59 AM, Amadeus W.M. wrote:
> Just swapped nouveau out for the nvidia driver from rpmfusion following 
> https://rpmfusion.org/Howto/NVIDIA. Basically I did

I just happen to have an old Acer laptop with nVidia HW that would need the 3400-xx
drivers to run.  This was a fresh install since I replaced the HD with an SSD to
hopefully speed things up....and it did.  I'd been running nouveau but since I'd run
nVidia on this system in the past I decided to install just to see what results I had.
> dnf install xorg-x11-drv-nvidia-340xx akmod-nvidia-340xx "kernel-devel-
> uname-r == $(uname -r)"
> dnf update -y

I simply did....

dnf install akmod-nvidia-340xx

This installed 22 packages...

akmods
 dwz
 fakeroot
 fakeroot-libs
 fedora-rpm-macros
 fpc-srpm-macros
 ghc-srpm-macros
 gnat-srpm-macros
 go-srpm-macros
 kernel-devel-4.11.11-300.fc26.x86_64          
 kmodtool
 ocaml-srpm-macros
 perl-srpm-macros
 qt5-srpm-macros
 redhat-rpm-config
 rpm-build
 rpmdevtools
 xemacs-filesystem
 xorg-x11-drv-nvidia-340xx
 xorg-x11-drv-nvidia-340xx-kmodsrc
 xorg-x11-drv-nvidia-340xx-libs
> Reboot and startx and only one (of 2) monitor was working, the other was 
> blank. I ran

I only have the laptop display hooked up so after reboot and after it build the kmod
it came up to a login under SDDM as it should.
> nvidia-config
>
> and this created a /etc/X11/xorg.conf, then restarted the X server and 
> both monitors are up and running and I'm running the nvidia drivers:

I've always found it a bad idea to run that and create an xorg.conf file as the
monitors should be detected on boot provided they are turned on.  I think I would
have done a bit more looking around with xrandr to see if I could find anything odd.
> root:~> lsmod | grep nvidia
>
> nvidia              10563584  24
> drm                   348160  4 nvidia
> 29) root:~> lsmod | grep nouveau
> 30) root:~>
>
> Now the problems:
> root:~> glxinfo
>
> name of display: :0.0
> X Error of failed request:  BadWindow (invalid Window parameter)
>   Major opcode of failed request:  154 (NV-GLX)
>   Minor opcode of failed request:  4 ()
>   Resource id in failed request:  0x4a00009
>   Serial number of failed request:  56
>   Current serial number in output stream:  56

I get....

name of display: :0
display: :0  screen: 0
direct rendering: Yes
server glx vendor string: NVIDIA Corporation
server glx version string: 1.4
server glx extensions:

and a whole lot of more output
> root:~> glxgears
>
> X Error of failed request:  BadWindow (invalid Window parameter)
>   Major opcode of failed request:  154 (NV-GLX)
>   Minor opcode of failed request:  4 ()
>   Resource id in failed request:  0x4a00002
>   Serial number of failed request:  38
>   Current serial number in output stream:  38

I get...

Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
297 frames in 5.0 seconds = 59.232 FPS
298 frames in 5.0 seconds = 59.434 FPS
298 frames in 5.0 seconds = 59.425 FPS
> root:~> xdriinfo
>
> libGL is too old.

That is the same for me....

As is everything else beyond this point in the original post.
> and MANY more. So I can't uninstall the libGL provided by fedora, it would 
> take out too many packages with it, including the X server.

You need not do that...

[egreshko@acer ~]$ rpm -q libglvnd-glx
libglvnd-glx-0.2.999-19.20170620gitd850cdd.fc26.x86_64
> Can I choose which libGL to use with alternatives somehow? How do switch 
> everything to use the nvidia libGL?
>
> I assume there are people there who are using the nvidia driver right now, 
> some maybe using the rpmfusion rpms. Can someone post which libGL they 
> have installed on their system and how they got the whole thing to work?

Yes, I am now.  With no difficulties.

To answer the question you had in your follow-up post.....

[egreshko@acer ~]$ locate libGL.so
/usr/lib64/libGL.so.1
/usr/lib64/libGL.so.1.0.0
/usr/lib64/nvidia-340xx/libGL.so.1
/usr/lib64/nvidia-340xx/libGL.so.340.102

I've, at times, thought I had issues with nVidia and to fall back to nouveau I simply
did.... and "rpm -qa | grep nvidia" and then removed the packages that were found. 
The rpmfusion folks did a good job, IMHO, and this results in nouveau getting fully
restored.

So, I would do that....   And then to go to nVidia I would just do

dnf install akmod-nvidia-340xx

OR, I would first get rid of the /etc/X11/xorg.conf and see if I can't get the second
screen working with no config file.  And, if not, see if with that file removed you
can run glxinfo and glxgears.

[Amadeus W.M.]

Possible solution:

The X server is trying to load /usr/lib64/xorg/modules/extensions/libglx.so
but on my system, that file belongs to xorg-x11-server-Xorg not to nvidia. 
Consequently, loading GLX fails, which can be seen in /var/log/X.log.0:

[  8947.068] (EE) NVIDIA(0): Failed to initialize the GLX module; please 
check in your X
etc.

Nvidia comes with its own libglx.so:

locate libglx.so
/usr/lib64/nvidia-340xx/xorg/libglx.so
/usr/lib64/nvidia-340xx/xorg/libglx.so.1
/usr/lib64/nvidia-340xx/xorg/libglx.so.340.102
/usr/lib64/xorg/modules/extensions/libglx.so

So I felt adventurous and linked 
/usr/lib64/xorg/modules/extensions/libglx.so to 
/usr/lib64/nvidia-340xx/xorg/libglx.so

Incidentally, /usr/lib64/xorg/modules/extensions/libglx.so was not a link 
(why not?) so I made a backup before linking it to the nvidia libglx.so.

Then startx, and lo and behold, glxgears, glxinfo, etc. worked as they 
should! Unfortunately xdriinfo still says libGL too old, so something is 
still off and I expect everything to come crumbling down upon a future 
update, but so far it's working.

Now on to installing cuda in a docker container!