akmods issue

[Neal Becker]

Speaking of nvidia and akmods, I just had an issue this morning.  On my
server I'm using nvidia proprietary driver from rpmfusion.  Apparently it
was updated (automatically) last night and when I tried to run experiments
on the gpu this morning I got the dreaded version mismatch between the
kernel driver and cuda library (or something).

I tried rebooting but that didn't fix the issue.

I tried running akmods manually but it didn't find anything to build.  The
man page for akmods was unhelpful, and looking in /lib/modules wasn't
useful because you can't see the version numbers on the modules to see that
everything is updated.

Finally I tried rebooting a 2nd time on the theory that the akmod got run
and the driver built on the first reboot, but too late - the old driver had
already loaded.

Surprise, that worked.  So what's going on?  It looks like akmod is run too
late and  2 reboots are needed to get the new driver built and loaded?

I checked
sudo systemctl status akmods
● akmods.service - Builds and install new kmods from akmod packages
     Loaded: loaded (/usr/lib/systemd/system/akmods.service; enabled;
vendor preset: enabled)
     Active: active (exited) since Wed 2022-07-06 07:32:31 EDT; 28min ago
    Process: 1166 ExecStart=/usr/sbin/akmods --from-init (code=exited,
status=0/SUCCESS)
   Main PID: 1166 (code=exited, status=0/SUCCESS)
        CPU: 215ms

Jul 06 07:32:29 nbecker8 systemd[1]: Starting Builds and install new kmods
from akmod packages...
Jul 06 07:32:31 nbecker8 akmods[1166]: Checking kmods exist for
5.18.9-100.fc35.x86_64[  OK  ]
Jul 06 07:32:31 nbecker8 systemd[1]: Finished Builds and install new kmods
from akmod packages.

Any ideas?

[John Pilkington]

On 06/07/2022 13:07, Neal Becker wrote:
> Speaking of nvidia and akmods, I just had an issue this morning.  On my 
> server I'm using nvidia proprietary driver from rpmfusion.  Apparently 
> it was updated (automatically) last night and when I tried to run 
> experiments on the gpu this morning I got the dreaded version mismatch 
> between the kernel driver and cuda library (or something).
>
> I tried rebooting but that didn't fix the issue.
>
> I tried running akmods manually but it didn't find anything to build.  
> The man page for akmods was unhelpful, and looking in /lib/modules 
> wasn't useful because you can't see the version numbers on the modules 
> to see that everything is updated.
>
> Finally I tried rebooting a 2nd time on the theory that the akmod got 
> run and the driver built on the first reboot, but too late - the old 
> driver had already loaded.
>
> Surprise, that worked.  So what's going on?  It looks like akmod is run 
> too late and  2 reboots are needed to get the new driver built and loaded?
>
> I checked
> sudo systemctl status akmods
> ● akmods.service - Builds and install new kmods from akmod packages
>       Loaded: loaded (/usr/lib/systemd/system/akmods.service; enabled; 
> vendor preset: enabled)
>       Active: active (exited) since Wed 2022-07-06 07:32:31 EDT; 28min ago
>      Process: 1166 ExecStart=/usr/sbin/akmods --from-init (code=exited, 
> status=0/SUCCESS)
>     Main PID: 1166 (code=exited, status=0/SUCCESS)
>          CPU: 215ms
>
> Jul 06 07:32:29 nbecker8 systemd[1]: Starting Builds and install new 
> kmods from akmod packages...
> Jul 06 07:32:31 nbecker8 akmods[1166]: Checking kmods exist for 
> 5.18.9-100.fc35.x86_64[  OK  ]
> Jul 06 07:32:31 nbecker8 systemd[1]: Finished Builds and install new 
> kmods from akmod packages.
>
> Any ideas?

Well, I'm still on Fedora 35 and the 470xx nvidia driver from rpmfusion. 
  Rebooting after a kernel upgrade is still giving nouveau and lower 
screen resolution than expected.  sudo depmod -ae, followed by sudo 
systemctl reboot, has worked for me.

https://rpmfusion.org/CommonBugs

John P

[Tom Horsley]

On Wed, 6 Jul 2022 08:07:10 -0400
Neal Becker wrote:
> Any ideas?

You can examine /var/log/akmods/akmods.log to see the progress of the
driver build and make sure the build finished before rebooting.

[Roger Heflin]

From what I can tell the boot sequence on an nvidia and/or kernel
update is this:

Machine boots up.
Akmods begins building new driver and/or driver for new kernel.
Available old driver loads (if new kernel the driver will be nouveau,
if new nvidia this will be older nvidia) and Graphics start up.
Akmods finishes and installs the nvidia module.

On the machine I have that uses the rpmfusion nvidia, I login and
manually start up graphics, and I have been checking dmesg and waiting
the 3-5 minutes it takes for akmods to finish building the module..
If I do not wait and immediately login(text console) and start
graphics then I will get nouveau and lower screen resolutions.

Note modinfo <drivername> will show you the internal version number
that is installed.

On Wed, Jul 6, 2022 at 7:27 AM Tom Horsley  wrote:
> On Wed, 6 Jul 2022 08:07:10 -0400
> Neal Becker wrote:
> > Any ideas?
>
> You can examine /var/log/akmods/akmods.log to see the progress of the
> driver build and make sure the build finished before rebooting.
> _______________________________________________
> users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Lester Petrie]

On 7/6/2022 10:27 AM, Roger Heflin wrote:
> From what I can tell the boot sequence on an nvidia and/or kernel
> update is this:
>
> Machine boots up.
> Akmods begins building new driver and/or driver for new kernel.
> Available old driver loads (if new kernel the driver will be nouveau,
> if new nvidia this will be older nvidia) and Graphics start up.
> Akmods finishes and installs the nvidia module.
>
> On the machine I have that uses the rpmfusion nvidia, I login and
> manually start up graphics, and I have been checking dmesg and waiting
> the 3-5 minutes it takes for akmods to finish building the module..
> If I do not wait and immediately login(text console) and start
> graphics then I will get nouveau and lower screen resolutions.
>
> Note modinfo <drivername> will show you the internal version number
> that is installed.
>
> On Wed, Jul 6, 2022 at 7:27 AM Tom Horsley  wrote:
> > On Wed, 6 Jul 2022 08:07:10 -0400
> > Neal Becker wrote:
> > > Any ideas?
> >
> > You can examine /var/log/akmods/akmods.log to see the progress of the
> > driver build and make sure the build finished before rebooting.

When I install a new kernel, a new nvidia is pulled in as a dependency, 
and akmod is rebuilt before I reboot. I need to wait long enough for 
this to finish. I am currently still running Fedora 35. Is this not what 
will happen with Fedora 36?

[Tom Horsley]

On Wed, 6 Jul 2022 11:04:01 -0400
Lester Petrie wrote:
> I need to wait long enough for 
> this to finish. I am currently still running Fedora 35. Is this not what 
> will happen with Fedora 36?

That's what happens for me. The only difference is the log file
moved, so if you "tail -f" to see when it is done, you have to find the
log in a different place now (/var/log/akmods/akmods.log, used to be
in /var/cache). If you reboot before it finishes, then it will start building
it again when the system boots (but I never make it have to do that).

[Roger Heflin]

I manually do the update and reboot, and likely do not wait for all of
the kernel module rebuilds to get done.

Likely others are doing similar since they are having similar issues.

On Wed, Jul 6, 2022 at 10:23 AM Tom Horsley  wrote:
> On Wed, 6 Jul 2022 11:04:01 -0400
> Lester Petrie wrote:
> > I need to wait long enough for
> > this to finish. I am currently still running Fedora 35. Is this not what
> > will happen with Fedora 36?
>
> That's what happens for me. The only difference is the log file
> moved, so if you "tail -f" to see when it is done, you have to find the
> log in a different place now (/var/log/akmods/akmods.log, used to be
> in /var/cache). If you reboot before it finishes, then it will start building
> it again when the system boots (but I never make it have to do that).
> _______________________________________________
> users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

Hi.

On Wed, 06 Jul 2022 11:05:54 -0500 Roger Heflin wrote:
> I manually do the update and reboot, and likely do not wait for all of
> the kernel module rebuilds to get done.

To prevent that, you can enable the akmods-shutdown.service: it will build and
install all the kmods for all the installed kernel during the shutdown.

This service does;

ExecStart=/bin/true
    ExecStop=-/usr/sbin/akmods-shutdown

[Doug Herr]

On Wed, Jul 6, 2022, at 11:12 PM, [e-mail] wrote:
> Hi.
>
> On Wed, 06 Jul 2022 11:05:54 -0500 Roger Heflin wrote:
> > I manually do the update and reboot, and likely do not wait for all of
> > the kernel module rebuilds to get done.
>
> To prevent that, you can enable the akmods-shutdown.service: it will build and
> install all the kmods for all the installed kernel during the shutdown.
>
> This service does;
> ExecStart=/bin/true
> ExecStop=-/usr/sbin/akmods-shutdown

If dnf has a kernel update I check for the modules with my 'kerncheck' alias:

kerncheck: 	 aliased to ls -ltr /lib/modules/*/extra;echo;ls -ltr /lib/modules;echo;echo -n "Currently booted to: ";uname -r;echo

That format is for csh/tcsh.