Hi,
ATM I'm upgrading a remote server from F38 to F39 via SSH login.
The bash runs a screen, so if ssh or the network dies, the upgrade can continue.
I just noticed that even if it keeps running, it's useless because openssl und openssh do not get upgraded after another.
So, atm, the server has a sshd that says, that openssl is newer as the required openssl version. It does not start it nor can the running sshd, that runs the upgrade connection, fork a new instance. In other words: no login possible to rescue or recover the upgrade session remotely.
if that wasn't enough, imapd and other services do also refuse to work due to openssl version mismatch.
I can imagine, that this class of problems is not soley with openssl.
O== Changerequest
find a way to cluster packages close together that need openssl, to keep the risk of an inaccessible server small.
i.e. by grouping the packages i.e. "network" "office" etc.
best regards, Marius Schwarz
On Wed, May 22, 2024 at 4:34 PM Marius Schwarz fedoradev@cloud-foo.de wrote:
ATM I'm upgrading a remote server from F38 to F39 via SSH login.
The bash runs a screen, so if ssh or the network dies, the upgrade can continue.
I just noticed that even if it keeps running, it's useless because openssl und openssh do not get upgraded after another.
So, atm, the server has a sshd that says, that openssl is newer as the required openssl version. It does not start it nor can the running sshd, that runs the upgrade connection, fork a new instance. In other words: no login possible to rescue or recover the upgrade session remotely.
if that wasn't enough, imapd and other services do also refuse to work due to openssl version mismatch.
I can imagine, that this class of problems is not soley with openssl.
O== Changerequest
find a way to cluster packages close together that need openssl, to keep the risk of an inaccessible server small.
i.e. by grouping the packages i.e. "network" "office" etc.
Were you following the steps outlined in https://docs.fedoraproject.org/en-US/quick-docs/upgrading-fedora-offline ?
Am 22.05.24 um 17:48 schrieb Alexander Sosedkin:
On Wed, May 22, 2024 at 4:34 PM Marius Schwarzfedoradev@cloud-foo.de wrote:
Were you following the steps outlined in https://docs.fedoraproject.org/en-US/quick-docs/upgrading-fedora-offline ?
I'm under the impression, there is a small misunderstanding here: The upgrade worked as it should for the 23th time in a row :D
It's just, that > while < the upgrade process was running, there was the risk, that it or the connection fails, and in that case, there would not have been a way to fix it remotely because of the mismatch of openssl and openssh versions. (yes of course, an IPMI would be a way ).
The risk that this happens can be shrunken immense, if openssh and openssl would be updated in direct serial order.
Yes, this is only relevant for remote-upgrades, but Fedora is a very good server distribution and widely spread ;)
best regards, Marius Schwarz
On 5/22/24 3:29 PM, Marius Schwarz wrote:
Am 22.05.24 um 17:48 schrieb Alexander Sosedkin:
On Wed, May 22, 2024 at 4:34 PM Marius Schwarzfedoradev@cloud-foo.de wrote:
Were you following the steps outlined in https://docs.fedoraproject.org/en-US/quick-docs/upgrading-fedora-offline ?
I'm under the impression, there is a small misunderstanding here: The upgrade worked as it should for the 23th time in a row :D
It's just, that > while < the upgrade process was running, there was the risk, that it or the connection fails, and in that case, there would not have been a way to fix it remotely because of the mismatch of openssl and openssh versions. (yes of course, an IPMI would be a way ).
The risk that this happens can be shrunken immense, if openssh and openssl would be updated in direct serial order.
You shouldn't be doing release upgrades like this anyway and there is not likely to be any work to reduce that potential issue. You should be using the system-upgrade method to upgrade releases which would avoid this completely.
Am 23.05.2024 um 00:29 schrieb Marius Schwarz fedoradev@cloud-foo.de:
Am 22.05.24 um 17:48 schrieb Alexander Sosedkin:
On Wed, May 22, 2024 at 4:34 PM Marius Schwarz fedoradev@cloud-foo.de wrote:
Were you following the steps outlined in https://docs.fedoraproject.org/en-US/quick-docs/upgrading-fedora-offline ?
I'm under the impression, there is a small misunderstanding here: The upgrade worked as it should for the 23th time in a row :D
So, your server is running with F39 now?
It's just, that > while < the upgrade process was running, there was the risk, that it or the connection fails, and in that case, there would not have been a way to fix it remotely because of the mismatch of openssl and openssh versions. (yes of course, an IPMI would be a way ).
Unfortunately, I don’t understand, what exactly the problem is. If you follow the procedure as described in the Quick Docs document mentioned above, there is no network connection while the actual update process is running. And even if there was one, you have no way of intervening in the process.
Probably I overlook something in your mail. But I would really like to understand your issue. We are always looking to improve Fedora Server. And I’m running a bunch of servers in a remote data center, too, w/o access to the console (but a temporary KVM in case of emergency). So I have the update adventure every 6 months as well.
Yes, this is only relevant for remote-upgrades, but Fedora is a very good server distribution and widely spread ;)
Wholehearted Agreement :-)
-- Peter Boy https://fedoraproject.org/wiki/User:Pboy PBoy@fedoraproject.org
Timezone: CET (UTC+1) / CEST (UTC+2)
Fedora Server Edition Working Group member Fedora Docs team contributor and board member Java developer and enthusiast
Short summery:
we just talk about a small improvement. It works like it is now, but risks can me minimized with small adaptions to the package sort order in dnf while upgrading.
Am 23.05.24 um 06:14 schrieb Peter Boy:
I'm under the impression, there is a small misunderstanding here: The upgrade worked as it should for the 23th time in a row :D
So, your server is running with F39 now?
Yes.
It's just, that > while < the upgrade process was running, there was the risk, that it or the connection fails, and in that case, there would not have been a way to fix it remotely because of the mismatch of openssl and openssh versions. (yes of course, an IPMI would be a way ).
Unfortunately, I don’t understand, what exactly the problem is. If you follow the procedure as described in the Quick Docs document mentioned above, there is no network connection while the actual update process is running. And even if there was one, you have no way of intervening in the process.
I guess you think of "|dnf system-upgrade reboot" when you upgrade, but honestly, it's not something I wanne do on a server.
The idea behind it may be ok, if the server can go offline for a while, i wanne see what it does, when it does it, and if it failes, which happens to post-scripts a lot lately.(yes i report them, but most of the time nothing happens, sniff..)
| I use the dnf distro-sync as mentioned in the fedora dnf upgrade guides for the upgrade process since at least FC15 and it works..and works
It has different issues, like hanging processes that need manually restarts, for which you need a working secondary ssh connection ;) .
i.e. rpc services have shown this on several occations in the past.
A desktop pc in berlin did a reset in the middle of an upgrade started in the softwarecenter and ended up with DBUS server and client mismatch, so it did not even boot into multiusermode. Had a hard time to get the person infront of it to boot into a root shell and start sshd manually .. at 0:30 in the morning ;)
In the end, it does not matter if you do a remote or a locally started upgrade.. it can fail. If it fails with a vital part like DBUS or ssh having version mismatches, you are in big trouble.
i would define "vital" very specific as : "all you need for bash" "all you need for ssh" "all you need for dbus". This would make a "dirty" little patch possible that would upgrade those ten packages at best as first/last ones, one after the other.
On the positive side, I remember pressing CTRL+C in the wrong terminal window after 1000 upgrade steps and just restarting the upgrade without any issues.
Like i said, it's in any situation a good idea if depending vital-packages get upgraded in close proximation in the chain to minimize the risk.
If DNF Devs can take this into consideration while they are on implementing new stuff, maybe someone has a brilliant idea how to archive this. All it takes is the right idea in the right moment.
Probably I overlook something in your mail. But I would really like to understand your issue. We are always looking to improve Fedora Server.
It just minimizing risks, that can be minimized. I assume it's just a small change on the sort order algorithm of dnf.
Of course we do snapshots to minimize upgrade issues.
The thing with snapshots is, that while such a distro-sync takes up to 15 minutes, in case of SATA hds 45 minutes, that can mean X Minutes of email losses once reverted.
It would be easy to stop the services, but as most of you know, customers do not like when services are not available, because that 1 soooo important visitor just shows up in this exact moment ;) And of course they do not pay for that hot-stand-by that would solve this easily :D And silverblue isn't an option either here.
In the past 13 years of distro-syncing our serverfarm I had to revert 2 times. Means: "GOOD JOB Everyone! Thanks."
And I’m running a bunch of servers in a remote data center, too, w/o access to the console (but a temporary KVM in case of emergency). So I have the update adventure every 6 months as well.
...one who knows... :)
best regards, Marius Schwarz
On 5/22/24 16:33, Marius Schwarz wrote:
So, atm, the server has a sshd that says, that openssl is newer as the required openssl version. It does not start it nor can the running sshd, that runs the upgrade connection, fork a new instance. In other words: no login possible to rescue or recover the upgrade session remotely.
Why can't sshd fork a new instance? There is no need to read libraries when forking.
Regards.
Am 24.05.24 um 12:01 schrieb Roberto Ragusa:
On 5/22/24 16:33, Marius Schwarz wrote:
So, atm, the server has a sshd that says, that openssl is newer as the required openssl version. It does not start it nor can the running sshd, that runs the upgrade connection, fork a new instance. In other words: no login possible to rescue or recover the upgrade session remotely.
Why can't sshd fork a new instance?
you have ask the sshd devs.
First guess: they do not use fork().
Best regards, Marius Schwarz
Dnia Fri, May 24, 2024 at 01:33:44PM +0200, Marius Schwarz napisał(a):
Am 24.05.24 um 12:01 schrieb Roberto Ragusa:
On 5/22/24 16:33, Marius Schwarz wrote:
So, atm, the server has a sshd that says, that openssl is newer as the required openssl version. It does not start it nor can the running sshd, that runs the upgrade connection, fork a new instance. In other words: no login possible to rescue or recover the upgrade session remotely.
Why can't sshd fork a new instance?
you have ask the sshd devs.
First guess: they do not use fork().
The whole premise of this thread comes from doing incorrect, unsupported system upgrade on live system. Look, this method is even described with red Warning frame: https://docs.fedoraproject.org/en-US/quick-docs/upgrading-fedora-new-release...
There's no need to continue digressing wrt sshd behaviour.
On 5/25/24 08:47, Tomasz Torcz wrote:
Dnia Fri, May 24, 2024 at 01:33:44PM +0200, Marius Schwarz napisał(a):
Am 24.05.24 um 12:01 schrieb Roberto Ragusa:
Why can't sshd fork a new instance?
you have ask the sshd devs.
First guess: they do not use fork().
The whole premise of this thread comes from doing incorrect, unsupported system upgrade on live system. Look, this method is even described with red Warning frame: https://docs.fedoraproject.org/en-US/quick-docs/upgrading-fedora-new-release...
There's no need to continue digressing wrt sshd behaviour.
Who cares about what is considered a "supported" upgrade method today?
The interesting topic on the table is the robustness (or lack of) of the sshd daemon during an upgrade of its binaries and libraries. That daemon is critical for remotely managed systems and the developers usually take care of the implementation quality. Never noticed that you can "systemctl restart sshd" without losing existing ssh connections?
The answer to why is sshd failing to fork during ssl libs upgrade is worth investigating. It may lead to something to fix in the sshd code or in the way openssh is patched/packaged. (hasn't the xz backdoor thing given everybody a lesson on this?)
Regards.
On 5/28/24 12:14 AM, Roberto Ragusa wrote:
The interesting topic on the table is the robustness (or lack of) of the sshd daemon during an upgrade of its binaries and libraries. That daemon is critical for remotely managed systems and the developers usually take care of the implementation quality. Never noticed that you can "systemctl restart sshd" without losing existing ssh connections?
How is that relevant? Restarting sshd doesn't touch existing ssh client processes.
The answer to why is sshd failing to fork during ssl libs upgrade is worth investigating.
No, it's not, because there is no expectation of that happening. There is no means to even do this in any reasonable way.
It may lead to something to fix in the sshd code or in the way openssh is patched/packaged.
There is nothing to fix. The ssh process has already loaded the libraries, so it won't crash if you replace them. After you upgrade the libraries, you just need to restart it to get the new ones loaded. Any existing connections will still be processes running with the old libraries. What exactly are you expecting?
On 5/28/24 09:38, Samuel Sieb wrote:
There is nothing to fix. The ssh process has already loaded the libraries, so it won't crash if you replace them. After you upgrade the libraries, you just need to restart it to get the new ones loaded. Any existing connections will still be processes running with the old libraries. What exactly are you expecting?
I am expecting exactly what you have described, but the first mail in this thread is claiming an inability to fork, and that is (we are stating the same thing!) not expected. Why is the original poster saying that he is not able to reach the machine during the upgrade because of inability to fork sshd? I've become suspicious of maybe dlopen tricks failing near the moment a new connection triggers a fork, which would be interesting to be aware of, if any such case.
(note that you say "any existing connections" uses the old libraries, but indeed also new connections will use the old libraries, if the restart has not been done yet, so the claimed issue should definitely never happen)
Regards.
On 5/28/24 6:54 AM, Roberto Ragusa wrote:
Why is the original poster saying that he is not able to reach the machine during the upgrade because of inability to fork sshd?
I think you misunderstood. As far as I could tell he didn't have any issues. It was a theoretical question about something he thought might happen. And it also required that the dnf process would crash while doing the update.
-
Alexander Sosedkin -
Marius Schwarz -
Peter Boy -
Roberto Ragusa -
Samuel Sieb -
Tomasz Torcz