-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
== The Problem ==
It is very common for users to have systems with encrypted root partitions (or even just /var and /etc). This may be due to a personal concern for their data or a corporate policy mandating full-disk encryption. Disk encryption requires a password (or other more complicated credentials) be be presented just after the kernel is booted and before the drives can be mounted and their data read.
With the current implementation of the offline updates in Fedora, this leads to a very unpleasant user experience when updating. We offer two ways to perform offline updates in the default user environment of Fedora Workstation: "Install updates and reboot" or "Install updates and shut down".
With "Install updates and reboot", the behavior is as follows:
1) The system shuts down and initiates an ACPI reboot. 2) The system presents the kernel boot menu and then starts the updater kernel. 3) The system presents the user with a password prompt for the disk encryption (possibly more than one, if the system is configured with different passwords for different partitions). 4) The offline updates occur. 5) The system shuts down and initiates an ACPI reboot. 6) The system presents the kernel boot menu and then starts the standard (possibly updated) kernel. 7) The system presents the user with a password prompt for the disk encryption (possibly more than one, if the system is configured with different passwords for different partitions). 8) The system completes booting.
During this experience, the user has been required to enter their disk encryption password *twice*. The same is true for the "Install and shut down" case, except that the two passwords are separated by some actual wallclock time.
== Proposed Improvements ==
We could significantly improve this situation by allowing the system to drop directly from the interactive system into the updater environment without doing a full reboot or relaunching the kernel.
Lennart, would it be possible to set up a special systemd target for performing updates that would essentially stop all processes except for systemd and then apply the updates?
In an ideal world, it would then also be possible after update is completed to restore operation to the standard boot targets of systemd so that the system comes back up without having to perform a total reboot. The exceptional case would of course be that in which either the kernel, libc or systemd[1] needed to be updated, in which case a reboot could be performed.
In this scenario, we can reduce the number of encrypted disk challenges to at most a single one, and that only if absolutely minimal plumbing packages saw an update.
I'd very much like to hear from the plumbers on this matter.
[1] I'm told that this might not be necessary; that systemd can re-exec itself to pick up its own updates. That would reduce the scope presumably to "only the kernel" forcing reboots.
On Fri, 2014-09-12 at 10:46 -0400, Stephen Gallagher wrote:
== Proposed Improvements ==
We could significantly improve this situation by allowing the system to drop directly from the interactive system into the updater environment without doing a full reboot or relaunching the kernel.
Lennart, would it be possible to set up a special systemd target for performing updates that would essentially stop all processes except for systemd and then apply the updates?
In an ideal world, it would then also be possible after update is completed to restore operation to the standard boot targets of systemd so that the system comes back up without having to perform a total reboot. The exceptional case would of course be that in which either the kernel, libc or systemd[1] needed to be updated, in which case a reboot could be performed.
In this scenario, we can reduce the number of encrypted disk challenges to at most a single one, and that only if absolutely minimal plumbing packages saw an update.
I'd very much like to hear from the plumbers on this matter.
Yeah, I almost never use the reboot & install method. 90% of the packages being installed/updated seem foolish to need a reboot to update. I typically do a yum update manually and then if I notice glibc/kernel/systemd or other big packages do a reboot.
All my systems have disk encryption since some of our projects could potentially include people's private information. The latest way of updating is just plain annoying for an unknown gain.
I would *love* if it was improved.
On 12 September 2014 16:16, Nathanael d. Noblet nathanael@gnat.ca wrote:
Yeah, I almost never use the reboot & install method. 90% of the packages being installed/updated seem foolish to need a reboot to update.
I've been called worse that foolish I guess...
I typically do a yum update manually and then if I notice glibc/kernel/systemd or other big packages do a reboot.
That's just not safe. Have you ever had firefox open and done a firefox update? Widgets start disappearing, redraws start having weird artifects and then after a little while it just crashes. Other applications like LibreOffice behave the same. Anyone that says things like "the old version of the library stays in memory" obviously hasn't actually done much real-world GUI programming in the last decade, or runs any kind of secure desktop system. The *only* way to do this securely and safely in the system we have now is in a clean pre-boot environment, which is sad and crap UX, but still nevertheless true. When we have application sandboxing and a stable OS platform to use, we can certainly do something more sane, but until then we're just hacking around the problem.
What we could do is do updates on shutdown by basically killing everything except PID 1, and then restart everything, but even then that relies on no systemd or kernel updates being present. Solving the real problem is much harder (OS + Apps split) but we also need to slow the firehose of needless updates.
Richard
On Fri, 2014-09-12 at 16:47 +0100, Richard Hughes wrote:
On 12 September 2014 16:16, Nathanael d. Noblet nathanael@gnat.ca wrote:
Yeah, I almost never use the reboot & install method. 90% of the packages being installed/updated seem foolish to need a reboot to update.
I've been called worse that foolish I guess...
I typically do a yum update manually and then if I notice glibc/kernel/systemd or other big packages do a reboot.
That's just not safe. Have you ever had firefox open and done a firefox update? Widgets start disappearing, redraws start having weird artifects and then after a little while it just crashes. Other applications like LibreOffice behave the same. Anyone that says things like "the old version of the library stays in memory" obviously hasn't actually done much real-world GUI programming in the last decade, or runs any kind of secure desktop system. The *only* way to do this securely and safely in the system we have now is in a clean pre-boot environment, which is sad and crap UX, but still nevertheless true. When we have application sandboxing and a stable OS platform to use, we can certainly do something more sane, but until then we're just hacking around the problem.
So I don't use Firefox anymore but I do know back in the day if we had FF open when we updated it would do a double request for each page/form. However when updating we just restarted FF and it would work fine after that. I've never noticed any other issues than FF but like I said I don't use it anymore.
Granted that doesn't matter obviously we don't want that kind of behaviour.
I am curious though. Everyone says the only way to do it securely and safely is with nothing running. Why can't updates be applied with stuff running prior to a reboot?
Am 12.09.2014 um 18:33 schrieb Nathanael d. Noblet:
So I don't use Firefox anymore but I do know back in the day if we had FF open when we updated it would do a double request for each page/form. However when updating we just restarted FF and it would work fine after that. I've never noticed any other issues than FF but like I said I don't use it anymore.
Granted that doesn't matter obviously we don't want that kind of behaviour.
I am curious though. Everyone says the only way to do it securely and safely is with nothing running. Why can't updates be applied with stuff running prior to a reboot?
1 out of a million cases needs offline updates
really - the only good at it is that you can stick at using YUM and decide what you have to do at your own - rarely updates really require a reboot
* lsof | grep DEL | grep /usr and restart services on servers * logout/login and the above on workstations
On Fri, Sep 12, 2014 at 06:37:07PM +0200, Reindl Harald wrote:
- lsof | grep DEL | grep /usr and restart services on servers
There is a convenient script called "needs-restarting" to do this. Checkrestart on debian even suggest potential init scripts that can be used to restart the services.
Am 12.09.2014 um 18:58 schrieb Till Maas:
On Fri, Sep 12, 2014 at 06:37:07PM +0200, Reindl Harald wrote:
- lsof | grep DEL | grep /usr and restart services on servers
There is a convenient script called "needs-restarting" to do this. Checkrestart on debian even suggest potential init scripts that can be used to restart the services
never worked relieable here on multiple machines
it often showed nothing where i knew the thing which should be restarted without looking and "lsof" proved it
On 09/12/2014 07:09 PM, Reindl Harald wrote:
never worked relieable here on multiple machines
it often showed nothing where i knew the thing which should be restarted without looking and "lsof" proved it
I am one of those guys who refuse to reboot after each upgrade (and it works for me) and needs-restarting is ugly and insufficient to me.
Therefore I initiated this project: https://github.com/FrostyX/tracer http://copr.fedoraproject.org/coprs/frostyx/tracer/
It is still not finished and ready for announcement, but if you are looking for some other way than offline-upgrade, this might be worth of participating.
On Mon, Sep 15, 2014 at 09:50:36AM +0200, Miroslav Suchý wrote:
On 09/12/2014 07:09 PM, Reindl Harald wrote:
never worked relieable here on multiple machines
it often showed nothing where i knew the thing which should be restarted without looking and "lsof" proved it
I am one of those guys who refuse to reboot after each upgrade (and it works for me) and needs-restarting is ugly and insufficient to me.
Therefore I initiated this project: https://github.com/FrostyX/tracer http://copr.fedoraproject.org/coprs/frostyx/tracer/
It is still not finished and ready for announcement, but if you are looking for some other way than offline-upgrade, this might be worth of participating.
It wasn't clear to me how tracer works for non-C programs.
However there was some Red Hat only discussion recently about how to do this for Python programs, with minimal overhead. Below I'm just reproducing a technique (untested) that I think will work for Python.
It requires a small patch to the Python interpreter, and a similar patch to any other language interpreters (eg. Perl, Ruby).
Rich.
------- For each module (*.py or *.pyc) that it imports, have it mmap the first page of that file into its memory.
The mmap would be PROT_NONE because it's not actually used, and the associated file descriptor should be closed.
This will appear in /proc/PID/maps, with a "(deleted)" flag if the underlying file gets deleted (and hence the process needs restarting).
The cost should be almost nothing:
- 4K of virtual memory, no real memory - an extra mmap syscall on import - an extra segment in the kernel's VM AVL -------
Rich.
On 09/15/2014 10:06 AM, Richard W.M. Jones wrote:
It wasn't clear to me how tracer works for non-C programs.
https://github.com/FrostyX/tracer/commit/4abfc4ecbc6d1d4cd89b7162e1ba3f63088...
Which basicaly checkout output of `ps` and if there is e.g. python as executable, it will check for arguments and use those. But I agree that interpreted languages are problem, because they open the file, read it and close the handler. So there are no footsteps to track.
However there was some Red Hat only discussion recently about how to do this for Python programs, with minimal overhead. Below I'm just reproducing a technique (untested) that I think will work for Python.
It requires a small patch to the Python interpreter, and a similar patch to any other language interpreters (eg. Perl, Ruby).
Rich.
For each module (*.py or *.pyc) that it imports, have it mmap the first page of that file into its memory.
The mmap would be PROT_NONE because it's not actually used, and the associated file descriptor should be closed.
This will appear in /proc/PID/maps, with a "(deleted)" flag if the underlying file gets deleted (and hence the process needs restarting).
The cost should be almost nothing:
- 4K of virtual memory, no real memory
- an extra mmap syscall on import
- an extra segment in the kernel's VM AVL
Very nice. Is there some bugzilla RFE report for this? Or should I file it?
Every of the script is based on assumption that you already read some library/unit whatever. But that is not enough. I wonder how you want to detect that you need restart in case that I have something like this:
$ ls foo.rb bar.rb
$ cat foo.rb
def some_function require 'bar' end
And now
1) I run some application, which loads my foo.rb file. 2) I later update the package which removes bar.rb file. 3) And I call some_function which fails due to missing bar.rb
There is no universal and reliable way how to detect this scenario IMO.
Vít
Dne 15.9.2014 10:06, Richard W.M. Jones napsal(a):
On Mon, Sep 15, 2014 at 09:50:36AM +0200, Miroslav Suchý wrote:
On 09/12/2014 07:09 PM, Reindl Harald wrote:
never worked relieable here on multiple machines
it often showed nothing where i knew the thing which should be restarted without looking and "lsof" proved it
I am one of those guys who refuse to reboot after each upgrade (and it works for me) and needs-restarting is ugly and insufficient to me.
Therefore I initiated this project: https://github.com/FrostyX/tracer http://copr.fedoraproject.org/coprs/frostyx/tracer/
It is still not finished and ready for announcement, but if you are looking for some other way than offline-upgrade, this might be worth of participating.
It wasn't clear to me how tracer works for non-C programs.
However there was some Red Hat only discussion recently about how to do this for Python programs, with minimal overhead. Below I'm just reproducing a technique (untested) that I think will work for Python.
It requires a small patch to the Python interpreter, and a similar patch to any other language interpreters (eg. Perl, Ruby).
Rich.
For each module (*.py or *.pyc) that it imports, have it mmap the first page of that file into its memory.
The mmap would be PROT_NONE because it's not actually used, and the associated file descriptor should be closed.
This will appear in /proc/PID/maps, with a "(deleted)" flag if the underlying file gets deleted (and hence the process needs restarting).
The cost should be almost nothing:
- 4K of virtual memory, no real memory
- an extra mmap syscall on import
- an extra segment in the kernel's VM AVL
Rich.
On 09/15/2014 10:57 AM, Vít Ondruch wrote:
Every of the script is based on assumption that you already read some library/unit whatever. But that is not enough. I wonder how you want to detect that you need restart in case that I have something like this:
$ ls foo.rb bar.rb
$ cat foo.rb
def some_function require 'bar' end
And now
- I run some application, which loads my foo.rb file.
- I later update the package which removes bar.rb file.
- And I call some_function which fails due to missing bar.rb
Indeed, would foo.rb and bar.rb comes from different packages, then there is really no way.
There is no universal and reliable way how to detect this scenario IMO.
Well, if you are operator of nuclear power plant, then I understand the need of reboot after each upgrade.
But *I* do not want to reboot after each upgrade. Those crashes will be 0.00001% of all crashes on my workstation, which is less PITA than rebooting because I upgraded 'foo-doc' package.
On 15 September 2014 13:06, Miroslav Suchý msuchy@redhat.com wrote:
But *I* do not want to reboot after each upgrade. Those crashes will be 0.00001% of all crashes on my workstation
I think you might change your mind when it's you're the one that has to triage those ABRT-reported bugs. Also, consoling users with lost data gets tiring really quick.
Richard
Am 16.09.2014 um 10:50 schrieb Richard Hughes:
On 15 September 2014 13:06, Miroslav Suchý msuchy@redhat.com wrote:
But *I* do not want to reboot after each upgrade. Those crashes will be 0.00001% of all crashes on my workstation
I think you might change your mind when it's you're the one that has to triage those ABRT-reported bugs. Also, consoling users with lost data gets tiring really quick
well, users which want full control typically disable or even uninstall ABRT - as long offline updates are not becoming mandatory all is fine
[harry@srv-rhsoft:~]$ rpm -qa | grep -i ABRT [harry@srv-rhsoft:~]$
On Mon, Sep 15, 2014 at 10:57:13AM +0200, Vít Ondruch wrote:
Every of the script is based on assumption that you already read some library/unit whatever. But that is not enough. I wonder how you want to detect that you need restart in case that I have something like this:
$ ls foo.rb bar.rb
$ cat foo.rb
def some_function require 'bar' end
And now
- I run some application, which loads my foo.rb file.
- I later update the package which removes bar.rb file.
- And I call some_function which fails due to missing bar.rb
How is this not 'foo' simply being broken? ie. Not expressing its needs properly in its RPM dependencies?
It would still have been broken even with a reboot.
Rich.
Am 15.09.2014 um 14:28 schrieb Richard W.M. Jones:
On Mon, Sep 15, 2014 at 10:57:13AM +0200, Vít Ondruch wrote:
- I run some application, which loads my foo.rb file.
- I later update the package which removes bar.rb file.
- And I call some_function which fails due to missing bar.rb
How is this not 'foo' simply being broken? ie. Not expressing its needs properly in its RPM dependencies?
It would still have been broken even with a reboot
no - why should it?
'foo' is loaded in memory, updated and now has different dependencies no longer require 'bar.rb' but your running version still do
yes, such corner cases are possible, regardless no reason for go the windows way and reboot after any minor update
On Mon, Sep 15, 2014 at 02:40:34PM +0200, Reindl Harald wrote:
Am 15.09.2014 um 14:28 schrieb Richard W.M. Jones:
On Mon, Sep 15, 2014 at 10:57:13AM +0200, Vít Ondruch wrote:
- I run some application, which loads my foo.rb file.
- I later update the package which removes bar.rb file.
- And I call some_function which fails due to missing bar.rb
How is this not 'foo' simply being broken? ie. Not expressing its needs properly in its RPM dependencies?
It would still have been broken even with a reboot
no - why should it?
'foo' is loaded in memory, updated and now has different dependencies no longer require 'bar.rb' but your running version still do
Please read closely. 'foo' has *not* been updated.
If 'foo' had been updated, we would have spotted it and restarted that process using my technique outlined in the previous email.
Rich.
Am 15.09.2014 um 14:44 schrieb Richard W.M. Jones:
On Mon, Sep 15, 2014 at 02:40:34PM +0200, Reindl Harald wrote:
Am 15.09.2014 um 14:28 schrieb Richard W.M. Jones:
On Mon, Sep 15, 2014 at 10:57:13AM +0200, Vít Ondruch wrote:
- I run some application, which loads my foo.rb file.
- I later update the package which removes bar.rb file.
- And I call some_function which fails due to missing bar.rb
How is this not 'foo' simply being broken? ie. Not expressing its needs properly in its RPM dependencies?
It would still have been broken even with a reboot
no - why should it?
'foo' is loaded in memory, updated and now has different dependencies no longer require 'bar.rb' but your running version still do
Please read closely. 'foo' has *not* been updated.
If 'foo' had been updated, we would have spotted it and restarted that process using my technique outlined in the previous email
cross deps coming in my mind foo -> library -> library -> library
* the first maybe already loaded * also loaded the second one in a previous call * that version relies on teh third one for some operations * in a update the deps have changed
so you may have a mix with different dep-chains in memory and some parts used the first time from disk with unexpected results
On Mon, Sep 15, 2014 at 02:49:34PM +0200, Reindl Harald wrote:
Am 15.09.2014 um 14:44 schrieb Richard W.M. Jones:
On Mon, Sep 15, 2014 at 02:40:34PM +0200, Reindl Harald wrote:
Am 15.09.2014 um 14:28 schrieb Richard W.M. Jones:
On Mon, Sep 15, 2014 at 10:57:13AM +0200, Vít Ondruch wrote:
- I run some application, which loads my foo.rb file.
- I later update the package which removes bar.rb file.
- And I call some_function which fails due to missing bar.rb
How is this not 'foo' simply being broken? ie. Not expressing its needs properly in its RPM dependencies?
It would still have been broken even with a reboot
no - why should it?
'foo' is loaded in memory, updated and now has different dependencies no longer require 'bar.rb' but your running version still do
Please read closely. 'foo' has *not* been updated.
If 'foo' had been updated, we would have spotted it and restarted that process using my technique outlined in the previous email
cross deps coming in my mind foo -> library -> library -> library
- the first maybe already loaded
- also loaded the second one in a previous call
- that version relies on teh third one for some operations
- in a update the deps have changed
so you may have a mix with different dep-chains in memory and some parts used the first time from disk with unexpected results
I have absolutely no idea what you are talking about.
Rich.
Dne 15.9.2014 14:28, Richard W.M. Jones napsal(a):
On Mon, Sep 15, 2014 at 10:57:13AM +0200, Vít Ondruch wrote:
Every of the script is based on assumption that you already read some library/unit whatever. But that is not enough. I wonder how you want to detect that you need restart in case that I have something like this:
$ ls foo.rb bar.rb
$ cat foo.rb
def some_function require 'bar' end
And now
- I run some application, which loads my foo.rb file.
- I later update the package which removes bar.rb file.
- And I call some_function which fails due to missing bar.rb
How is this not 'foo' simply being broken?
They might come from different packages. Or there might be also another level of requires, where bar.rb requires by baz.rb. In case that bar.rb stays and baz.rb is removed, you still cannot predict that this will fail in the future, since neither of these files was loaded before.
Or there might be another example of code with similar issues:
$ cat foo.rb
def some_function $files.each {|f| require f} end
$files = Dir.glob('*.rb')
I.e. during initialization, you list available files and you want to load them later, but at that moment, they are not there already.
ie. Not expressing its needs properly in its RPM dependencies?
We typically express dependencies between packages, the separate library files are typically considered just implementation detail. But I agree that it would be more precise to express dependencies between separate library files, instead of packages. Anyway, this is case of dynamic loading, so I am not sure how you would specify the dependencies.
Vít
On Mon, Sep 15, 2014 at 04:07:39PM +0200, Vít Ondruch wrote:
Dne 15.9.2014 14:28, Richard W.M. Jones napsal(a):
On Mon, Sep 15, 2014 at 10:57:13AM +0200, Vít Ondruch wrote:
Every of the script is based on assumption that you already read some library/unit whatever. But that is not enough. I wonder how you want to detect that you need restart in case that I have something like this:
$ ls foo.rb bar.rb
$ cat foo.rb
def some_function require 'bar' end
And now
- I run some application, which loads my foo.rb file.
- I later update the package which removes bar.rb file.
- And I call some_function which fails due to missing bar.rb
How is this not 'foo' simply being broken?
They might come from different packages.
OK, in which case the package that needs bar was broken because it didn't express that need in its dependencies, nor guard against the possibility that bar was not installed.
Or there might be also another level of requires, where bar.rb requires by baz.rb. In case that bar.rb stays and baz.rb is removed, you still cannot predict that this will fail in the future, since neither of these files was loaded before.
In which case bar.rb was similarly broken, for the same reason as above.
Or there might be another example of code with similar issues:
$ cat foo.rb
def some_function $files.each {|f| require f} end
$files = Dir.glob('*.rb')
I.e. during initialization, you list available files and you want to load them later, but at that moment, they are not there already.
This code is still by any measure broken. There are lots of ways that such code could fail to work.
Plainly what I'm trying to say is: If the potentially insecure code has been loaded into a Python process, and the python interpreter has the small modification that I suggested, then we will be able to detect that the insecure code is loaded into memory and flag the process/service as needing to be restarted.
That's all.
Rich.
On Fri, 12.09.14 18:37, Reindl Harald (h.reindl@thelounge.net) wrote:
Am 12.09.2014 um 18:33 schrieb Nathanael d. Noblet:
So I don't use Firefox anymore but I do know back in the day if we had FF open when we updated it would do a double request for each page/form. However when updating we just restarted FF and it would work fine after that. I've never noticed any other issues than FF but like I said I don't use it anymore.
Granted that doesn't matter obviously we don't want that kind of behaviour.
I am curious though. Everyone says the only way to do it securely and safely is with nothing running. Why can't updates be applied with stuff running prior to a reboot?
1 out of a million cases needs offline updates
really - the only good at it is that you can stick at using YUM and decide what you have to do at your own - rarely updates really require a reboot
- lsof | grep DEL | grep /usr and restart services on servers
Well, some deps are not visible like that, because they do not involve continuous mappings or open fds.
Moreover, it won't help you much anyway, as some daemons are not restarble right now, most prominently dbus-daemon.
And strictly speaking as you cannot restart all daemons at the very same instant, or even at the same instant as you install the new files and remove the old ones you will always have races where daemons might make use of resources or interfaces that are either newer than what they expect or older.
offline updates are really about make updates fully reliable. Yes, in most cases a "yum update" during runtime works well enough, and yes, I usually do my updates that way too. But I am actually able to help myself if something goes wrong. And so are you.
Offline updates are more for the cases where things need to be reliable, because no well educated admin is available to instantly fix things. Possibly because the machine is used by noobs only, or because the machine is buried somewhere under the see, or where so many instances of the machine are running that a human admins don't scale.
Hope that makes some sense.
Lennart
Am 21.10.2014 um 22:08 schrieb Lennart Poettering:
On Fri, 12.09.14 18:37, Reindl Harald (h.reindl@thelounge.net) wrote:
1 out of a million cases needs offline updates
really - the only good at it is that you can stick at using YUM and decide what you have to do at your own - rarely updates really require a reboot
- lsof | grep DEL | grep /usr and restart services on servers
Well, some deps are not visible like that, because they do not involve continuous mappings or open fds.
may be true but in practice no problem over many years
Moreover, it won't help you much anyway, as some daemons are not restarble right now, most prominently dbus-daemon
you repeat that again and again while i restart dbus over years on headless machines for web/file/db-servers and frankly before F15 even messagebus was completly disabled on all that machines
And strictly speaking as you cannot restart all daemons at the very same instant, or even at the same instant as you install the new files and remove the old ones you will always have races where daemons might make use of resources or interfaces that are either newer than what they expect or older.
interesting is that not so long ago there where just not much such dependencies - mandatory presence of dbus is very recent
other services like some webapp talking to a db-server? frankly i wrote 10 years ago db-layers to wait and retry so you can restart the db server after an update
offline updates are really about make updates fully reliable. Yes, in most cases a "yum update" during runtime works well enough, and yes, I usually do my updates that way too. But I am actually able to help myself if something goes wrong. And so are you.
true
Offline updates are more for the cases where things need to be reliable, because no well educated admin is available to instantly fix things. Possibly because the machine is used by noobs only, or because the machine is buried somewhere under the see, or where so many instances of the machine are running that a human admins don't scale
Hope that makes some sense
yes, but keep in mind not introduce more and more dependencies to make them mandatory somewhere in the future
On 10/21/2014 10:08 PM, Lennart Poettering wrote:
Offline updates are more for the cases where things need to be reliable, because no well educated admin is available to instantly fix things.
I will print it an pin up on my notice board.
And the implication is that offline updates are not for readers of devel@lists.fedoraproject.org
On Wed, Oct 22, 2014 at 08:48:59AM +0200, Miroslav Suchý wrote:
Offline updates are more for the cases where things need to be reliable, because no well educated admin is available to instantly fix things.
I will print it an pin up on my notice board.
And the implication is that offline updates are not for readers of devel@lists.fedoraproject.org
I don't think that's very fair, without the context. First, of course, we're developing for more than just ourselves. And second, this isn't a reversible statement: just because offline updates primarily target one user type doesn't mean that other user types can't or shouldn't use it.
Why do I care about this? The non-techie user wants less rebooting too. I'd love to see the updates toolchain get more smarts about recognizing when an update is "safe" (or at least "safer") and not reboot in those cases. That's possible but would be an investment of work. In the meantime, "offline updates if you want simple and guaranteed", "use yum or dnf online if you're able to handle unlikely but possible consequences" seems workable enough.
On Fri, Sep 12, 2014 at 10:33:13AM -0600, Nathanael d. Noblet wrote:
I am curious though. Everyone says the only way to do it securely and safely is with nothing running. Why can't updates be applied with stuff running prior to a reboot?
There's no reason, apart from the kernel. You're doing it right already.
Rich.
Am 12.09.2014 um 23:36 schrieb Richard W.M. Jones:
On Fri, Sep 12, 2014 at 10:33:13AM -0600, Nathanael d. Noblet wrote:
I am curious though. Everyone says the only way to do it securely and safely is with nothing running. Why can't updates be applied with stuff running prior to a reboot?
There's no reason, apart from the kernel. You're doing it right already
well, some libraries like glibc leads also in better reboot since they are used by nearly any component, but for 99% of all updates a reboot is a joke since you easily can restart servcices with nearly zero downtime
systemctl condrestart service1 service2 service3
condrestart since i distribute such commands over the whole infrsastructure after tests and deployment and not every installed service is enabled on all hosts
On Sep 12, 2014, at 9:47 AM, Richard Hughes hughsient@gmail.com wrote:
The *only* way to do this securely and safely in the system we have now is in a clean pre-boot environment,
Mostly clean post-boot environment, with the system we have now?
What we could do is do updates on shutdown by basically killing everything except PID 1, and then restart everything, but even then that relies on no systemd or kernel updates being present.
Even if the system is fully rebooted once, after the update is done in a post-boot environment, it's half the reboots needed now.
One step up from this would be a way for packages to contain metadata indicating they need a reboot after an update; if none require it, then isolate graphical.target rather than a reboot.
Chris Murphy
On Sat, Sep 13, 2014 at 3:32 AM, Chris Murphy lists@colorremedies.com wrote:
On Sep 12, 2014, at 9:47 AM, Richard Hughes hughsient@gmail.com wrote:
The *only* way to do this securely and safely in the system we have now is in a clean pre-boot environment,
Mostly clean post-boot environment, with the system we have now?
What we could do is do updates on shutdown by basically killing everything except PID 1, and then restart everything, but even then that relies on no systemd or kernel updates being present.
Even if the system is fully rebooted once, after the update is done in a post-boot environment, it's half the reboots needed now.
One step up from this would be a way for packages to contain metadata indicating they need a reboot after an update; if none require it, then isolate graphical.target rather than a reboot.
One step up from this would be something like a kpatch process in rpm combined with packaged metadata that replaces in-memory modules so that reboots wouldn't be necessary. Yeh, probably impossible.
Fred
On 09/13/2014 03:59 AM, Fred New wrote:
One step up from this would be something like a kpatch process in rpm combined with packaged metadata that replaces in-memory modules so that reboots wouldn't be necessary. Yeh, probably impossible.
This has almost certainly already been considered by people smarter than me, but it occurs to me that there's no reason that kexec couldn't some- how pass LUKS/dm-crypt keys to the new kernel.
the keys are in kernelspace IIRC and thus updated / passed on initrd /initramfs updates and kernel updates
Corey W Sheldon Freelance IT Consultant, Multi-Discipline Tutor 310.909.7672 www.facebook.com/1stclassmobileshine
On Sat, Sep 13, 2014 at 7:01 PM, Ian Pilcher arequipeno@gmail.com wrote:
On 09/13/2014 03:59 AM, Fred New wrote:
One step up from this would be something like a kpatch process in rpm combined with packaged metadata that replaces in-memory modules so that reboots wouldn't be necessary. Yeh, probably impossible.
This has almost certainly already been considered by people smarter than me, but it occurs to me that there's no reason that kexec couldn't some- how pass LUKS/dm-crypt keys to the new kernel.
--
Ian Pilcher arequipeno@gmail.com
-------- "I grew up before Mark Zuckerberg invented friendship" --------
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
On 09/12/2014 05:47 PM, Richard Hughes wrote:
That's just not safe. Have you ever had firefox open and done a firefox update? Widgets start disappearing, redraws start having weird artifects and then after a little while it just crashes. Other applications like LibreOffice behave the same. Anyone that says things like "the old version of the library stays in memory" obviously hasn't actually done much real-world GUI programming in the last decade, or runs any kind of secure desktop system.
You are basically saying that "modern" software is just breaking things which were perfectly solved decades ago. If applications would just use libraries correctly, the kernel would be able to let parts of deleted files be available for lazy loading.
On 16 September 2014 10:36, Roberto Ragusa mail@robertoragusa.it wrote:
If applications would just use libraries correctly, the kernel would be able to let parts of deleted files be available for lazy loading.
Sure, as long as all[1] the resources were either open()d when the user started the program, or linked in as resources into the library itself then it works fine. You can certainly design an application that does this correctly, it's just that very few do. A lot of GNOME apps compile in resources into the binary itself, using GResource, and this very much helps live updating.
Richard
[1] Including help files, all clipart, all icons, all .ui files, and all the .css for all the dialogs.
Dne 16.9.2014 v 11:48 Richard Hughes napsal(a):
On 16 September 2014 10:36, Roberto Ragusa mail@robertoragusa.it wrote:
If applications would just use libraries correctly, the kernel would be able to let parts of deleted files be available for lazy loading.
Sure, as long as all[1] the resources were either open()d when the user started the program, or linked in as resources into the library itself then it works fine. You can certainly design an application that does this correctly, it's just that very few do. A lot of GNOME apps compile in resources into the binary itself, using GResource, and this very much helps live updating.
Just a thought - but wouldn't be better spend time to enlighten Gnome/Firefox developers how to write applications in a way the could be upgraded runtime, instead of forcing users to reboot machines which seems seriously ridiculous.
And if we can't fix Gnome - then let's have there daemon/systemd service waiting in background till the 'problematic' app is still running - and run upgrade when the app is finally closed (eventually kindly ask user to close the app in case of some CVE....)
Zdenek
On 16 September 2014 10:55, Zdenek Kabelac zkabelac@redhat.com wrote:
Just a thought - but wouldn't be better spend time to enlighten Gnome/Firefox developers how to write applications in a way the could be upgraded runtime
So, it's not just the application, it's every application and D-Bus service the application uses. Even glibc opens files-as-resources at runtime. The much bigger issues is if you're using a D-Bus service like most applications seem to do (and most use quite a few system and session, directly and indirectly) then you've also got to co-ordinate and handle changing D-Bus API (which typically isn't versioned). This means you've got to restart the system DBus service at the right time in the on-line update, which means co-ordinating with any other open sessions on the system. Most desktop apps don't just sit in islands on their own. It's probably okay to live-update powertop on a single-user system when it's not being run, but anything much more integrated than that just isn't going to work very well.
Richard
Am 16.09.2014 um 12:21 schrieb Richard Hughes:
On 16 September 2014 10:55, Zdenek Kabelac zkabelac@redhat.com wrote:
Just a thought - but wouldn't be better spend time to enlighten Gnome/Firefox developers how to write applications in a way the could be upgraded runtime
So, it's not just the application, it's every application and D-Bus service the application uses. Even glibc opens files-as-resources at runtime. The much bigger issues is if you're using a D-Bus service like most applications seem to do (and most use quite a few system and session, directly and indirectly) then you've also got to co-ordinate and handle changing D-Bus API (which typically isn't versioned). This means you've got to restart the system DBus service at the right time in the on-line update, which means co-ordinating with any other open sessions on the system. Most desktop apps don't just sit in islands on their own. It's probably okay to live-update powertop on a single-user system when it's not being run, but anything much more integrated than that just isn't going to work very well
and the other side of the story are some hundret dist-upgrades i made in the past years with YUM (in a screen session for safety) while just continue to browse the web and read / answer email
On 16 September 2014 11:26, Reindl Harald h.reindl@thelounge.net wrote:
and the other side of the story are some hundret dist-upgrades i made in the past years with YUM (in a screen session for safety) while just continue to browse the web and read / answer email
So you're saying my technical analysis is incorrect because it happens to work for you? I've triaged many bugs to do with online and offline update failures, and if we're going to say that we actually care about the users data, it becomes increasingly hard to defend the "old" way of doing it. I'm sure I could find numerous bugs numbers where doing an online update made the session/terminal crash which of course leaves you with duplicate packages on your system which may or may not be fixable.
Richard
On 09/16/2014 06:33 AM, Richard Hughes wrote:
I've triaged many bugs to do with online and offline update failures, and if we're going to say that we actually care about the users data, it becomes increasingly hard to defend the "old" way of doing it. I'm sure I could find numerous bugs numbers where doing an online update made the session/terminal crash which of course leaves you with duplicate packages on your system which may or may not be fixable. Richard
OK, but this is means that we painted ourselves in the corner---something is wrong if my Android phone, which I don't have to reboot for updates, has higher uptime than my computer.
We are in a bind: on one hand, the best security practice is to upgrade daily to avoid emerging vulnerabilities; on the other hand daily reboots aren't really a nutritious alternative. Something has to give---which one do we do:
- create a separate daily security upgrade stream, curated to not require reboots if at all possible
- follow Microsoft and do a fixed 'patch Tuesday' schedule instead of ASAP updates
- rewrite Linux or at least Gnome/DBus for safe updates :)
On Tue, Sep 16, 2014 at 8:29 AM, Przemek Klosowski przemek.klosowski@nist.gov wrote:
On 09/16/2014 06:33 AM, Richard Hughes wrote:
I've triaged many bugs to do with online and offline update failures, and if we're going to say that we actually care about the users data, it becomes increasingly hard to defend the "old" way of doing it. I'm sure I could find numerous bugs numbers where doing an online update made the session/terminal crash which of course leaves you with duplicate packages on your system which may or may not be fixable. Richard
OK, but this is means that we painted ourselves in the corner---something is wrong if my Android phone, which I don't have to reboot for updates, has higher uptime than my computer.
You don't have to restart your phone for app updates, but the app itself is restarted. For Android updates, it does require you to reboot the phone. Android updates happen rather infrequently. Your analogy is slightly flawed.
We are in a bind: on one hand, the best security practice is to upgrade daily to avoid emerging vulnerabilities; on the other hand daily reboots aren't really a nutritious alternative. Something has to give---which one do we do:
- create a separate daily security upgrade stream, curated to not require
reboots if at all possible
This has been suggested before and it isn't a terrible idea. It requires people to work on it though.
- follow Microsoft and do a fixed 'patch Tuesday' schedule instead of ASAP
updates
There have been suggestions around having "levels" of updates. A firehose level, which is the equivalent of today, and then a batched level, etc. Also needs people to work on it.
josh
On 16 September 2014 13:29, Przemek Klosowski przemek.klosowski@nist.gov wrote:
OK, but this is means that we painted ourselves in the corner---something is wrong if my Android phone, which I don't have to reboot for updates, has higher uptime than my computer.
Right, we certainly have. By not having an OS / app layering policy we've basically lumped everything into the OS layer. We're now as good as Android, when you upgrade releases that you basically do an offline update like we do in GNOME.
The only way to do application online updating (IMO) is to solve four things:
* Defining an OS platform * Not breaking platform API * Sandboxing * Per-user applications
Richard
On 2014-09-16, Richard Hughes hughsient@gmail.com wrote:
The much bigger issues is if you're using a D-Bus service like most applications seem to do (and most use quite a few system and session, directly and indirectly) then you've also got to co-ordinate and handle changing D-Bus API (which typically isn't versioned).
Maybe it's time to version the API.
Look at microkernel based systems which utilize messaging heavily and the API consumers (applications or another subsystems) have to be prepared for spurious API provider (server) restarts.
A server can refuse a message any time (especially if it does not understand the request). Simple operations are usualy implemented as a sequence of requests and responses where initial request obtains a descriptor (a session, a handle) and subsequent requests passe it as context (a session) identifier. If the server is restarted in between, the context identifier becomes unrecognized and it's up to the application to deal with it.
Just the fact that somebody calls functions over dbus instead of jumping into a shared library do not free them from maintaing API.
-- Petr
On Tue, 16.09.14 13:35, Petr Pisar (ppisar@redhat.com) wrote:
On 2014-09-16, Richard Hughes hughsient@gmail.com wrote:
The much bigger issues is if you're using a D-Bus service like most applications seem to do (and most use quite a few system and session, directly and indirectly) then you've also got to co-ordinate and handle changing D-Bus API (which typically isn't versioned).
Maybe it's time to version the API.
Look at microkernel based systems which utilize messaging heavily and the API consumers (applications or another subsystems) have to be prepared for spurious API provider (server) restarts.
A server can refuse a message any time (especially if it does not understand the request). Simple operations are usualy implemented as a sequence of requests and responses where initial request obtains a descriptor (a session, a handle) and subsequent requests passe it as context (a session) identifier. If the server is restarted in between, the context identifier becomes unrecognized and it's up to the application to deal with it.
Just the fact that somebody calls functions over dbus instead of jumping into a shared library do not free them from maintaing API.
Well, the theory for this might be great. But reality tells us that code that isn't regularly tested tends to be broken. Hence: the assumption it would be reasonably possible to comprehensively test all kinds of updates between any combination of software versions, executed in any order, simultaneously with the user using the machine, then is simply wrong. You explode the test matrix, and without testing such upgrades will never be reliable. The offline update logic is about making the test matrix smaller, and adding determinism where it normally is missing.
Lennart
Dne 16.9.2014 v 12:21 Richard Hughes napsal(a):
On 16 September 2014 10:55, Zdenek Kabelac zkabelac@redhat.com wrote:
Just a thought - but wouldn't be better spend time to enlighten Gnome/Firefox developers how to write applications in a way the could be upgraded runtime
So, it's not just the application, it's every application and D-Bus service the application uses. Even glibc opens files-as-resources at
So it's time to fix D-Bus then as well!
If something is broken - it will not get fixed by hiding broken design behind reboot&upgrade.
Guys it's 21st. century - it's so much broken idea to reboot 'machine' because I want to upgrade i.e. Firefox - that I simply don't have words for it.
If you can't fix the Firefox (or some other broken tool) and you want to allow install of new version - then you need to allow i.e. parallel installs of such packages - it's that simple - I've been doing this more then 15 years ago at University - so really nothing new here...
While the old-firefox is in use - new one could be installed in parallel and the old one is removed when the last user closes app - of course this has another problem with dependency hell - again solvable thing - look at i.e. NixOS.
Just fix things in their core instead of hiding problem in pointless reboots.
Have any of those 'inventors' of reboot&update ever managed multiuser system?
This 'reboot' idea is just 'somehow' usable maybe for a single seat single user desktop - but not anywhere else.
Has Fedora given up Unix ??
Zdenek
----- Original Message -----
Dne 16.9.2014 v 12:21 Richard Hughes napsal(a):
On 16 September 2014 10:55, Zdenek Kabelac zkabelac@redhat.com wrote:
Just a thought - but wouldn't be better spend time to enlighten Gnome/Firefox developers how to write applications in a way the could be upgraded runtime
So, it's not just the application, it's every application and D-Bus service the application uses. Even glibc opens files-as-resources at
So it's time to fix D-Bus then as well!
If something is broken - it will not get fixed by hiding broken design behind reboot&upgrade.
Quite true.
If you can't fix the Firefox (or some other broken tool) and you want to allow install of new version - then you need to allow i.e. parallel installs of such packages - it's that simple - I've been doing this more then 15 years ago at University - so really nothing new here...
Well, what we would need is: 1. Ability to keep multiple versions (both ABI-compatible and ABI-incompatible) of a single application or library or service installed and running at the same time. 2. Ability to detect which processes depend on which versions of which components. 3. Ability to automatically restart such processes without loosing state (either completely transparently or with some user notification for GUIs).
This has all been done before, and can be done again. (And it would make at least half of the userbase clamoring for containers what they need, without playing ugly complex nontransparent namespace games.) But let’s be clear about it, 1. means completely changing our filesystem layout, and 3. means changing our process model to go way beyond int main(...).
It is technically possible, it is the right thing to do. Do we want to do it and can we do it?
This 'reboot' idea is just 'somehow' usable maybe for a single seat single user desktop - but not anywhere else.
Has Fedora given up Unix ??
The Unix history is actually closer to “edit header files to match your hardware, then rebuild the kernel and userpace with (make world), and reboot“. Packaged applications with an ISV source and an update stream separate from the OS have certainly been built on top of Unix but have never been a major design focus. Arguably the whole point of a “Linux distribution” has been to get closer to the BSD-style single-kernel-and-userspace distribution updated always as a whole. Mirek
Dne 16.9.2014 v 16:39 Miloslav Trmač napsal(a):
----- Original Message -----
Dne 16.9.2014 v 12:21 Richard Hughes napsal(a):
On 16 September 2014 10:55, Zdenek Kabelac zkabelac@redhat.com wrote:
Just a thought - but wouldn't be better spend time to enlighten Gnome/Firefox developers how to write applications in a way the could be upgraded runtime
So, it's not just the application, it's every application and D-Bus service the application uses. Even glibc opens files-as-resources at
So it's time to fix D-Bus then as well!
If something is broken - it will not get fixed by hiding broken design behind reboot&upgrade.
Quite true.
If you can't fix the Firefox (or some other broken tool) and you want to allow install of new version - then you need to allow i.e. parallel installs of such packages - it's that simple - I've been doing this more then 15 years ago at University - so really nothing new here...
Well, what we would need is:
- Ability to keep multiple versions (both ABI-compatible and ABI-incompatible) of a single application or library or service installed and running at the same time.
Other distributions allow to install multiple version of same libraries - I've never understood the Fedora policy to recompile whole Fedora when a new version of library is released.
This policy design is quite 'show-stopper' for anything like this...
- Ability to detect which processes depend on which versions of which components.
We already managed to brought in systemd....
- Ability to automatically restart such processes without loosing state (either completely transparently or with some user notification for GUIs).
I'm not quite sure why we would need restart - simply delayed lazy release of unused packages would do the trick here - doing here state-full design is much more complex thing....
The primary use-case to target should be to allow reinstall user packages while they are in use.
This has all been done before, and can be done again. (And it would make at least half of the userbase clamoring for containers what they need, without playing ugly complex nontransparent namespace games.) But let’s be clear about it, 1. means completely changing our filesystem layout, and 3. means changing our process model to go way beyond int main(...).
It is technically possible, it is the right thing to do. Do we want to do it and can we do it?
Fedora made not much useful /usr move thing - so maybe it's time to think and design something really useful.
As mentioned there are OSes which do handle things much better...
And surprisingly even Systemd guru realized there is something broken with current filesystem layout - except solving it with Btrfs is not really a fix...
Has Fedora given up Unix ??
The Unix history is actually closer to “edit header files to match your hardware, then rebuild the kernel and userpace with (make world), and reboot“. Packaged applications with an ISV source and an update stream separate from the OS have certainly been built on top of Unix but have never been a major design focus. Arguably the whole point of a “Linux distribution” has been to get closer to the BSD-style single-kernel-and-userspace distribution updated always as a whole.
My view rather is - Fedora is taking feature-by-feature away from my box.
Zdenek
----- Original Message -----
Well, what we would need is:
- Ability to keep multiple versions (both ABI-compatible and
ABI-incompatible) of a single application or library or service installed and running at the same time.
Other distributions allow to install multiple version of same libraries
AFAIK only when the ABI (soname) is different. We also need to allow for the case when the ABI stays the same but the internal implementation changes (e.g. changing a format of a resource file that is used by the library, making old processes incompatible with the newly installed resource).
- Ability to detect which processes depend on which versions of which
components.
We already managed to brought in systemd....
I can’t see how systemd helps. See the other discussions about Python/Ruby modules that leave no obvious trace of their dependency after being loaded into memory.
- Ability to automatically restart such processes without loosing state
(either completely transparently or with some user notification for GUIs).
I'm not quite sure why we would need restart - simply delayed lazy release of unused packages would do the trick here - doing here state-full design is much more complex thing....
Because otherwise you end up with an old version of Firefox running for 60 days (or however long a laptop can run with suspends and no restarts; most people about never quit their browser), and that version of Firefox keeping an old version of a system-wide daemon running for 60 days as well..
And surprisingly even Systemd guru realized there is something broken with current filesystem layout - except solving it with Btrfs is not really a fix...
The sad thing is that adding more workarounds like namespaces and the like really might be easier than agreeing on making a real change and getting it done :( But we will live with the costs forever. Mirek
Dne 16.9.2014 v 17:08 Miloslav Trmač napsal(a):
----- Original Message -----
Well, what we would need is:
- Ability to keep multiple versions (both ABI-compatible and
ABI-incompatible) of a single application or library or service installed and running at the same time.
Other distributions allow to install multiple version of same libraries
AFAIK only when the ABI (soname) is different. We also need to allow for the case when the ABI stays the same but the internal implementation changes (e.g. changing a format of a resource file that is used by the library, making old processes incompatible with the newly installed resource).
That's what I mean - when something changes in the library - soname should change. It's quite broken to release same version of library with the same name when there is different internal implementation. It's something library package maintainer should take care of...
There is also support for versioned symbols.
- Ability to detect which processes depend on which versions of which
components.
We already managed to brought in systemd....
I can’t see how systemd helps. See the other discussions about Python/Ruby modules that leave no obvious trace of their dependency after being loaded into memory.
It has similar complexity and maybe packages could be described just like services - so similar thing could be possibly reused here ?
Anyway - I've not been thinking too deeply about this...
- Ability to automatically restart such processes without loosing state
(either completely transparently or with some user notification for GUIs).
I'm not quite sure why we would need restart - simply delayed lazy release of unused packages would do the trick here - doing here state-full design is much more complex thing....
Because otherwise you end up with an old version of Firefox running for 60 days (or however long a laptop can run with suspends and no restarts; most people about never quit their browser), and that version of Firefox keeping an old version of a system-wide daemon running for 60 days as well..
Sure if user is able to run Firefox for 60 days (though my usually goes out of RAM in just a week or so, if it's not crashing earlier...) - then he is happy user.
I guess something may show to user some nice gui dialog warning like - 'Hey there is a new version installed - so 'restart' your browser to get new cool feature' (FF normally already detects upgrades and suggest restart)
I don't see nothing wrong with this case - it's per user and doesn't require upgrade - and if I've something important is running in my browser I could delay restart to the moment I don't care.
And surprisingly even Systemd guru realized there is something broken with current filesystem layout - except solving it with Btrfs is not really a fix...
The sad thing is that adding more workarounds like namespaces and the like really might be easier than agreeing on making a real change and getting it done :( But we will live with the costs forever.
Look at NixOS - there are no namespace or btrfs requirements....
Zdenek
----- Original Message -----
Dne 16.9.2014 v 17:08 Miloslav Trmač napsal(a):
----- Original Message -----
Well, what we would need is:
- Ability to keep multiple versions (both ABI-compatible and
ABI-incompatible) of a single application or library or service installed and running at the same time.
Other distributions allow to install multiple version of same libraries
AFAIK only when the ABI (soname) is different. We also need to allow for the case when the ABI stays the same but the internal implementation changes (e.g. changing a format of a resource file that is used by the library, making old processes incompatible with the newly installed resource).
That's what I mean - when something changes in the library - soname should change. It's quite broken to release same version of library with the same name when there is different internal implementation.
No. Let me try to explain the issue in detail.
libfoo-1.0.0 and libfoo-1.0.1 both use /usr/share/libfoo/magic_database, but the internal format has changed between the versions. It is _internal_ so changing the soname would be both unnecessary and _inappropriate_ (breaking users). Looking at current packaging, neither RPM nor debian would install libfoo-1.0.0 and libfoo-1.0.1 in parallel. Looking at the filesystem (i.e. assuming that we can change packaging), we may have both /usr/lib*/foo-1.0.0.so and /usr/lib*/foo-1.0.1.so installed on the filesystem, but that /usr/share/libfoo/magic_database can only have only one of the versions. (OK, in this case it could be argued that it is the upstream’s fault and that they should version the magic_database path if they change the internal format, to explicitly design for parallel installs. But now consider the distribution patching the library (changing “release” as opposed to “version” in RPM terminology); upstream can’t reasonably for that, parallel installation of same-version-different-release packages must be a distribution-focused filesystem design.
- Ability to detect which processes depend on which versions of which
components.
We already managed to brought in systemd....
I can’t see how systemd helps. See the other discussions about Python/Ruby modules that leave no obvious trace of their dependency after being loaded into memory.
It has similar complexity and maybe packages could be described just like services - so similar thing could be possibly reused here ?
Extra manual dependency information that would get obsolete?
- Ability to automatically restart such processes without loosing state
(either completely transparently or with some user notification for GUIs).
I'm not quite sure why we would need restart - simply delayed lazy release of unused packages would do the trick here - doing here state-full design is much more complex thing....
Because otherwise you end up with an old version of Firefox running for 60 days (or however long a laptop can run with suspends and no restarts; most people about never quit their browser), and that version of Firefox keeping an old version of a system-wide daemon running for 60 days as well..
Sure if user is able to run Firefox for 60 days (though my usually goes out of RAM in just a week or so, if it's not crashing earlier...) - then he is happy user.
Until an exploit _that they have already installed an update for_ steals their data.
I guess something may show to user some nice gui dialog warning like - 'Hey there is a new version installed - so 'restart' your browser to get new cool feature' (FF normally already detects upgrades and suggest restart)
No, that’s not good enough. The OS should restart it for the user (other OSes already can do this). (In the case of user-invisible components there is no other option, and for user-visible GUIs it is also the right thing to do.) Mirek
Dne 16.9.2014 v 17:36 Miloslav Trmač napsal(a):
----- Original Message -----
- Ability to detect which processes depend on which versions of which
components.
We already managed to brought in systemd....
I can’t see how systemd helps. See the other discussions about Python/Ruby modules that leave no obvious trace of their dependency after being loaded into memory.
It has similar complexity and maybe packages could be described just like services - so similar thing could be possibly reused here ?
Extra manual dependency information that would get obsolete?
Runtime removal of unused packages might be fun. It could be something like 'fstrim' tool to run through cron...
- Ability to automatically restart such processes without loosing state
(either completely transparently or with some user notification for GUIs).
I'm not quite sure why we would need restart - simply delayed lazy release of unused packages would do the trick here - doing here state-full design is much more complex thing....
Because otherwise you end up with an old version of Firefox running for 60 days (or however long a laptop can run with suspends and no restarts; most people about never quit their browser), and that version of Firefox keeping an old version of a system-wide daemon running for 60 days as well..
Sure if user is able to run Firefox for 60 days (though my usually goes out of RAM in just a week or so, if it's not crashing earlier...) - then he is happy user.
Until an exploit _that they have already installed an update for_ steals their data.
I guess something may show to user some nice gui dialog warning like - 'Hey there is a new version installed - so 'restart' your browser to get new cool feature' (FF normally already detects upgrades and suggest restart)
No, that’s not good enough. The OS should restart it for the user (other OSes already can do this). (In the case of user-invisible components there is no other option, and for user-visible GUIs it is also the right thing to do.)
As long as Firefox can't restart 'unnoticeable' while playing youtube video - this is not an option - when admins updates machine - it just can't kill every users' running firefox.
It's fine to place a warning somewhere and require restart with some 'many hours' timeout - but for almost all "Firefox" updates - it's good enough to restart it just sometimes latest - and for those 'have to restart' - it still upon admins policy - not upon Fedora package maintainer....
Zdenek
On Tue, 2014-09-16 at 16:21 +0200, Zdenek Kabelac wrote:
Has Fedora given up Unix ??
This thread has gone quite far out into the weeds. It started with a fairly concrete question: can we improve the offline update experience by requiring only a single reboot, instead of two ?
I'd still be interested in hearing Lennarts opinion on this.
Matthias
Dne 16.9.2014 v 17:29 Matthias Clasen napsal(a):
On Tue, 2014-09-16 at 16:21 +0200, Zdenek Kabelac wrote:
Has Fedora given up Unix ??
This thread has gone quite far out into the weeds. It started with a fairly concrete question: can we improve the offline update experience by requiring only a single reboot, instead of two ?
That's the core of the problem, while other systems have been trying for years to get rid of this nonsense (updates requiring reboots), we are moving the opposite direction. Technical reasons aside(*), this is NOT right for our users.
(*) I agree offline updates are the safest way technically but that should be only regarded as a workaround for a more fundamental problem, lying deep below the desktop level.
----- Original Message -----
Dne 16.9.2014 v 17:29 Matthias Clasen napsal(a):
On Tue, 2014-09-16 at 16:21 +0200, Zdenek Kabelac wrote:
Has Fedora given up Unix ??
This thread has gone quite far out into the weeds. It started with a fairly concrete question: can we improve the offline update experience by requiring only a single reboot, instead of two ?
That's the core of the problem, while other systems have been trying for years to get rid of this nonsense (updates requiring reboots), we are moving the opposite direction. Technical reasons aside(*), this is NOT right for our users.
(*) I agree offline updates are the safest way technically but that should be only regarded as a workaround for a more fundamental problem, lying deep below the desktop level.
There are no desktop systems that can do live application updates. Applications on mobile OSes get killed (and state is saved) when you update them.
All those OSes require reboots when updating the OS.
On 09/17/2014 11:54 AM, Bastien Nocera wrote:
All those OSes require reboots when updating the OS.
Define OS.
Firefox is definitely not OS. While systemd is OS. I am fine with reboot after systemd upgrade, but not after upgrading Firefox.
And there we have lots of packages in between those two. Where you would draw a line? What is OS? (And this brought us to core of Fedora.next discussion :) )
On Wed, 17.09.14 13:58, Miroslav Suchý (msuchy@redhat.com) wrote:
On 09/17/2014 11:54 AM, Bastien Nocera wrote:
All those OSes require reboots when updating the OS.
Define OS.
Firefox is definitely not OS. While systemd is OS. I am fine with reboot after systemd upgrade, but not after upgrading Firefox.
Well, on Fedora and Unixes the apps are just part of the OS, they are all dropped into /usr, and not isolated out.
It would be great if we could nicely isolate the apps from the OS so that we can restart the apps independently from the OS, but this requires isolating things first.
We are working on app sandboxes, and they will make this available, but it's the traditional Linux model cannot really deliver this.
Lennart
On 22 October 2014 04:31, Lennart Poettering mzerqung@0pointer.de wrote:
On Wed, 17.09.14 13:58, Miroslav Suchý (msuchy@redhat.com) wrote:
On 09/17/2014 11:54 AM, Bastien Nocera wrote:
All those OSes require reboots when updating the OS.
Define OS.
Firefox is definitely not OS. While systemd is OS. I am fine with reboot after systemd upgrade, but not after upgrading Firefox.
Well, on Fedora and Unixes the apps are just part of the OS, they are all dropped into /usr, and not isolated out.
It would be great if we could nicely isolate the apps from the OS so that we can restart the apps independently from the OS, but this requires isolating things first.
We are working on app sandboxes, and they will make this available, but it's the traditional Linux model cannot really deliver this.
Well it depends on the traditional model that you are going to refer to. In the model of the Unix systems I had to set up in the late 1980's and the 1990's... there was a separation in that items for the OS were in the root directories (say /bin and /sbin), and stuff that was not OS critical but user critical were in /usr/{bin,sbin}. And then local critical were in /opt or /usr/local depending on the OS.
We (Linux distributions) lost that distinction sometime in the past decade and then undid it completely with the various /usr/ merges. [This isn't meant to open that can of worms.. the distinction was broken before the merge.. the merge just made it clear.]
I am not sure how to best move from here. A complete reinvent of hierarchies is always tempting.. but it has always been the deathknell of every OS that has done it in the past due to chasm crossing issues (too much old stuff needing old things causing any benefits from new stuff to be actual detriments). Doing a more thorough job of packaging items so that system only items were in /bin,/lib, etc has never worked because too many things sit between the two. [its a user component AND a system component!] At best I can say it comes down to operating systems are too damn complicated and I need to go back to potato farming :)
Lennart
-- Lennart Poettering, Red Hat -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
On 10/22/2014 06:31, Lennart Poettering wrote:
It would be great if we could nicely isolate the apps from the OS so that we can restart the apps independently from the OS, but this requires isolating things first.
Isn't the differentiation between kernel space and user space sufficient for identifying what is the OS and what is an application? I know Windows blurred those lines during the Browser Wars with Netscape, but to my knowledge that separation still exists in Linux.
Tom
On Wed, Oct 22, 2014 at 4:45 PM, Tom Rivers tom@impact-crater.com wrote:
On 10/22/2014 06:31, Lennart Poettering wrote:
It would be great if we could nicely isolate the apps from the OS so that we can restart the apps independently from the OS, but this requires isolating things first.
Isn't the differentiation between kernel space and user space sufficient for identifying what is the OS and what is an application? I know Windows blurred those lines during the Browser Wars with Netscape, but to my knowledge that separation still exists in Linux.
No the OS is more than just a kernel.
On 10/22/2014 10:58, drago01 wrote:
No the OS is more than just a kernel.
Kernel Space contains more than just the kernel. It also contains device drivers, kernel extensions, and other privileged processes that require full system access. User Space exists as a barrier to keep applications separate from the OS itself.
Tom
On 17.09.2014 13:58, Miroslav Suchý wrote:
On 09/17/2014 11:54 AM, Bastien Nocera wrote:
All those OSes require reboots when updating the OS.
Define OS.
Firefox is definitely not OS. While systemd is OS. I am fine with reboot after systemd upgrade, but not after upgrading Firefox.
the important point in that case is not reboot after upgrading Firefox but *before* upgrading Firefox, which means that at the time of the upgrade no Firefox will be running and potentially crashing because one of the 100s of DSOs it loads on-demand has changed in an incompatible way.
there used to be quite a few ABRT crashes reported against desktop applications with impossible looking stack traces (although with the automatic micro-reports it's less of a problem nowadays as they are easier to ignore), and sometimes the reporter gives feedback that the application was running across a yum update...
On 22 October 2014 20:07, Michael Stahl mstahl@redhat.com wrote:
On 17.09.2014 13:58, Miroslav Suchý wrote:
On 09/17/2014 11:54 AM, Bastien Nocera wrote:
All those OSes require reboots when updating the OS.
Define OS.
Firefox is definitely not OS. While systemd is OS. I am fine with reboot after systemd upgrade, but not after upgrading Firefox.
the important point in that case is not reboot after upgrading Firefox but *before* upgrading Firefox, which means that at the time of the upgrade no Firefox will be running and potentially crashing because one of the 100s of DSOs it loads on-demand has changed in an incompatible way.
there used to be quite a few ABRT crashes reported against desktop applications with impossible looking stack traces (although with the automatic micro-reports it's less of a problem nowadays as they are easier to ignore), and sometimes the reporter gives feedback that the application was running across a yum update...
While it can be a bit confusing the first time it happens to you, the solution is just to start and stop firefox again in that case. If it the goal is just to tidy ABRT crash reports (and I'm not sure it is) then forcing reboots on users wouldn't be very kind.
On Wed, Oct 22, 2014 at 11:35:00PM +0100, Ian Malone wrote:
On 22 October 2014 20:07, Michael Stahl mstahl@redhat.com wrote:
On 17.09.2014 13:58, Miroslav Suchý wrote:
On 09/17/2014 11:54 AM, Bastien Nocera wrote:
All those OSes require reboots when updating the OS.
Define OS.
Firefox is definitely not OS. While systemd is OS. I am fine with reboot after systemd upgrade, but not after upgrading Firefox.
the important point in that case is not reboot after upgrading Firefox but *before* upgrading Firefox, which means that at the time of the upgrade no Firefox will be running and potentially crashing because one of the 100s of DSOs it loads on-demand has changed in an incompatible way.
there used to be quite a few ABRT crashes reported against desktop applications with impossible looking stack traces (although with the automatic micro-reports it's less of a problem nowadays as they are easier to ignore), and sometimes the reporter gives feedback that the application was running across a yum update...
While it can be a bit confusing the first time it happens to you, the solution is just to start and stop firefox again in that case. If it the goal is just to tidy ABRT crash reports (and I'm not sure it is) then forcing reboots on users wouldn't be very kind.
Beyond cleaning crash reports and reducing them, there is also the perception of users.
If we want to have the same reputation as some older windows with "this software crashed and I do not know why" from our users, then having regular random crashes after a random delay after upgrade is the way to go. If we still aim for "the software is solid and stable", then we should try to reduce the random crash.
On 10/22/2014 09:07 PM, Michael Stahl wrote:
the important point in that case is not reboot after upgrading Firefox but*before* upgrading Firefox, which means that at the time of the upgrade no Firefox will be running and potentially crashing because one of the 100s of DSOs it loads on-demand has changed in an incompatible way.
The upgrade can last one hour (more or less). I'm not going to wait for upgrade and looking at black screen (albeit with progress bar). When upgrading - I save critical work in progress, and start reading articles from my RSS queue. And if Firefox crash (which happens very very rarely) I lost nothing, but get one hour of my life for free.
To be constructive: I can imagine displaying dialog: Going to upgrade your system. Do you want to do it offline? * Yes - most safe way * No - there is non-trivial chance that some running application would crash (proceed on your own risk)
And let the user decide if he is power user or not.
On 23 October 2014 09:56, Miroslav Suchý msuchy@redhat.com wrote:
The upgrade can last one hour (more or less).
An hour?! Most of my offline updates take a few tens of seconds with F21. Is this on fairly up-to-date SSD hardware? Are any specific packages taking longer than the others?
Richard
On Thu, Oct 23, 2014 at 11:43:29AM +0100, Richard Hughes wrote:
On 23 October 2014 09:56, Miroslav Suchý msuchy@redhat.com wrote:
The upgrade can last one hour (more or less).
An hour?! Most of my offline updates take a few tens of seconds with F21. Is this on fairly up-to-date SSD hardware? Are any specific packages taking longer than the others?
Don't assume everyone is using SSDs with Fedora....
Regards, Daniel
Am 23.10.2014 um 12:52 schrieb Daniel P. Berrange:
On Thu, Oct 23, 2014 at 11:43:29AM +0100, Richard Hughes wrote:
On 23 October 2014 09:56, Miroslav Suchý msuchy@redhat.com wrote:
The upgrade can last one hour (more or less).
An hour?! Most of my offline updates take a few tens of seconds with F21. Is this on fairly up-to-date SSD hardware? Are any specific packages taking longer than the others?
Don't assume everyone is using SSDs with Fedora....
that assumption is sadly a common mistake on many places and leads to implement things less efficient because "it's fast enough on 'fairly' recent hardware)
frankly it will take many years before i replace my 4x2 TB RAID10 disks by 4x2 TB SSD just because the price and no i won't re-install from scratch to just have the OS only on SSD and my virtual machines (which are the reason for that large disks) stay on slower disks
Am 23.10.2014 um 12:43 schrieb Richard Hughes:
On 23 October 2014 09:56, Miroslav Suchý msuchy@redhat.com wrote:
The upgrade can last one hour (more or less).
An hour?! Most of my offline updates take a few tens of seconds with F21. Is this on fairly up-to-date SSD hardware? Are any specific packages taking longer than the others?
an hour was surely exaggerated
but there was not so long ago a bug in some package updateing caches which took on a hyperfast RAID10 machine with enabled barriers up to 10 minutes for a ordinary KDE update
before it got fixid without barriers (which are are default) a few seconds - you don't know if and how often such things may get triggered inside a update transaction (like the useless ldconfig and systemd reloads after every single related package)
# https://bugzilla.redhat.com/show_bug.cgi?id=1052173#c32 export PKGSYSTEM_ENABLE_FSYNC=0
On 10/23/2014 12:57 PM, Reindl Harald wrote:
The upgrade can last one hour (more or less).
An hour?! Most of my offline updates take a few tens of seconds with F21. Is this on fairly up-to-date SSD hardware? Are any specific packages taking longer than the others?
an hour was surely exaggerated
Not at all. I'm not updating daily. I upgraded my machine IIRC 2-3 weeks ago. So lets benchmark it and provide you real data.
My machine have classic magnetic disk, however in SW RAID1. Timing cached reads: 12236 MB in 2.00 seconds = 6124.59 MB/sec Timing buffered disk reads: 412 MB in 3.00 seconds = 137.12 MB/sec Fedora 21, 16 GB RAM (2GB free), 8 CPU cores, swap available but none used
I run "dnf upgrade" and I have been offered 853 packages and 1.3 GB to download. Download lasted 3mins 20secs. Then installation started and since beginning "transaction started" till the end lasted exactly 53 minutes. No specific package is blocking the process, dnf was chewing packages one by one in steady pace. Veryfing phase lasted ~4 minutes, so it means approximately 3 second per package, which is what I am seeing on screen.
And this is nothing exceptional. I see similar times across all machines I maintain. When I'm updating box of my mother (old EeeBox, updating aprox every 3 months) then the time is usually 3 hours (however ~1 hour is just download phase).
On Fri, 2014-10-24 at 12:00 +0200, Miroslav Suchý wrote:
I'm not updating daily. I upgraded my machine IIRC 2-3 weeks ago. So lets benchmark it and provide you real data.
My machine have classic magnetic disk, however in SW RAID1. Timing cached reads: 12236 MB in 2.00 seconds = 6124.59 MB/sec Timing buffered disk reads: 412 MB in 3.00 seconds = 137.12 MB/sec Fedora 21, 16 GB RAM (2GB free), 8 CPU cores, swap available but none used
I run "dnf upgrade" and I have been offered 853 packages and 1.3 GB to download. Download lasted 3mins 20secs. Then installation started and since beginning "transaction started" till the end lasted exactly 53 minutes. No specific package is blocking the process, dnf was chewing packages one by one in steady pace. Veryfing phase lasted ~4 minutes, so it means approximately 3 second per package, which is what I am seeing on screen.
And this is nothing exceptional. I see similar times across all machines I maintain. When I'm updating box of my mother (old EeeBox, updating aprox every 3 months) then the time is usually 3 hours (however ~1 hour is just download phase).
More than anything, doesn't this just shows that we simply push way too many updates in Fedora?
Am 24.10.2014 um 12:02 schrieb Mathieu Bridon:
On Fri, 2014-10-24 at 12:00 +0200, Miroslav Suchý wrote:
I'm not updating daily. I upgraded my machine IIRC 2-3 weeks ago. So lets benchmark it and provide you real data.
My machine have classic magnetic disk, however in SW RAID1. Timing cached reads: 12236 MB in 2.00 seconds = 6124.59 MB/sec Timing buffered disk reads: 412 MB in 3.00 seconds = 137.12 MB/sec Fedora 21, 16 GB RAM (2GB free), 8 CPU cores, swap available but none used
I run "dnf upgrade" and I have been offered 853 packages and 1.3 GB to download. Download lasted 3mins 20secs. Then installation started and since beginning "transaction started" till the end lasted exactly 53 minutes. No specific package is blocking the process, dnf was chewing packages one by one in steady pace. Veryfing phase lasted ~4 minutes, so it means approximately 3 second per package, which is what I am seeing on screen.
And this is nothing exceptional. I see similar times across all machines I maintain. When I'm updating box of my mother (old EeeBox, updating aprox every 3 months) then the time is usually 3 hours (however ~1 hour is just download phase).
More than anything, doesn't this just shows that we simply push way too many updates in Fedora?
no
* first: the above is Rawhide/Alpha * second: the reason i run Fedora and not Debian/RHEL is fast updates * third: nobody should apply updates every 3 weeks but that above is Alpha and so no "production" machine
On 24 October 2014 11:00, Miroslav Suchý msuchy@redhat.com wrote:
I run "dnf upgrade" and I have been offered 853 packages and 1.3 GB to download.
I'm pretty sure F21 for the last few weeks is not representative of a normal installed system over a 2 week period. 853 must be close to every package on your box...
Richard
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thu, 23 Oct 2014 10:56:00 +0200 Miroslav Suchý msuchy@redhat.com wrote:
On 10/22/2014 09:07 PM, Michael Stahl wrote:
the important point in that case is not reboot after upgrading Firefox but*before* upgrading Firefox, which means that at the time of the upgrade no Firefox will be running and potentially crashing because one of the 100s of DSOs it loads on-demand has changed in an incompatible way.
The upgrade can last one hour (more or less). I'm not going to wait for upgrade and looking at black screen (albeit with progress bar). When upgrading - I save critical work in progress, and start reading articles from my RSS queue. And if Firefox crash (which happens very very rarely) I lost nothing, but get one hour of my life for free.
I have had some of my arm systems take 6-8 hours and even more to update storage is a class 10 microsd card. the system was Fedora 21 and about 2-3 weeks worth of updates to be applied, about 700 packages.
Dennis
To be constructive: I can imagine displaying dialog: Going to upgrade your system. Do you want to do it offline? * Yes - most safe way * No - there is non-trivial chance that some running application would crash (proceed on your own risk)
And let the user decide if he is power user or not.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 09/16/2014 04:02 PM, Lukáš Tinkl wrote:
Dne 16.9.2014 v 17:29 Matthias Clasen napsal(a):
On Tue, 2014-09-16 at 16:21 +0200, Zdenek Kabelac wrote:
Has Fedora given up Unix ??
This thread has gone quite far out into the weeds. It started with a fairly concrete question: can we improve the offline update experience by requiring only a single reboot, instead of two ?
That's the core of the problem, while other systems have been trying for years to get rid of this nonsense (updates requiring reboots), we are moving the opposite direction. Technical reasons aside(*), this is NOT right for our users.
While that may be a wonderfully ideal state to reach (and certainly worth discussion), I agree with Matthias that this has gone far afield of my original question.
I'd like to try to tie this thread back to answering a single question: Without a fundamental redesign of the way things work, which would be a VERY large and long-term effort, can we improve the user experience for people *today* with fairly little effort?
If the answer to that question is "yes", is there anything fundamentally wrong with adding that workaround while we solve the situation in a more complete manner?
I think this is a great example of the adage "perfect is the enemy of the good". We have a situation that everyone seems to agree is less than ideal and we have two proposed solutions: one that probably requires only a few weeks of work and testing and one that probably requires a fundamental change to the way the system works and therefore could take years to accomplish.
Ask yourself this question: are you as a user patient enough to wait for a perfect solution or are you happy with incremental improvement?
(*) I agree offline updates are the safest way technically but that should be only regarded as a workaround for a more fundamental problem, lying deep below the desktop level.
On Fri, 2014-09-12 at 10:46 -0400, Stephen Gallagher wrote:
== The Problem ==
It is very common for users to have systems with encrypted root partitions (or even just /var and /etc). This may be due to a personal concern for their data or a corporate policy mandating full-disk encryption. Disk encryption requires a password (or other more complicated credentials) be be presented just after the kernel is booted and before the drives can be mounted and their data read.
With the current implementation of the offline updates in Fedora, this leads to a very unpleasant user experience when updating. We offer two ways to perform offline updates in the default user environment of Fedora Workstation: "Install updates and reboot" or "Install updates and shut down".
With "Install updates and reboot", the behavior is as follows:
- The system shuts down and initiates an ACPI reboot.
- The system presents the kernel boot menu and then starts the
updater kernel. 3) The system presents the user with a password prompt for the disk encryption (possibly more than one, if the system is configured with different passwords for different partitions). 4) The offline updates occur. 5) The system shuts down and initiates an ACPI reboot. 6) The system presents the kernel boot menu and then starts the standard (possibly updated) kernel. 7) The system presents the user with a password prompt for the disk encryption (possibly more than one, if the system is configured with different passwords for different partitions). 8) The system completes booting.
During this experience, the user has been required to enter their disk encryption password *twice*. The same is true for the "Install and shut down" case, except that the two passwords are separated by some actual wallclock time.
== Proposed Improvements ==
We could significantly improve this situation by allowing the system to drop directly from the interactive system into the updater environment without doing a full reboot or relaunching the kernel.
Lennart, would it be possible to set up a special systemd target for performing updates that would essentially stop all processes except for systemd and then apply the updates?
In an ideal world, it would then also be possible after update is completed to restore operation to the standard boot targets of systemd so that the system comes back up without having to perform a total reboot. The exceptional case would of course be that in which either the kernel, libc or systemd[1] needed to be updated, in which case a reboot could be performed.
In this scenario, we can reduce the number of encrypted disk challenges to at most a single one, and that only if absolutely minimal plumbing packages saw an update.
I'd very much like to hear from the plumbers on this matter.
[1] I'm told that this might not be necessary; that systemd can re-exec itself to pick up its own updates. That would reduce the scope presumably to "only the kernel" forcing reboots.
I'm bumping this thread to get comments from Lennart (CCed). There's a lot of chatter in the conversation, but I'd very much like to hear an answer to the specific questions I posed in this first email.
On Thu, 2014-10-02 at 07:53 -0400, Stephen Gallagher wrote:
On Fri, 2014-09-12 at 10:46 -0400, Stephen Gallagher wrote:
== The Problem ==
It is very common for users to have systems with encrypted root partitions (or even just /var and /etc). This may be due to a personal concern for their data or a corporate policy mandating full-disk encryption. Disk encryption requires a password (or other more complicated credentials) be be presented just after the kernel is booted and before the drives can be mounted and their data read.
With the current implementation of the offline updates in Fedora, this leads to a very unpleasant user experience when updating. We offer two ways to perform offline updates in the default user environment of Fedora Workstation: "Install updates and reboot" or "Install updates and shut down".
With "Install updates and reboot", the behavior is as follows:
- The system shuts down and initiates an ACPI reboot.
- The system presents the kernel boot menu and then starts the
updater kernel. 3) The system presents the user with a password prompt for the disk encryption (possibly more than one, if the system is configured with different passwords for different partitions). 4) The offline updates occur. 5) The system shuts down and initiates an ACPI reboot. 6) The system presents the kernel boot menu and then starts the standard (possibly updated) kernel. 7) The system presents the user with a password prompt for the disk encryption (possibly more than one, if the system is configured with different passwords for different partitions). 8) The system completes booting.
During this experience, the user has been required to enter their disk encryption password *twice*. The same is true for the "Install and shut down" case, except that the two passwords are separated by some actual wallclock time.
== Proposed Improvements ==
We could significantly improve this situation by allowing the system to drop directly from the interactive system into the updater environment without doing a full reboot or relaunching the kernel.
Lennart, would it be possible to set up a special systemd target for performing updates that would essentially stop all processes except for systemd and then apply the updates?
In an ideal world, it would then also be possible after update is completed to restore operation to the standard boot targets of systemd so that the system comes back up without having to perform a total reboot. The exceptional case would of course be that in which either the kernel, libc or systemd[1] needed to be updated, in which case a reboot could be performed.
In this scenario, we can reduce the number of encrypted disk challenges to at most a single one, and that only if absolutely minimal plumbing packages saw an update.
I'd very much like to hear from the plumbers on this matter.
[1] I'm told that this might not be necessary; that systemd can re-exec itself to pick up its own updates. That would reduce the scope presumably to "only the kernel" forcing reboots.
I'm bumping this thread to get comments from Lennart (CCed). There's a lot of chatter in the conversation, but I'd very much like to hear an answer to the specific questions I posed in this first email.
Trying one more time to get Lennart to chime in here.
On Mon, 2014-10-13 at 11:26 -0400, Stephen Gallagher wrote:
On Thu, 2014-10-02 at 07:53 -0400, Stephen Gallagher wrote:
On Fri, 2014-09-12 at 10:46 -0400, Stephen Gallagher wrote:
== The Problem ==
It is very common for users to have systems with encrypted root partitions (or even just /var and /etc). This may be due to a personal concern for their data or a corporate policy mandating full-disk encryption. Disk encryption requires a password (or other more complicated credentials) be be presented just after the kernel is booted and before the drives can be mounted and their data read.
With the current implementation of the offline updates in Fedora, this leads to a very unpleasant user experience when updating. We offer two ways to perform offline updates in the default user environment of Fedora Workstation: "Install updates and reboot" or "Install updates and shut down".
With "Install updates and reboot", the behavior is as follows:
- The system shuts down and initiates an ACPI reboot.
- The system presents the kernel boot menu and then starts the
updater kernel. 3) The system presents the user with a password prompt for the disk encryption (possibly more than one, if the system is configured with different passwords for different partitions). 4) The offline updates occur. 5) The system shuts down and initiates an ACPI reboot. 6) The system presents the kernel boot menu and then starts the standard (possibly updated) kernel. 7) The system presents the user with a password prompt for the disk encryption (possibly more than one, if the system is configured with different passwords for different partitions). 8) The system completes booting.
During this experience, the user has been required to enter their disk encryption password *twice*. The same is true for the "Install and shut down" case, except that the two passwords are separated by some actual wallclock time.
== Proposed Improvements ==
We could significantly improve this situation by allowing the system to drop directly from the interactive system into the updater environment without doing a full reboot or relaunching the kernel.
Lennart, would it be possible to set up a special systemd target for performing updates that would essentially stop all processes except for systemd and then apply the updates?
In an ideal world, it would then also be possible after update is completed to restore operation to the standard boot targets of systemd so that the system comes back up without having to perform a total reboot. The exceptional case would of course be that in which either the kernel, libc or systemd[1] needed to be updated, in which case a reboot could be performed.
In this scenario, we can reduce the number of encrypted disk challenges to at most a single one, and that only if absolutely minimal plumbing packages saw an update.
I'd very much like to hear from the plumbers on this matter.
[1] I'm told that this might not be necessary; that systemd can re-exec itself to pick up its own updates. That would reduce the scope presumably to "only the kernel" forcing reboots.
I'm bumping this thread to get comments from Lennart (CCed). There's a lot of chatter in the conversation, but I'd very much like to hear an answer to the specific questions I posed in this first email.
Trying one more time to get Lennart to chime in here.
I spotted on another mailing list that Lennart is using a different email address these days. I'm now trying to reach him on that one instead.
On Fri, 12.09.14 10:46, Stephen Gallagher (sgallagh@redhat.com) wrote:
It is very common for users to have systems with encrypted root partitions (or even just /var and /etc). This may be due to a personal
Nitpicking: we currently do not support split-off /etc on Fedora/Dracut. /var may be split out though.
We could significantly improve this situation by allowing the system to drop directly from the interactive system into the updater environment without doing a full reboot or relaunching the kernel.
Lennart, would it be possible to set up a special systemd target for performing updates that would essentially stop all processes except for systemd and then apply the updates?
Well, the entire logic we have in place for offline updates is to ensure that they are applied in a well-defined very clean, minimal environment. Actually, that's not just the logic that is in place, that's the whole reason for having them at all!
I mean, if it is OK to allow leaking runtime state into the upgrade process we could just run the updates in parallel to normal runtime, how we always did things.
Or in even otherwords: offline updates are supposed to make update strictly reliable, and for that I figure a full reset and fresh boot, with everything initialized to the boot-up defaults is the closest if not only option.
That all said, we have been discussing supporting a new reboot-mode in systemd, which would essentially shut down userspace, until only PID 1 remains, and then immediately start userspace again on the same kernel. This would be pretty close to what you are asking for here... However, so far we have quite some concerns about adding this, precisely for the reason that it pretends to be a reset of everything to the boot-up defaults, but actually isn't, as a ton of runtime state is retained in /sys and /proc/sys and other runtime objects.
[1] I'm told that this might not be necessary; that systemd can re-exec itself to pick up its own updates. That would reduce the scope presumably to "only the kernel" forcing reboots.
Well, you cannot really do this atomically. Offline updates are about reliability. But you cannot restart userspace reliably during runtime. Think about this: daemons talk to each other, supporting a variety of APIs and API versions. If we want to do an update, then we'd have to restart all daemons at the same instant to ensure that no daemon starts talking with another daemon at the wrong time where it might be confused by either a too old or too new daemon. However, restarten them all at once is not possible. Moreover, some daemons cannot be restarted at all (for example dbus-daemon; and journald is difficult too, as it will lose connections to all other daemon's stdout/stderr if you do, even though it other works fine).
Hence: I'd really recommend leaving offline updates as is.
I see the problem though and acknowledge the fact that entering the passwords multiple times is annoying. Interestingly hw hdd encryption is nicer there as the keys are not flushed out on resets... Maybe that's actually a strategy to adopt here: upload the encryption keys into the firmware as efi vars, and then pull them out on next boots or so (assuming that efi vars can be marked to survive soft reboots without making them fully persistent...)
I understand this is all unsatisfying. Maybe fixing related problems might improve the situation a bit though. For example, I find it quite annoying if we always ask for both a hdd passphrase and a user passphrase at boot. It has been a long-time todo list item for systemd to add the hdd passphrases (with strict expiry) to the kernel keyring and then optionally pull them out there for gdm's autologin mode, so that they can be used for decrypting the gnome keyring and such. However, so far we haven't worked on that yet...
Lennart
On 10/21/2014 10:02 PM, Lennart Poettering wrote:
Maybe that's actually a strategy to adopt here: upload the encryption keys into the firmware as efi vars, and then pull them out on next boots or so (assuming that efi vars can be marked to survive soft reboots without making them fully persistent...)
Hmmm, surrendering your encryption keys to the only software part which you do not have control on?
On Wed, 22.10.14 14:11, Roberto Ragusa (mail@robertoragusa.it) wrote:
On 10/21/2014 10:02 PM, Lennart Poettering wrote:
Maybe that's actually a strategy to adopt here: upload the encryption keys into the firmware as efi vars, and then pull them out on next boots or so (assuming that efi vars can be marked to survive soft reboots without making them fully persistent...)
Hmmm, surrendering your encryption keys to the only software part which you do not have control on?
The firmware runs at the highest priviliges anyway, it can do whatever it wants... You don't have to "surrender" you keys to it. If it wants them it can take them anyway...
Lennart
-
Bastien Nocera -
Chris Murphy -
Corey Sheldon -
Daniel P. Berrange -
Dennis Gilmore -
drago01 -
Fred New -
Ian Malone -
Ian Pilcher -
Josh Boyer -
Lennart Poettering -
Lukáš Tinkl -
Mathieu Bridon -
Matthew Miller -
Matthias Clasen -
Michael Scherer -
Michael Stahl -
Miloslav Trmač -
Miroslav Suchý -
Nathanael D. Noblet -
Petr Pisar -
Przemek Klosowski -
Reindl Harald -
Richard Hughes -
Richard W.M. Jones -
Roberto Ragusa -
Stephen Gallagher -
Stephen John Smoogen -
Till Maas -
Tom Rivers -
Vít Ondruch -
Zdenek Kabelac