5:55 p.m.
Hi,
it is time to analyze the fallout from the following catastrophic Fedora 20
regression:
https://bugzilla.redhat.com/show_bug.cgi?id=1054350
"rpm scriptlets are exiting with status 127"
The impact:
* EVERYONE with Fedora 20 installed with SELinux enabled and in enforcing
mode, and who updated to the current stable updates, was hit by this bug.
* The bug completely breaks upgrading any package through both GUI and CLI
tools. Even the fix itself cannot be installed correctly.
* The only possible workaround requires use of the command line. It is
IMPOSSIBLE to fix this using GUI tools installed by default. The
system-config-selinux tool which can be used to fix this in a pure GUI
method is NOT installed by default in Fedora 20 for some stupid reason
(because somebody decided to make it as painful as possible to disable that
SELinux junk? Now I have to install system-config-selinux first thing post-
install just so I can disable the dreaded thing), and of course cannot be
installed after the fact because of the bug. Normal users do not use
terminals, so they can only reinstall Fedora or (more likely) a competing
distribution (or even operating system)!
* The only possible workaround also requires root access to the machine.
PolicyKit policy allows all users to install official updates by default,
but those users then cannot fix the breakage without bothering an
administrator.
* As per the above, there are several installations that can be considered
BRICKED.
* We are losing users to Ubuntu because of this issue. People are explicitly
saying they are switching to Ubuntu because of this bug (e.g.
https://bugzilla.redhat.com/show_bug.cgi?id=1054312#c5 , later confirmed:
https://bugzilla.redhat.com/show_bug.cgi?id=1054312#c10 ), and I am sure
there are many more who are silently doing it without telling us.
* The bug now has 38 (!) duplicates in Bugzilla, plus many complaints on
IRC, mailing lists, comments to other unrelated bugs (the fix for which
cannot be installed due to the SELinux bug) etc.
So it is time to draw some lessons from this issue to prevent such a bug
from ever occurring again!
So, what happened:
* We are enabling SELinux enabled (enforcing) by default, a tool designed to
prevent anything it does not like from happening. (Reread this carefully:
The ONLY thing that tool is designed to do at all is PREVENT things. It does
not have a SINGLE feature other than being a roadblock and an annoyance.)
* SELinux works by shipping a "policy" that effectively tries to specify in
one single place (read: single point of failure!) everything any program in
Fedora (scalability disaster!) ever wants to do (second-guessing its actual
code, i.e., duplication of all logic!). (Note the 3 (!) major antipatterns
in a single-sentence (!) description of how SELinux works!)
* An update to that SELinux policy was shipped that BREAKS the most critical
tools in Fedora, the ones required to update the system and thus install the
fixes for any regressions, including the very regression that caused the
breakage. And also any automated workarounds are blocked by design.
* That update made it out to the stable updates! In other words, the
draconian Update Policies that were enacted in a vain attempt to prevent
such issues from happening utterly failed at catching this bug.
Meanwhile, SELinux is also causing similarly fatal issues in Rawhide:
https://bugzilla.redhat.com/show_bug.cgi?id=1052317
"selinux-policy preventing login through sddm and ssh"
which are still NOT fixed! At least in that case, RPM is apparently not
affected, but if you cannot log in to your system (SDDM is the default
display manager for KDE in Rawhide), it is totally unusable ("bricked")!
So, what needs to happen:
* SELinux must be disabled (or preferably, not installed in the first place,
to avoid wasting space for nothing) by default! Just consider the benefits
(none!) vs. the risks (what you are seeing now: bricked systems in both F20
and Rawhide, the users switching to other distributions). If we want to have
any users left, SELinux needs to go away NOW!
* The Update Policies must be repealed. This regression has shown us that
not only they totally failed at preventing it, but they are actively
contributing to exposing MORE users to broken updates by delaying regression
fixes. (This kind of regression fixes needs to go out DIRECTLY to stable!)
Last time an issue like that happened (the D-Bus regression that broke
updates), a big drama was made that ultimately lead to the (flawed) Update
Policies. And even a "catastrophe" that hit only a very small portion of our
users (those running the server part of bind) was used as a(n additional)
justification for the Update Policies, whereas this one now hits ALL users
who merely had the mishap of sticking to our flawed defaults (SELinux
enforcing). Why would we stick our heads in the sand this time?
DISABLE/DROP SELINUX NOW!
Thank you for your consideration,
Kevin Kofler
PS: I still recommend to ALL Fedora users to disable SELinux immediately
after installing Fedora. That is the most effective way to avoid ever being
hit by catastrophical breakage such as bug #1054350 or bug #1052317. But we
should not ship with a broken default in the first place!
6:02 p.m.
On Fri, 2014-01-24 at 00:55 +0100, Kevin Kofler wrote:
Hi,
"catastrophic Fedora 20 regression"
https://bugzilla.redhat.com/show_bug.cgi?id=1054350
"rpm scriptlets are exiting with status 127"
"EVERYONE"
"IMPOSSIBLE" to fix this using GUI tools installed by default. The
"some stupid reason (because somebody decided to make it as painful as possible to
disable that
SELinux junk?"
"dreaded thing"
"(or even operating system)!"
"BRICKED."
"The ONLY thing that tool is designed to do at all is PREVENT things. It does
not have a SINGLE feature"
(read: single point of failure!)
(scalability disaster!)
duplication of all logic!)
(Note the 3 (!) major antipatterns in a single-sentence (!) description of how SELinux
works!)
BREAKS
draconian Update Policies
vain attempt
utterly failed
default! Just consider the benefits (none!)
If we want to have any users left, SELinux needs to go away NOW!
The Update Policies must be repealed
totally failed
Why would we stick our heads in the sand this time?
DISABLE/DROP SELINUX NOW!
That's a great way to go about having a calm and reasoned discussion and
building consensus, Kevin.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
6:06 p.m.
On 1/23/14, 5:55 PM, Kevin Kofler wrote:
* We are enabling SELinux enabled (enforcing) by default, a tool
designed to
prevent anything it does not like from happening. (Reread this carefully:
The ONLY thing that tool is designed to do at all is PREVENT things. It does
not have a SINGLE feature other than being a roadblock and an annoyance.)
In the same way that the lock on your front door is an annoyance, I guess.
* SELinux works by shipping a "policy" that effectively
tries to specify in
one single place (read: single point of failure!) everything any program in
Fedora (scalability disaster!) ever wants to do (second-guessing its actual
code, i.e., duplication of all logic!). (Note the 3 (!) major antipatterns
in a single-sentence (!) description of how SELinux works!)
If you think SELinux is "duplicating all logic" in application code,
I do not think you quite grasp how SELinux works.
If the solution to every serious bug that slips through the cracks of a release
is to disable the package, over time we may not have much left in Fedora.
I know that pretty much all filesystems would be out by now. ;)
-Eric
6:37 p.m.
Eric Sandeen wrote:
If the solution to every serious bug that slips through the cracks of
a
release is to disable the package, over time we may not have much left in
Fedora.
But SELinux is the one package (OK, one of the few, along with, e.g.,
firewall stuff) whose removal would actually INCREASE Fedora's
functionality!
Kevin Kofler
6:50 p.m.
On 1/23/14, 6:37 PM, Kevin Kofler wrote:
Eric Sandeen wrote:
> If the solution to every serious bug that slips through the cracks of a
> release is to disable the package, over time we may not have much left in
> Fedora.
But SELinux is the one package (OK, one of the few, along with, e.g.,
firewall stuff) whose removal would actually INCREASE Fedora's
functionality!
Sure, removing firewalls & selinux would be a serious enhancement
of functionality.
For malware botnets & spam hosts, especially...
-Eric
Kevin Kofler
7:34 p.m.
Eric Sandeen wrote:
Sure, removing firewalls & selinux would be a serious
enhancement
of functionality.
For malware botnets & spam hosts, especially...
That would mean that all the distributions that do not enable SELinux (nor
AppArmor) by default are all owned by botnets, not to mention the many
people who disable those "features". Yet, the only machines that get hit are
those that have not been updated for months if not years (often running
ancient EOL distributions, but not even having the last updates provided for
those). SELinux is by no means necessary to protect your machine (especially
a firewalled non-server machine). The firewall can be of some use (and I'm
not advocating dropping that by default), though ideally we shouldn't have
servers trying to listen to non-local connections by default in the first
place!
Kevin Kofler
7:38 p.m.
On Fri, 2014-01-24 at 02:34 +0100, Kevin Kofler wrote:
though ideally we shouldn't have
servers trying to listen to non-local connections by default in the first
place!
Doing so is specifically forbidden by policy, and the few packages that
do it had to seek a special exemption from FESCo to do so.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
7:37 p.m.
On Fri, Jan 24, 2014 at 10:37 AM, Kevin Kofler <kevin.kofler(a)chello.at>wrote:
Eric Sandeen wrote:
> If the solution to every serious bug that slips through the cracks of a
> release is to disable the package, over time we may not have much left in
> Fedora.
But SELinux is the one package (OK, one of the few, along with, e.g.,
firewall stuff) whose removal would actually INCREASE Fedora's
functionality!
Increase for who? Unauthorised attackers?
SELinux is the primary reason for me and my company choosing
Fedora/Redhat over Ubuntu/Debian or other distributions in the first place.
9:43 p.m.
On Fri, 2014-01-24 at 00:55 +0100, Kevin Kofler wrote:
Last time an issue like that happened (the D-Bus regression that
broke
updates),
That was my fault. Something which left an impact on me, you can be
sure. Like SELinux, DBus impacts everything nearly everything in early
userspace.
In fact, this particular regression is exactly one of the reasons I made
OSTree. With both this and the SELinux bug, knowing you can *always*
reboot into the previous system state and recover makes things
fundamentally better for a fast-moving system like Fedora.
Note OSTree is fully capable of *atomic* upgrades to SELinux - where
your running system is untouched, with the old policy. When you reboot,
you have the new policy, with the new daemon code.
With the RPM live updates model by default, you have old policy, until
rpm reloads it in the middle of a "transaction", restarts some daemons,
but not all of them, leaving you some *old* code with *new* policy -
something hard to test because you have to go out of your way to
reproduce it. You can't boot into that state directly.
Secondarily, on the server side, as many people have noted - this type
of thing can be caught by automated testing. It's not hardware
specific.
OSTree pairs extremely well with automated testing, because it allows
fast incremental updates for offline VMs. I've had this working for
about a year in the gnome-continuous context, and I can bring it to
rpm-ostree too. As in "a week or two".
So no, SELinux doesn't need to be disabled. We can make it much better
- if we step beyond the current philosophy of "test a package" to "test
many system states as atomic units".
10:18 p.m.
On Sex, 2014-01-24 at 00:55 +0100, Kevin Kofler wrote:
So, what needs to happen:
* SELinux must be disabled (or preferably, not installed in the first
place,
to avoid wasting space for nothing) by default! Just consider the
benefits
(none!) vs. the risks (what you are seeing now: bricked systems in
both F20
and Rawhide, the users switching to other distributions). If we want
to have
any users left, SELinux needs to go away NOW!
TBH: I always disable selinux , and yes, I vote on SELinux not be
install by default, not necessarily remove it.
* The Update Policies must be repealed. This regression has shown us
that
not only they totally failed at preventing it, but they are actively
contributing to exposing MORE users to broken updates by delaying
regression
fixes. (This kind of regression fixes needs to go out DIRECTLY to
stable!)
Also agree, this critical packages should go directly to stable and or
we should be able to revoke it.
Best regards,
--
Sérgio M. B.
10:20 p.m.
On Fri, 2014-01-24 at 04:18 +0000, Sérgio Basto wrote:
> * The Update Policies must be repealed. This regression has
shown us
> that
> not only they totally failed at preventing it, but they are actively
> contributing to exposing MORE users to broken updates by delaying
> regression
> fixes. (This kind of regression fixes needs to go out DIRECTLY to
> stable!)
Also agree, this critical packages should go directly to stable and or
we should be able to revoke it.
TBH this has always been the one of Kevin's Big Book Of Update Policy
Complaints I find the most baffling. If we know you managed to screw up
your update once, why exactly would we just trust you to get it right
the *second* time without any testing?
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
10:49 p.m.
On Qui, 2014-01-23 at 20:20 -0800, Adam Williamson wrote:
On Fri, 2014-01-24 at 04:18 +0000, Sérgio Basto wrote:
> > * The Update Policies must be repealed. This regression has shown us
> > that
> > not only they totally failed at preventing it, but they are actively
> > contributing to exposing MORE users to broken updates by delaying
> > regression
> > fixes. (This kind of regression fixes needs to go out DIRECTLY to
> > stable!)
>
> Also agree, this critical packages should go directly to stable and or
> we should be able to revoke it.
TBH this has always been the one of Kevin's Big Book Of Update Policy
Complaints I find the most baffling. If we know you managed to screw up
your update once, why exactly would we just trust you to get it right
the *second* time without any testing?
yeah , so revoke an update could be a better idea.
--
Sérgio M. B.
12:58 a.m.
On Fri, 2014-01-24 at 01:35 -0500, Rahul Sundaram wrote:
Hi
On Thu, Jan 23, 2014 at 11:49 PM, Sérgio Basto wrote:
yeah , so revoke an update could be a better idea.
How would that work? We don't control the mirrors. Mirrors can choose
to sync anytime they want to and retain packages we have removed.
Even if we can do it on the mirrors, we have no way to 'recall' a
package from systems where it's already been installed (of course in the
current case that wouldn't have worked anyway, but we're discussing the
generic case here).
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
6:39 a.m.
Adam Williamson wrote:
Even if we can do it on the mirrors, we have no way to
'recall' a
package from systems where it's already been installed (of course in the
current case that wouldn't have worked anyway, but we're discussing the
generic case here).
Crazy idea of the day: Maybe our update tools should default to distro-sync
rather than update? Together with ensuring timestamp monotonicity on the
metadata (don't accept older metadata if you already have newer one), it
would allow easily pulling faulty updates (except when RPM is broken as in
this case, of course) and could even render the dreaded Epoch hack obsolete.
Kevin Kofler
8:55 a.m.
On 01/24/2014 01:39 PM, Kevin Kofler wrote:
Adam Williamson wrote:
> Even if we can do it on the mirrors, we have no way to 'recall' a
> package from systems where it's already been installed (of course in the
> current case that wouldn't have worked anyway, but we're discussing the
> generic case here).
Crazy idea of the day: Maybe our update tools should default to distro-sync
rather than update?
No, for 2 reasons:
a) This would blow away all installed packages, which aren't available
in permanently enabled repos.
Most common such case is having selectively installed packages from
updates-testing, because users are facing problems with these packages'
nominal versions.
b) A much more common packaging bug class than the SELinux-case are
packages, which can not be uninstalled or downgraded or not be
downgraded properly. Classic such cases are packages with defective
rpm-scriptlets or with scriptlet which perform persistent changes.
Ralf
9:06 a.m.
Am 24.01.2014 15:55, schrieb Ralf Corsepius:
On 01/24/2014 01:39 PM, Kevin Kofler wrote:
> Adam Williamson wrote:
>> Even if we can do it on the mirrors, we have no way to 'recall' a
>> package from systems where it's already been installed (of course in the
>> current case that wouldn't have worked anyway, but we're discussing the
>> generic case here).
>
> Crazy idea of the day: Maybe our update tools should default to distro-sync
> rather than update?
No, for 2 reasons:
a) This would blow away all installed packages, which aren't available in permanently
enabled repos
that is not true, try it out
otherwise some packages would be not installed on my machines after a dist-upgrade
namely the ones never came from any repo and installed locally
Most common such case is having selectively installed packages from
updates-testing, because users are facing
problems with these packages' nominal versions
*that* is the reason not to do so because it would downgrade anything updated
explicitly from updates-testing,kde-testing,koji which would be a bad default
9:40 a.m.
On 01/24/2014 04:06 PM, Reindl Harald wrote:
Am 24.01.2014 15:55, schrieb Ralf Corsepius:
> On 01/24/2014 01:39 PM, Kevin Kofler wrote:
>> Adam Williamson wrote:
>>> Even if we can do it on the mirrors, we have no way to 'recall' a
>>> package from systems where it's already been installed (of course in the
>>> current case that wouldn't have worked anyway, but we're discussing
the
>>> generic case here).
>>
>> Crazy idea of the day: Maybe our update tools should default to distro-sync
>> rather than update?
> No, for 2 reasons:
>
> a) This would blow away all installed packages, which aren't available in
permanently enabled repos
that is not true, try it out
Been there many times.
Real world example with a package I maintain, which currently has an
update pending in updates-testing:
# yum install gumbo-parser
...
Installing : gumbo-parser-1.0-0.2.20131001gitd90ea2b.fc20.x86_64
...
[Note: updates-testing is disabled in
/etc/yum.repo.d/fedora-updates-testing.repo]
Now temporarily enable updates-testing to pull in the package from
updates-testing for testing:
# yum update --enablerepo=updates-testing gumbo-parser
...
Updating : gumbo-parser-1.0-0.2.20131204git87b99f2.fc20.x86_64
...
# yum distro-sync
...
Downgrading:
gumbo-parser x86_64
1.0-0.2.20131001gitd90ea2b.fc20 fedora
...
Removed:
gumbo-parser.x86_64 0:1.0-0.2.20131204git87b99f2.fc20
Installed:
gumbo-parser.x86_64 0:1.0-0.2.20131001gitd90ea2b.fc20
...
=>
qed
Ralf
9:57 a.m.
Am 24.01.2014 16:40, schrieb Ralf Corsepius:
On 01/24/2014 04:06 PM, Reindl Harald wrote:
>> a) This would blow away all installed packages, which aren't available in
permanently enabled repos
>
> that is not true, try it out
Been there many times
no, you did not and you did also not in your example below
Real world example with a package I maintain, which currently has an
update pending in updates-testing:
# yum distro-sync
...
Downgrading:
gumbo-parser x86_64
1.0-0.2.20131001gitd90ea2b.fc20 fedora
...
Removed:
gumbo-parser.x86_64 0:1.0-0.2.20131204git87b99f2.fc20
Installed:
gumbo-parser.x86_64 0:1.0-0.2.20131001gitd90ea2b.fc20
nothing is blown away, you only did not read the output
because it was *downgraded* and *not* removed
this is *completly* different than "blown away"
this is what distro-sync *is supposed to do*
upgrade or downgrade any package which is in whatever current repo
but it *does not* blow away packages not in any repo at all
________________________________________
and if you would not have stripped this paragraph of my original
reply you maybe had looked twice
> *that* is the reason not to do so because it would downgrade
anything updated
> explicitly from updates-testing,kde-testing,koji which would be a bad default
10:12 a.m.
On 01/24/2014 04:57 PM, Reindl Harald wrote:
Am 24.01.2014 16:40, schrieb Ralf Corsepius:
> On 01/24/2014 04:06 PM, Reindl Harald wrote:
>>> a) This would blow away all installed packages, which aren't available in
permanently enabled repos
>>
>> that is not true, try it out
> Been there many times
no, you did not and you did also not in your example below
> Real world example with a package I maintain, which currently has an update pending
in updates-testing:
>
> # yum distro-sync
> ...
> Downgrading:
> gumbo-parser x86_64
> 1.0-0.2.20131001gitd90ea2b.fc20 fedora
> ...
> Removed:
> gumbo-parser.x86_64 0:1.0-0.2.20131204git87b99f2.fc20
>
> Installed:
> gumbo-parser.x86_64 0:1.0-0.2.20131001gitd90ea2b.fc20
nothing is blown away, you only did not read the output
because it was *downgraded* and *not* removed
Rubbish - Stop being childish.
this is *completly* different than "blown away"
this is what distro-sync *is supposed to do*
upgrade or downgrade any package which is in whatever current repo
but it *does not* blow away packages not in any repo at all
It if the package from
updates-testing was fixing a critical bug on your
system, your system would be malfunctioning afterwards.
10:31 a.m.
Am 24.01.2014 17:12, schrieb Ralf Corsepius:
On 01/24/2014 04:57 PM, Reindl Harald wrote:
> Am 24.01.2014 16:40, schrieb Ralf Corsepius:
>> On 01/24/2014 04:06 PM, Reindl Harald wrote:
>>>> a) This would blow away all installed packages, which aren't
available in permanently enabled repos
>>>
>>> that is not true, try it out
>> Been there many times
>
> no, you did not and you did also not in your example below
>
>> Real world example with a package I maintain, which currently has an update
pending in updates-testing:
>>
>> # yum distro-sync
>> ...
>> Downgrading:
>> gumbo-parser x86_64
>> 1.0-0.2.20131001gitd90ea2b.fc20 fedora
>> ...
>> Removed:
>> gumbo-parser.x86_64 0:1.0-0.2.20131204git87b99f2.fc20
>>
>> Installed:
>> gumbo-parser.x86_64 0:1.0-0.2.20131001gitd90ea2b.fc20
>
> nothing is blown away, you only did not read the output
> because it was *downgraded* and *not* removed
Rubbish - Stop being childish.
nobody here is childish, except maybe you
> this is *completly* different than "blown away"
> this is what distro-sync *is supposed to do*
> upgrade or downgrade any package which is in whatever current repo
> but it *does not* blow away packages not in any repo at all
It if the package from updates-testing was fixing a critical bug on your system, your
system would be
malfunctioning afterwards
and exactly *that* was what i said in my first reply while you
stripped *exactly* that part out from your quote, most likely
because you replied with a reflex without read exactly 5 lines
completly
but that is *not* "a) This would blow away all installed packages, which aren't
available in
permanently enabled repos" because that would mean *uninstall* any package which is
currently
not in a enabled repo - and that is *not* what distro-sync does
below *again* my complete reply which is and was technical correct
while your "would blow away" is not
so before call others childish the next time before you reply to a message
read also the second pararaph to avoid useless discussions
-------- Original-Nachricht --------
Betreff: Re: Drawing lessons from fatal SELinux bug #1054350
Datum: Fri, 24 Jan 2014 16:06:21 +0100
Von: Reindl Harald <h.reindl(a)thelounge.net>
An: devel(a)lists.fedoraproject.org
Am 24.01.2014 15:55, schrieb Ralf Corsepius:
On 01/24/2014 01:39 PM, Kevin Kofler wrote:
> Adam Williamson wrote:
>> Even if we can do it on the mirrors, we have no way to 'recall' a
>> package from systems where it's already been installed (of course in the
>> current case that wouldn't have worked anyway, but we're discussing the
>> generic case here).
>
> Crazy idea of the day: Maybe our update tools should default to distro-sync
> rather than update?
No, for 2 reasons:
a) This would blow away all installed packages, which aren't available in permanently
enabled repos
that is not true, try it out
otherwise some packages would be not installed on my machines after a dist-upgrade
namely the ones never came from any repo and installed locally
Most common such case is having selectively installed packages from
updates-testing, because users are facing
problems with these packages' nominal versions
*that* is the reason not to do so because it would downgrade anything updated
explicitly from updates-testing,kde-testing,koji which would be a bad default
10:35 a.m.
Ralf, Harald, you both actually mean the same thing, you're just
misunderstanding each other due to inexact wording!
Yes, distro-sync will not remove packages which are not in the default-
enabled repositories at all (in any version) (nor will it downgrade them,
obviously, because there is no version to downgrade them to).
And yes, distro-sync WILL downgrade packages if the new version is not (or
not yet) available in a default-enabled repository.
It is clear that both of you know this, there was just a misunderstanding.
Kevin Kofler
1:04 a.m.
On 01/24/2014 07:35 AM, Rahul Sundaram wrote:
Hi
On Thu, Jan 23, 2014 at 11:49 PM, Sérgio Basto wrote:
yeah , so revoke an update could be a better idea.
How would that work? We don't control the mirrors.
Fedora controls advertising
mirrors (mirrorlists) through
mirrormanager. I.e. mirrormanager only advertising mirrors, which are
in-sync should keep the risks fairly low.
Certainly, downgrading installations which already upgraded to faulty
packages would not work.
Ralf
3:58 a.m.
2014/1/24 Ralf Corsepius <rc040203(a)freenet.de>
Certainly, downgrading installations which already upgraded to faulty
packages would not work.
Ralf
The situation (a broken system that cannot be upgraded) could be mitigated
a little bit by using yum + system snapshots. You can rollback to a
previous sane system.
There is a plugin yum-plugin-fs-snapshot, but it requires better
documentation and system integration.
Currently (I don't know how current is F16 documentation) it requires
running lvm by hand
http://docs.fedoraproject.org/en-US/Fedora/16/html/System_Administrators_...
Sergio
--
devel mailing list
devel(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
11:41 a.m.
On Fri, 2014-01-24 at 10:58 +0100, Sergio Pascual wrote:
2014/1/24 Ralf Corsepius <rc040203(a)freenet.de>
Certainly, downgrading installations which already upgraded to
faulty packages would not work.
Ralf
The situation (a broken system that cannot be upgraded) could be
mitigated a little bit by using yum + system snapshots. You can
rollback to a previous sane system.
There is a plugin yum-plugin-fs-snapshot, but it requires better
documentation and system integration.
Currently (I don't know how current is F16 documentation) it requires
running lvm by hand
http://docs.fedoraproject.org/en-US/Fedora/16/html/System_Administrators_...
AIUI there is/was a long-term plan to integrate this as core
functionality using btrfs snapshots - in fact that was one of the major
attractions of the idea of switching to btrfs-by-default in the first
place. I believe those involved didn't think the LVM-based
implementation was clean/robust enough to use by default, but a
btrfs-based implementation would be. Do correct me if I'm wrong.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
12:19 p.m.
On Fri, 24 Jan 2014 09:41:13 -0800
Adam Williamson <awilliam(a)redhat.com> wrote:
AIUI there is/was a long-term plan to integrate this as core
functionality using btrfs snapshots - in fact that was one of the
major attractions of the idea of switching to btrfs-by-default in the
first place. I believe those involved didn't think the LVM-based
implementation was clean/robust enough to use by default, but a
btrfs-based implementation would be. Do correct me if I'm wrong.
I don't think snapshots are a partcularly good solution, unless there's
some way to only roll back the rpm/yum transaction without also rolling
back unrelated changes.
kevin
4:10 p.m.
On Jan 24, 2014, at 11:19 AM, Kevin Fenzi <kevin(a)scrye.com> wrote:
On Fri, 24 Jan 2014 09:41:13 -0800
Adam Williamson <awilliam(a)redhat.com> wrote:
> AIUI there is/was a long-term plan to integrate this as core
> functionality using btrfs snapshots - in fact that was one of the
> major attractions of the idea of switching to btrfs-by-default in the
> first place. I believe those involved didn't think the LVM-based
> implementation was clean/robust enough to use by default, but a
> btrfs-based implementation would be. Do correct me if I'm wrong.
I don't think snapshots are a partcularly good solution, unless there's
some way to only roll back the rpm/yum transaction without also rolling
back unrelated changes.
If there is a directory that contains update and non-update related file changes,
that's a problem. If there's segmentation, then this can be done.
Clearly /home needs to be separate (it's OK to take a snapshot but just don't use
it by default in a rollback) or we lose changes in /home in a rollback from the time of
the snapshot to the time of the decision to rollback.
Another possible case it's /etc/ where the either a package or the user could make
changes during the update. Btrfs allows per file snapshots with cp --reflink so there
might be a way to carve the snapshot with a scalpel but I prefer doing it with subvolume
granularity. Plus that granularity translates to LVM.
Chris Murphy
10:41 a.m.
Chris Murphy wrote:
If there is a directory that contains update and non-update related
file
changes, that's a problem. If there's segmentation, then this can be done.
Clearly /home needs to be separate (it's OK to take a snapshot but just
don't use it by default in a rollback) or we lose changes in /home in a
rollback from the time of the snapshot to the time of the decision to
rollback.
Another possible case it's /etc/ where the either a package or the user
could make changes during the update.
There's also /root, and then the most annoying case: /var. /var/lib/rpm
definitely needs to be rolled back, but you DON'T want to roll back things
such as log files in /var/log or systemwide databases (other than the RPM
database).
Kevin Kofler
5:36 p.m.
On Jan 25, 2014, at 9:41 AM, Kevin Kofler <kevin.kofler(a)chello.at> wrote:
Chris Murphy wrote:
> If there is a directory that contains update and non-update related file
> changes, that's a problem. If there's segmentation, then this can be done.
>
> Clearly /home needs to be separate (it's OK to take a snapshot but just
> don't use it by default in a rollback) or we lose changes in /home in a
> rollback from the time of the snapshot to the time of the decision to
> rollback.
>
> Another possible case it's /etc/ where the either a package or the user
> could make changes during the update.
There's also /root, and then the most annoying case: /var. /var/lib/rpm
definitely needs to be rolled back, but you DON'T want to roll back things
such as log files in /var/log or systemwide databases (other than the RPM
database).
Well it might be woefully ignorant for me to say, and seem like flamebaiting, but the
mixing of such domains tells me that the FHS needs revision even outside of the context of
snapshots. It's just that snapshots makes it more obvious the organization is
deficient.
Another weird one for me is /var/lib/libvirt/images which I certainly wouldn't want to
snapshot (more specifically I wouldn't want to rollback by default in the face of a
bad update).
Another way of dealing with this is many more subvolumes so that they can be selectively
snapshotted for rollbacks while others remain persistent. Again it's fine to snapshot
them at the same time also, but the rollback behavior by default would only rollback the
software updates.
For point of comparison, when choosing the default Btrfs layout opensuse's installer
creates three partitions: swap, root (btrfs), home (btrfs). And creates the following
subvolumes on root:
boot/grub2/x86_64-efi
home
opt
srv
tmp
usr/local
var/crash
var/log
var/opt
var/spool
var/tmp
There's more snapshot granularity available with this setup.
Chris Murphy
10:46 a.m.
On Fri, Jan 24, 2014 at 03:10:04PM -0700, Chris Murphy wrote:
On Jan 24, 2014, at 11:19 AM, Kevin Fenzi <kevin(a)scrye.com> wrote:
> On Fri, 24 Jan 2014 09:41:13 -0800
> Adam Williamson <awilliam(a)redhat.com> wrote:
>
>> AIUI there is/was a long-term plan to integrate this as core
>> functionality using btrfs snapshots - in fact that was one of the
>> major attractions of the idea of switching to btrfs-by-default in the
>> first place. I believe those involved didn't think the LVM-based
>> implementation was clean/robust enough to use by default, but a
>> btrfs-based implementation would be. Do correct me if I'm wrong.
>
> I don't think snapshots are a partcularly good solution, unless there's
> some way to only roll back the rpm/yum transaction without also rolling
> back unrelated changes.
If there is a directory that contains update and non-update related file
changes, that's a problem. If there's segmentation, then this can be done.
Clearly /home needs to be separate (it's OK to take a snapshot but just don't
use it by default in a rollback) or we lose changes in /home in a rollback from
the time of the snapshot to the time of the decision to rollback.
Another possible case it's /etc/ where the either a package or the user could
make changes during the update. Btrfs allows per file snapshots with cp
--reflink so there might be a way to carve the snapshot with a scalpel but I
prefer doing it with subvolume granularity. Plus that granularity translates to
LVM.
Note that this situation is perfectly handled by Offline Updates.
After reboot, there aren't collateral changes to filesystem, only upgrade-related
ones. So if there's a need for revert, the previous state is clearly defined.
--
Tomasz Torcz There exists no separation between gods and men:
xmpp: zdzichubg(a)chrome.pl one blends softly casual into the other.
10:50 a.m.
Am 25.01.2014 17:46, schrieb Tomasz Torcz:
Note that this situation is perfectly handled by Offline Updates.
After reboot, there aren't collateral changes to filesystem, only upgrade-related
ones. So if there's a need for revert, the previous state is clearly defined
says who?
UsrMove was as example forced with the excuse to support this as well
as /usr on a own partition beause one snapshot of the system
so and now imagine a common setup
* /boot
* /
* /var
have fun with restore your snapshot or / or /usr where you bomb the rootfs back
and the rpmdb is still like the restore never did happen
1:55 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Sat, 2014-01-25 at 17:46 +0100, Tomasz Torcz wrote:
On Fri, Jan 24, 2014 at 03:10:04PM -0700, Chris Murphy wrote:
>
> On Jan 24, 2014, at 11:19 AM, Kevin Fenzi <kevin(a)scrye.com> wrote:
>
> > On Fri, 24 Jan 2014 09:41:13 -0800
> > Adam Williamson <awilliam(a)redhat.com> wrote:
> >
> >> AIUI there is/was a long-term plan to integrate this as core
> >> functionality using btrfs snapshots - in fact that was one of the
> >> major attractions of the idea of switching to btrfs-by-default in the
> >> first place. I believe those involved didn't think the LVM-based
> >> implementation was clean/robust enough to use by default, but a
> >> btrfs-based implementation would be. Do correct me if I'm wrong.
> >
> > I don't think snapshots are a partcularly good solution, unless
there's
> > some way to only roll back the rpm/yum transaction without also rolling
> > back unrelated changes.
>
>
> If there is a directory that contains update and non-update related file
> changes, that's a problem. If there's segmentation, then this can be done.
>
> Clearly /home needs to be separate (it's OK to take a snapshot but just
don't
> use it by default in a rollback) or we lose changes in /home in a rollback from
> the time of the snapshot to the time of the decision to rollback.
>
> Another possible case it's /etc/ where the either a package or the user could
> make changes during the update. Btrfs allows per file snapshots with cp
> --reflink so there might be a way to carve the snapshot with a scalpel but I
> prefer doing it with subvolume granularity. Plus that granularity translates to
> LVM.
Note that this situation is perfectly handled by Offline Updates.
After reboot, there aren't collateral changes to filesystem, only upgrade-related
ones. So if there's a need for revert, the previous state is clearly defined.
Sorry, but this is simply not true.
I would really like to DISABUSE people of the notion that automated (or
not) rollbacks can be easily done in bulk, by the magic wand of file
system snapshots.
The ONLY way to do that is if you do not care at all about user's data
and simply accept that a rollback will also remove user data.
The reason is simple: lot's of software *changes* data as part of its
normal functioning, including and often in rollback-incompatible ways.
You cannot assume that upgrading a program that uses a database X from
version A to B can still work if you keep database X unchanged and then
rollback from B to A. Lot of applications apply changes to database at
upgrade time, either in the rpm scriplets or automatically as soon as a
new version binary is run.
It is basically impossible to find applications that handle the case
where you downgrade, in any more graceful way than punting and failing
to start in the *good* case. In the bad case they start and trash the
database.
And by database, do not think SQL/NOSQL engines only, it can be any
simple dataset in a file, including configuration files in user's homes.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
2:04 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Hi Simo,
On Sat, 2014-01-25 at 14:55 -0500, Simo Sorce wrote:
The reason is simple: lot's of software *changes* data as part of
its
normal functioning, including and often in rollback-incompatible ways.
I wrote and maintain a system that has been doing continuous deployment
of GNOME. It's been running for over a year, and is nearing it's
10000th build.
I have "rolled" back many times - both on the server side, and on the
client side. Here's one I *just did* a few minutes ago because vala git
master broke the build of gnome-calculator:
https://git.gnome.org/browse/gnome-continuous/commit/?id=32a52e53100e92aa...
That's me telling the system "please stop building git master, and
freeze to this specific commit". All clients get that change when they
upgrade - OSTree cares not at all for version numbers.
The vala maintainers continue to work out the issue in git master, and I
continue to ship a working system. Double win.
Now it's true, programs in GNOME do sometimes make the type of data
format transition you're talking about. Evolution has done it at least
twice.
But you know what? My real world experience has been that having the
ability to roll back has *far* *far* *far* outweighed the downsides when
applications do format transitions. It's comparatively rare.
Far more people are bit by things like hardware-specific issues where
gnome-shell fails to render on this particular card - and rollback works
beautifully for that.
12:41 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Sat, 2014-01-25 at 15:04 -0500, Colin Walters wrote:
Hi Simo,
On Sat, 2014-01-25 at 14:55 -0500, Simo Sorce wrote:
> The reason is simple: lot's of software *changes* data as part of its
> normal functioning, including and often in rollback-incompatible ways.
I wrote and maintain a system that has been doing continuous deployment
of GNOME. It's been running for over a year, and is nearing it's
10000th build.
I have "rolled" back many times - both on the server side, and on the
client side. Here's one I *just did* a few minutes ago because vala git
master broke the build of gnome-calculator:
https://git.gnome.org/browse/gnome-continuous/commit/?id=32a52e53100e92aa...
That's me telling the system "please stop building git master, and
freeze to this specific commit". All clients get that change when they
upgrade - OSTree cares not at all for version numbers.
The vala maintainers continue to work out the issue in git master, and I
continue to ship a working system. Double win.
Now it's true, programs in GNOME do sometimes make the type of data
format transition you're talking about. Evolution has done it at least
twice.
But you know what? My real world experience has been that having the
ability to roll back has *far* *far* *far* outweighed the downsides when
applications do format transitions. It's comparatively rare.
Far more people are bit by things like hardware-specific issues where
gnome-shell fails to render on this particular card - and rollback works
beautifully for that.
I never said it won't work in absolute, it probably will work ok in many
cases, just to cause incredible issues in others.
It is a fine tool in the hands of an expert that knows how to check
whether reverting to a snapshot is safe.
It is not going to be a good solution for non-expert users though
*unless* you provide system APIs that *all* applications use to signal
when they are doing irreversible changes so that the user can be warned
about potential data loss right when he asks the system to revert a
snapshot.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
2:13 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Jan 26, 2014, at 11:41 AM, Simo Sorce <simo(a)redhat.com> wrote:
I never said it won't work in absolute, it probably will work ok
in many
cases, just to cause incredible issues in others.
It is a fine tool in the hands of an expert that knows how to check
whether reverting to a snapshot is safe.
Why is the snapshot case any different from a user who reverts doing a clean install or
yum downgrade?
It is not going to be a good solution for non-expert users though
*unless* you provide system APIs that *all* applications use to signal
when they are doing irreversible changes so that the user can be warned
about potential data loss right when he asks the system to revert a
snapshot.
Developers should not be sneak attacking non-expert users with file format changes that
aren't well announced in advance of consequences they probably won't be able to
read their data if they downgrade the application.
Chris Murphy
2:18 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 26.01.2014 21:13, schrieb Chris Murphy:
On Jan 26, 2014, at 11:41 AM, Simo Sorce <simo(a)redhat.com>
wrote:
> I never said it won't work in absolute, it probably will work ok in many
> cases, just to cause incredible issues in others.
>
> It is a fine tool in the hands of an expert that knows how to check
> whether reverting to a snapshot is safe.
Why is the snapshot case any different from a user who reverts doing a clean install or
yum downgrade?
because the snapshot restores *a whole filesystem* and not only the affected application?
* restore a snapshot of /usr and you have fun with /var/lib/rpm
* restore a snapshot of /var/lib/ without /usr and you have fun with the rpmdb and others
* restore a snapshot of /usr without /etc and you *may have* random fun
and there are *hundrets* of such combinations where the last thing you
really would want is restore a snapshot because you have no plan about
the real-world impact in doing so
> It is not going to be a good solution for non-expert users
though
> *unless* you provide system APIs that *all* applications use to signal
> when they are doing irreversible changes so that the user can be warned
> about potential data loss right when he asks the system to revert a
> snapshot.
Developers should not be sneak attacking non-expert users with file format changes that
aren't well
announced in advance of consequences they probably won't be able to read their data
if they downgrade
the application
the perfect world won't happen, sad but true
5:26 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Jan 26, 2014, at 1:18 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
Am 26.01.2014 21:13, schrieb Chris Murphy:
> On Jan 26, 2014, at 11:41 AM, Simo Sorce <simo(a)redhat.com> wrote:
>
>> I never said it won't work in absolute, it probably will work ok in many
>> cases, just to cause incredible issues in others.
>>
>> It is a fine tool in the hands of an expert that knows how to check
>> whether reverting to a snapshot is safe.
>
> Why is the snapshot case any different from a user who reverts doing a clean install
or yum downgrade?
because the snapshot restores *a whole filesystem* and not only the affected application?
If I knew the problem was with a particular affected application, why would I be using a
snapshot rollback approach or clean install rather than a yum downgrade <app>
approach?
* restore a snapshot of /usr and you have fun with /var/lib/rpm
* restore a snapshot of /var/lib/ without /usr and you have fun with the rpmdb and
others
* restore a snapshot of /usr without /etc and you *may have* random fun
and there are *hundrets* of such combinations where the last thing you
really would want is restore a snapshot because you have no plan about
the real-world impact in doing so
Well what sort of moron would do rollbacks like this? You're saying if someone puts a
stick of dynamite in their mouth then ZOMG! going to die!, but not accounting for why they
would put dynamite in their mouth in the first place. This is simply not how rollbacks are
done. Yes there are hundreds of mindnumbingly stupid ways a user could break their system.
No one is recommending rollbacks that work the way you describe.
Chris Murphy
5:29 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 27.01.2014 00:26, schrieb Chris Murphy:
On Jan 26, 2014, at 1:18 PM, Reindl Harald
<h.reindl(a)thelounge.net> wrote:
> Am 26.01.2014 21:13, schrieb Chris Murphy:
>> On Jan 26, 2014, at 11:41 AM, Simo Sorce <simo(a)redhat.com> wrote:
>>
>>> I never said it won't work in absolute, it probably will work ok in many
>>> cases, just to cause incredible issues in others.
>>>
>>> It is a fine tool in the hands of an expert that knows how to check
>>> whether reverting to a snapshot is safe.
>>
>> Why is the snapshot case any different from a user who reverts doing a clean
install or yum downgrade?
>
> because the snapshot restores *a whole filesystem* and not only the affected
application?
If I knew the problem was with a particular affected application, why would I be using a
snapshot rollback approach or clean install rather than a yum downgrade <app>
approach?
>
> * restore a snapshot of /usr and you have fun with /var/lib/rpm
> * restore a snapshot of /var/lib/ without /usr and you have fun with the rpmdb and
others
> * restore a snapshot of /usr without /etc and you *may have* random fun
>
> and there are *hundrets* of such combinations where the last thing you
> really would want is restore a snapshot because you have no plan about
> the real-world impact in doing so
Well what sort of moron would do rollbacks like this? You're saying if someone puts a
stick of dynamite in their mouth then ZOMG! going to die!, but not accounting for why they
would put dynamite in their mouth in the first place. This is simply not how rollbacks are
done. Yes there are hundreds of mindnumbingly stupid ways a user could break their system.
No one is recommending rollbacks that work the way you describe.
do yourself and everybody a favour and
* don't claim others are rude while you talk like above and worser half of the thread
* don't talk about things above your technical scope
* discuss with software engineers while lacking basic understanding of the topic
posts like yours in that thread belongs to the users list and *not* to a development list
5:51 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Jan 26, 2014, at 4:29 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
do yourself and everybody a favour and
* don't claim others are rude while you talk like above and worser half of the
thread
* don't talk about things above your technical scope
* discuss with software engineers while lacking basic understanding of the topic
posts like yours in that thread belongs to the users list and *not* to a development list
Request declined.
You are the only person who has suggested a method of rollbacks that fundamentally would
not work, and then argued against it. You created what's referred to as, a strawman
argument. Also known as being full of crap. So you can take your silly logical fallacies
and mock victimization and stick 'em where the sun don't shine, Reindl. Put it all
right back where you got your ridiculous snapshot-rollback concept.
Chris Murphy
5:55 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 27.01.2014 00:51, schrieb Chris Murphy:
On Jan 26, 2014, at 4:29 PM, Reindl Harald
<h.reindl(a)thelounge.net> wrote:
> do yourself and everybody a favour and
>
> * don't claim others are rude while you talk like above and worser half of the
thread
> * don't talk about things above your technical scope
> * discuss with software engineers while lacking basic understanding of the topic
>
> posts like yours in that thread belongs to the users list and *not* to a development
list
Request declined.
You are the only person who has suggested a method of rollbacks that fundamentally would
not work
are you drunken?
i have *not* requested any method of rollback
i just gave a few warnings what problems has to be considered if rollbacks
of snapshots are taken as possible solution - so *stop* talk to threads if
you have no clue about the topic and about who said what
* read what others said
* start at the begin of the thread doing that
* try to understand what you read before commetn any word
* look at the *context* of several posts becaus eoyu need that information to understand
* after that claim what people suggested or in reality you would say nothing at all
6:18 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Jan 26, 2014, at 4:55 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
Am 27.01.2014 00:51, schrieb Chris Murphy:
> On Jan 26, 2014, at 4:29 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
>
>> do yourself and everybody a favour and
>>
>> * don't claim others are rude while you talk like above and worser half of
the thread
>> * don't talk about things above your technical scope
>> * discuss with software engineers while lacking basic understanding of the topic
>>
>> posts like yours in that thread belongs to the users list and *not* to a
development list
>
> Request declined.
>
> You are the only person who has suggested a method of rollbacks that fundamentally
would not work
are you drunken?
I haven't had anything to drink in 24+ hours, but this seems off topic.
i have *not* requested any method of rollback
You gave several examples of rollback-snapshot methods - same thing as you suggested them.
I never said you requested them.
i just gave a few warnings what problems has to be considered if rollbacks
of snapshots are taken as possible solution - so *stop* talk to threads if
you have no clue about the topic and about who said what
Yes, problems as a result of improper rollback methods. I will not stop challenging junk
suggestions which you then use to cast a wide net to argue against all forms of
snapshotting and rollback. It's absurd argumentation.
* read what others said
* start at the begin of the thread doing that
* try to understand what you read before commetn any word
* look at the *context* of several posts becaus eoyu need that information to understand
* after that claim what people suggested or in reality you would say nothing at all
Take your own advice. I've been following the thread from the very start, and your
snapshot-rollback examples are all junk. Just for instance:
On Jan 26, 2014, at 1:18 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
* restore a snapshot of /usr and you have fun with /var/lib/rpm
* restore a snapshot of /var/lib/ without /usr and you have fun with the rpmdb and
others
* restore a snapshot of /usr without /etc and you *may have* random fun
Only you have come up with such utterly implausible examples of snapshot-rollback behavior
and then chosen to argue against *all* possible snapshot-rollbacks in general. No one
would do rollbacks the way you describe above. It would almost certainly break the
system.
Chris Murphy
6:20 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 27.01.2014 01:18, schrieb Chris Murphy:
On Jan 26, 2014, at 4:55 PM, Reindl Harald
<h.reindl(a)thelounge.net> wrote:
> Am 27.01.2014 00:51, schrieb Chris Murphy:
>> On Jan 26, 2014, at 4:29 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
>>
>>> do yourself and everybody a favour and
>>>
>>> * don't claim others are rude while you talk like above and worser half
of the thread
>>> * don't talk about things above your technical scope
>>> * discuss with software engineers while lacking basic understanding of the
topic
>>>
>>> posts like yours in that thread belongs to the users list and *not* to a
development list
>>
>> Request declined.
>>
>> You are the only person who has suggested a method of rollbacks that
fundamentally would not work
>
> are you drunken?
I haven't had anything to drink in 24+ hours, but this seems off topic.
>
> i have *not* requested any method of rollback
You gave several examples of rollback-snapshot methods - same thing as you suggested
them. I never said you requested them
oh my god - i gave several examples *what could be dangerous* in doing that
i *never* ever suggested them
please re-read the thread and then come back with an excuse
6:32 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Jan 26, 2014, at 5:20 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
Am 27.01.2014 01:18, schrieb Chris Murphy:
> On Jan 26, 2014, at 4:55 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
>> Am 27.01.2014 00:51, schrieb Chris Murphy:
>>> On Jan 26, 2014, at 4:29 PM, Reindl Harald <h.reindl(a)thelounge.net>
wrote:
>>>
>>>> do yourself and everybody a favour and
>>>>
>>>> * don't claim others are rude while you talk like above and worser
half of the thread
>>>> * don't talk about things above your technical scope
>>>> * discuss with software engineers while lacking basic understanding of
the topic
>>>>
>>>> posts like yours in that thread belongs to the users list and *not* to a
development list
>>>
>>> Request declined.
>>>
>>> You are the only person who has suggested a method of rollbacks that
fundamentally would not work
>>
>> are you drunken?
>
> I haven't had anything to drink in 24+ hours, but this seems off topic.
>>
>> i have *not* requested any method of rollback
>
> You gave several examples of rollback-snapshot methods - same thing as you suggested
them. I never said you requested them
oh my god - i gave several examples *what could be dangerous* in doing that
i *never* ever suggested them
please re-read the thread and then come back with an excuse
"suggested them" can mean two things in English: you recommend them, or they are
examples. I mean the 2nd case. I understand that you were not ever recommending your
examples. You were suggesting them as examples why snapshots in general are bad.
The problem is that your examples are crap. They're bad examples because they would
break the system, therefore no one would actually do snapshots-rollbacks per your
examples, unless they wanted to blow up their system.
Chris Murphy
6:37 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 27.01.2014 01:32, schrieb Chris Murphy:
On Jan 26, 2014, at 5:20 PM, Reindl Harald
<h.reindl(a)thelounge.net> wrote:
> Am 27.01.2014 01:18, schrieb Chris Murphy:
>> You gave several examples of rollback-snapshot methods - same thing as you
suggested them. I never said you requested them
>
> oh my god - i gave several examples *what could be dangerous* in doing that
> i *never* ever suggested them
> please re-read the thread and then come back with an excuse
"suggested them" can mean two things in English: you recommend them, or they
are examples. I mean the 2nd case. I understand that you were not ever recommending your
examples. You were suggesting them as examples why snapshots in general are bad.
The problem is that your examples are crap. They're bad examples because they would
break the system, therefore no one would actually do snapshots-rollbacks per your
examples, unless they wanted to blow up their system.
boah the fact "therefore no one would actually do snapshots-rollbacks per your
examples" needs to be proven
i only just warned about cases where a rollback would do harm and to *make sure* that
really no one would
do it without take care
so where is now the point you started a flamewar against me instead
be quite or say "ok, that would be bad and hopefully does not happen"
this is a *dvelopent dicussion* and the goal of it is to *prevent*
mistakes ever happen *before* they are implemented or widely used
7:11 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Jan 26, 2014, at 5:37 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
Am 27.01.2014 01:32, schrieb Chris Murphy:
> On Jan 26, 2014, at 5:20 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
>> Am 27.01.2014 01:18, schrieb Chris Murphy:
>>> You gave several examples of rollback-snapshot methods - same thing as you
suggested them. I never said you requested them
>>
>> oh my god - i gave several examples *what could be dangerous* in doing that
>> i *never* ever suggested them
>> please re-read the thread and then come back with an excuse
>
> "suggested them" can mean two things in English: you recommend them, or
they are examples. I mean the 2nd case. I understand that you were not ever recommending
your examples. You were suggesting them as examples why snapshots in general are bad.
>
> The problem is that your examples are crap. They're bad examples because they
would break the system, therefore no one would actually do snapshots-rollbacks per your
examples, unless they wanted to blow up their system.
boah the fact "therefore no one would actually do snapshots-rollbacks per your
examples" needs to be proven
Really? That seems like saying "no one would stick dynamite in their mouth unless
they wanted to die" needs to be proven. I think it will only take a handful of such
instances to convince most rational people this isn't a good course of action.
i only just warned about cases where a rollback would do harm and to
*make sure* that really no one would
do it without take care
That was my *entire* point going back around 36 hours ago…
> Chris Murphy wrote:
> If there is a directory that contains update and non-update related file
> changes, that's a problem. If there's segmentation, then this can be done.
>
> Clearly /home needs to be separate (it's OK to take a snapshot but just
> don't use it by default in a rollback) or we lose changes in /home in a
> rollback from the time of the snapshot to the time of the decision to
> rollback.
>
> Another possible case it's /etc/ where the either a package or the user
> could make changes during the update.
so where is now the point you started a flamewar against me instead
be quite or say "ok, that would be bad and hopefully does not happen"
I did in fact state your examples were FUD. Where the flaming starts is when you said
"blabla - nobody talks about the mailserver" when Simo *had* just mentioned
server side changes which is what I was responding to. And "blabla" is just
f'n rude from the outset, so yeah I'm going to be a bit of a dick when someone is
a.) condescending, b.) says no one said X when someone did in fact say X; and c.) deletes
the reply where someone said X; and d.) proceeds with a dozen emails about how I'm the
one not paying attention when I asked for context clarification and you decided to jump
down my throat and it went downhill quickly from there.
I do mostly just monitor this list, for several years. When people jump on you, are you
quiet? No, you jump right back and you argue like hell. So don't tell me that I should
be quiet, or how I should respond. From my perspective you were picking a fight, so I
decide to play along and maybe mine was a little bit disproportionate of a response, but
don't play victim just because you got burned.
this is a *dvelopent dicussion* and the goal of it is to *prevent*
mistakes ever happen *before* they are implemented or widely used
Which is exactly why I've involved myself in a thread on snapshotting because unlike
you, I have been doing snapshots and rollbacks with LVM and Btrfs for quite a few years.
I'm aware that there are some challenges that users will likely face and development
needs to account for these things so they aren't easily getting into trouble or
confused about where their data is.
Snapshots are a reality, simply sticking our head in the sand for a feature people have
been asking for is simply not the way forward. I am not suggesting at all that your
workflow should change to include snapshots, so I ask that you have the courtesy to not
claim with bad examples that snapshots generally are a bad idea that will hose user's
systems and make developers lazy and careless. This is an entirely voluntary project, you
are not required to participate in some aspect of its technology you don't use and
seem to not even care about.
Chris Murphy
7:34 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 27.01.2014 02:11, schrieb Chris Murphy:
> i only just warned about cases where a rollback would do harm and
to *make sure* that really no one would
> do it without take care
That was my *entire* point going back around 36 hours ago
and that is why i do not understand your turn around 180 degree
against Simo and me beause *we both supported* your initial
viewpoint until you started to claim all the cases are invalid
I did in fact state your examples were FUD
with no reason to do so
Where the flaming starts is when you said "blabla - nobody talks
about the mailserver"
when Simo *had* just mentioned server side changes which is what I was responding to
hmpf - read again - "server side changes" != "mailservers"
after that you told about Apple Mail and what not and then switeched to mailservers
my problem was that you truned 180 degree and fighted against any argument going
in the direction where restore of snapshots may be tricky and dangerous while you
orginally before the subject changed even said the same
And "blabla" is just f'n rude from the outset
because i had already enough of the turn 180 degree around and your
again and again argumentation about user documents and that they
don't change their format while never said that with a single line
so yeah I'm going to be a bit of a dick when someone is a.)
condescending, b.) says
no one said X when someone did in fact say X
still: nobody said "mailserver", but forget it
and c.) deletes the reply where someone said X
server side changes doe snot imply change *the format* how mails itself are stored
and d.) proceeds with a dozen emails about how I'm the one not
paying attention when I
asked for context clarification and you decided to jump down my throat and it went
downhill quickly from there.
then you maybe should have asked *only* about clarification instead start
calling developers names if they would change the format of user documents
which was *never* part of the context
I do mostly just monitor this list, for several years. When people
jump on you,
are you quiet?
no, but i am not a dickhead and listen if people telling me that
talking about user documents is not any part of the discussion
in case of downgrades and internal environment data of applications
may have changed unnoticed
No, you jump right back and you argue like hell. So don't tell me
that I
should be quiet, or how I should respond
and if you really would look you have noticed a difference
From my perspective you were picking a fight
if that would be true i have called you a lot of names in the public
which i *really* avoided while with some replies you begged for it
so I decide to play along
why?
and maybe mine was a little bit disproportionate of a response
a little? come on!
but don't play victim just because you got burned
please calm more down and re-read the whole thread including the point
Simo even gave up completly to try explain you the context
Which is exactly why I've involved myself in a thread on
snapshotting because
unlike you, I have been doing snapshots and rollbacks with LVM and Btrfs for
quite a few years.
i statet that i do not use snapshots nor the graphical stuf fnor gnome to make
clear *i am not affected* of any decision in that direction but *care* about
others, otherwise the whole sub-thread would not have been relevant for me
I'm aware that there are some challenges that users will likely
face and development
needs to account for these things so they aren't easily getting into trouble or
confused
about where their data is.
which was my whole point
Snapshots are a reality, simply sticking our head in the sand for a
feature people
have been asking for is simply not the way forward. I am not suggesting at all that
your workflow should change to include snapshots, so I ask that you have the courtesy
to not claim with bad examples that snapshots generally are a bad idea that will hose
user's systems and make developers lazy and careless
i did not say anything about snapshots in general
the topic is "snapshots in case of updates and make it easy to roll them back"
this needs *a lot of special care* that is my whole point
This is an entirely voluntary project, you are not required to
participate in some
aspect of its technology you don't use and seem to not even care about
sorry that i case about the project in general
5:57 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
I don't think this subthread is being particularly useful...
And the personal attacks are undesirable.
Please stop or at least take it to private email.
Thanks,
kevin
6:11 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 27.01.2014 00:57, schrieb Kevin Fenzi:
I don't think this subthread is being particularly useful...
And the personal attacks are undesirable.
Please stop or at least take it to private email
*sorry* for not early enough realize trolling in first start with
the same argumentation as Simon and me to later fight against it
while now claim i came up with the idea of snapshots while
warning all the time and tried to explain Chris *why* i
warn
-------- Original-Nachricht --------
Betreff: Re: Drawing lessons from fatal SELinux bug #1054350
Datum: Sat, 25 Jan 2014 16:42:13 -0700
Von: Chris Murphy <lists(a)colorremedies.com>
Antwort an: Development discussions related to Fedora
<devel(a)lists.fedoraproject.org>
An: Development discussions related to Fedora <devel(a)lists.fedoraproject.org>
I don't follow this. The realization an update is bad doesn't necessarily occur
right away. So we still need a way
to separate system domain vs user domain, at least, so that system files are rolled back
separately from user files
_______________________________________________________
can someone *please stop that troll telling lies*
And then you propose a ridonkulous snapshot-rollback strategy that
would for certain cause
major problems if the rollback were actually done, and then use that as fait accompli
for
why the entire concept of fs rollbacks are stupid. Your arguments are asinine. Your
emails
belong in a kill file.
2:22 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Sun, 2014-01-26 at 13:13 -0700, Chris Murphy wrote:
On Jan 26, 2014, at 11:41 AM, Simo Sorce <simo(a)redhat.com>
wrote:
> I never said it won't work in absolute, it probably will work ok in many
> cases, just to cause incredible issues in others.
>
> It is a fine tool in the hands of an expert that knows how to check
> whether reverting to a snapshot is safe.
Why is the snapshot case any different from a user who reverts doing a
clean install or yum downgrade?
It is not.
> It is not going to be a good solution for non-expert users
though
> *unless* you provide system APIs that *all* applications use to signal
> when they are doing irreversible changes so that the user can be warned
> about potential data loss right when he asks the system to revert a
> snapshot.
Developers should not be sneak attacking non-expert users with file
format changes that aren't well announced in advance of consequences
they probably won't be able to read their data if they downgrade the
application.
Users should not expect miracles either.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
4:26 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Sat, Jan 25, 2014 at 02:55:32PM -0500, Simo Sorce wrote:
On Sat, 2014-01-25 at 17:46 +0100, Tomasz Torcz wrote:
> >
> > If there is a directory that contains update and non-update related file
> > changes, that's a problem. If there's segmentation, then this can be
done.
>
> Note that this situation is perfectly handled by Offline Updates.
> After reboot, there aren't collateral changes to filesystem, only
upgrade-related
> ones. So if there's a need for revert, the previous state is clearly defined.
Sorry, but this is simply not true.
The ONLY way to do that is if you do not care at all about user's data
and simply accept that a rollback will also remove user data.
The reason is simple: lot's of software *changes* data as part of its
normal functioning, including and often in rollback-incompatible ways.
What user data? There is no user data touched/created during offline upgrade.
--
Tomasz Torcz There exists no separation between gods and men:
xmpp: zdzichubg(a)chrome.pl one blends softly casual into the other.
4:29 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 25.01.2014 23:26, schrieb Tomasz Torcz:
On Sat, Jan 25, 2014 at 02:55:32PM -0500, Simo Sorce wrote:
> The ONLY way to do that is if you do not care at all about user's data
> and simply accept that a rollback will also remove user data.
>
> The reason is simple: lot's of software *changes* data as part of its
> normal functioning, including and often in rollback-incompatible ways.
What user data? There is no user data touched/created during offline upgrade
and what is with the data *between* the upgrade and decision to roll back?
5:12 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Sat, 2014-01-25 at 23:26 +0100, Tomasz Torcz wrote:
On Sat, Jan 25, 2014 at 02:55:32PM -0500, Simo Sorce wrote:
> On Sat, 2014-01-25 at 17:46 +0100, Tomasz Torcz wrote:
> > >
> > > If there is a directory that contains update and non-update related file
> > > changes, that's a problem. If there's segmentation, then this can
be done.
> >
> > Note that this situation is perfectly handled by Offline Updates.
> > After reboot, there aren't collateral changes to filesystem, only
upgrade-related
> > ones. So if there's a need for revert, the previous state is clearly
defined.
>
> Sorry, but this is simply not true.
>
>
> The ONLY way to do that is if you do not care at all about user's data
> and simply accept that a rollback will also remove user data.
>
> The reason is simple: lot's of software *changes* data as part of its
> normal functioning, including and often in rollback-incompatible ways.
What user data? There is no user data touched/created during offline upgrade.
No, but you may have to use the system somewhat before you can find out
there was a problem with the upgrade, and at *that* point, your user
data may now be tied to the new versions of system apps, as Simo
describes.
So, it goes like this:
* Do an offline update that includes Foo v2.0
* Boot the updated system, run Foo, it migrates its configuration to
some new scheme
* Realize there was something wrong with the update, roll it back
* Run Foo again, find it doesn't work because it's been migrated to the
new config scheme which the old version of Foo doesn't work with
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
6:54 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Jan 25, 2014, at 4:12 PM, Adam Williamson <awilliam(a)redhat.com> wrote:
* Do an offline update that includes Foo v2.0
* Boot the updated system, run Foo, it migrates its configuration to
some new scheme
* Realize there was something wrong with the update, roll it back
* Run Foo again, find it doesn't work because it's been migrated to the
new config scheme which the old version of Foo doesn't work with
I would grumble, but a configuration file being updated and made incompatible with the
prior version would be tolerated. Ideally the application makes an unmodified copy. If it
doesn't, new school restore with --reflink from snapshot, regular cp if using LVM
thinp snapshots, and old school just restore the file from a conventional backup. Not such
a big deal.
If it's something far less throw away than configuration files being changed, it's
a bit more complicated how badly and quickly the conversation degrades. But I can hardly
recall a recent example of this happening. It's just not that common in my
experience.
Chris Murphy
7:10 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 26.01.2014 01:54, schrieb Chris Murphy:
On Jan 25, 2014, at 4:12 PM, Adam Williamson
<awilliam(a)redhat.com> wrote:
>
> * Do an offline update that includes Foo v2.0
> * Boot the updated system, run Foo, it migrates its configuration to
> some new scheme
> * Realize there was something wrong with the update, roll it back
> * Run Foo again, find it doesn't work because it's been migrated to the
> new config scheme which the old version of Foo doesn't work with
I would grumble, but a configuration file being updated and made incompatible with the
prior version would be tolerated. Ideally the application makes an unmodified copy. If it
doesn't, new school restore with --reflink from snapshot, regular cp if using LVM
thinp snapshots, and old school just restore the file from a conventional backup. Not such
a big deal
the short version of ahwat you said could have been "forget snapshots at all to
solve
such problems" to not lead dvelopers into temptation of "i can be less caeful
because
we have snapshots"
in other words: don't work around problems by create new ones
12:38 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Sat, 2014-01-25 at 17:54 -0700, Chris Murphy wrote:
On Jan 25, 2014, at 4:12 PM, Adam Williamson
<awilliam(a)redhat.com> wrote:
>
> * Do an offline update that includes Foo v2.0
> * Boot the updated system, run Foo, it migrates its configuration to
> some new scheme
> * Realize there was something wrong with the update, roll it back
> * Run Foo again, find it doesn't work because it's been migrated to the
> new config scheme which the old version of Foo doesn't work with
I would grumble, but a configuration file being updated and made
incompatible with the prior version would be tolerated. Ideally the
application makes an unmodified copy. If it doesn't, new school
restore with --reflink from snapshot, regular cp if using LVM thinp
snapshots, and old school just restore the file from a conventional
backup. Not such a big deal.
If it's something far less throw away than configuration files being
changed, it's a bit more complicated how badly and quickly the
conversation degrades. But I can hardly recall a recent example of
this happening. It's just not that common in my experience.
What about mail application change the format of the mail folders ?
It happens, and it is *not* data you want to risk throwing away. There
are many other examples like this especially on the server side.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
1:56 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Jan 26, 2014, at 11:38 AM, Simo Sorce <simo(a)redhat.com> wrote:
On Sat, 2014-01-25 at 17:54 -0700, Chris Murphy wrote:
> On Jan 25, 2014, at 4:12 PM, Adam Williamson <awilliam(a)redhat.com> wrote:
>>
>> * Do an offline update that includes Foo v2.0
>> * Boot the updated system, run Foo, it migrates its configuration to
>> some new scheme
>> * Realize there was something wrong with the update, roll it back
>> * Run Foo again, find it doesn't work because it's been migrated to the
>> new config scheme which the old version of Foo doesn't work with
>
> I would grumble, but a configuration file being updated and made
> incompatible with the prior version would be tolerated. Ideally the
> application makes an unmodified copy. If it doesn't, new school
> restore with --reflink from snapshot, regular cp if using LVM thinp
> snapshots, and old school just restore the file from a conventional
> backup. Not such a big deal.
>
> If it's something far less throw away than configuration files being
> changed, it's a bit more complicated how badly and quickly the
> conversation degrades. But I can hardly recall a recent example of
> this happening. It's just not that common in my experience.
What about mail application change the format of the mail folders ?
Good question because I experienced this recently. So the way Apple does this on OS X with
Mail, there is no such thing as a mail format change during the life of a major OS
version. So major OS versions are 10.7, 10.8, and now 10.9. So it's impossible the
mail format would change between 10.7.1 and 10.7.2 in such a way that 10.7.2 mail could
not be read by the 10.7.1 or 10.7.0 mail application. I can upgrade and downgrade all
along 10.7.x and the file format is the same.
Recently I learned that there are two mail formats. There's the every day used format
that is apparently completely incompatible between major versions of Mail. That is, 10.8.x
Mail will not read (or write) 10.7.x Mail. I must first do an export from 10.7 Mail, to an
intermediate format, that 10.8 Mail can read. Likewise, I can export 10.8 Mail in this
format, and 10.7 Mail can also read it. And even this pissed me off as well as several
thousand users (at least) based on Apple's community forums on the subject because
most of us expect to be able to directly import 10.7 mail into 10.8 Mail. But at least
there is a means to downgrade even to a major version of Mail should I wish.
It happens, and it is *not* data you want to risk throwing away.
There
are many other examples like this especially on the server side.
Well, the mail servers regularly get updated by the company I pay for such things, and
I've never noticed the change. It uses IMAP so I don't think the server even
cares, its just a bunch of folders and files. Apple Mail and the Android phone read mails
from the same server, files and folders without problems. So from my perspective nothing
has changed format wise.
Chris Murphy
2:07 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 26.01.2014 20:56, schrieb Chris Murphy:
> What about mail application change the format of the mail folders
?
Good question because I experienced this recently. So the way Apple does this on OS X
with Mail,
there is no such thing as a mail format change during the life of a major OS version. So
major
OS versions are 10.7, 10.8, and now 10.9
and you have a warranty for that?
So it's impossible the mail format would change between 10.7.1
and 10.7.2 in such a way that 10.7.2
mail could not be read by the 10.7.1 or 10.7.0 mail application
and you have a warranty for that?
no you do *not*
I can upgrade and downgrade all along 10.7.x and the file format is
the same.
you *think* that
Recently I learned that there are two mail formats. There's the
every day used format that
is apparently completely incompatible between major versions of Mail
I can export 10.8 Mail in this format, and 10.7 Mail can also read it. And even this
pissed
me off as well as several thousand users (at least) based on Apple's community forums
on the
subject because most of us expect to be able to directly import 10.7 mail into 10.8 Mail.
where you prove that what you said above is wrong
Well, the mail servers regularly get updated by the company I pay for
such things, and I've
never noticed the change. It uses IMAP so I don't think the server even cares, its
just a bunch
of folders and files
blabla - nobody talks about the mailserver
the topic is *internal* data of *local* software
you may have luck and nothing happens
with bad luck you even won't realize that there are new mails you never face
because of happy upgrade/downgrade internal caches are accessed with
*undefined bahvior*
any software on that planet will recognize upgrades and convert *internal* data
but nobody will give you a warranty how the same software behaves after a downgrade
yes, in most cases nothing bad happens
in rare cases you recognize the problem and find a solution
in some cases you even don't recognize that internal things are slightly going wrong
2:56 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Jan 26, 2014, at 1:07 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
Am 26.01.2014 20:56, schrieb Chris Murphy:
>> What about mail application change the format of the mail folders ?
>
> Good question because I experienced this recently. So the way Apple does this on OS X
with Mail,
> there is no such thing as a mail format change during the life of a major OS version.
So major
> OS versions are 10.7, 10.8, and now 10.9
and you have a warranty for that?
It's a long standing expectation and tradition, and considering millions of users had
this sort of behavior occurred by now no doubt I'd have heard about it.
> I can upgrade and downgrade all along 10.7.x and the file format is the same.
you *think* that
If there is a change, it's not a downward incompatible change.
> Recently I learned that there are two mail formats. There's the every day used
format that
> is apparently completely incompatible between major versions of Mail
> I can export 10.8 Mail in this format, and 10.7 Mail can also read it. And even this
pissed
> me off as well as several thousand users (at least) based on Apple's community
forums on the
> subject because most of us expect to be able to directly import 10.7 mail into 10.8
Mail.
where you prove that what you said above is wrong
No you just have reading comprehension problem. The minor versions are compatible. The
major versions aren't.
> Well, the mail servers regularly get updated by the company I pay
for such things, and I've
> never noticed the change. It uses IMAP so I don't think the server even cares,
its just a bunch
> of folders and files
blabla - nobody talks about the mailserver
Jerk. Simo said, in the line right above this that you cut: "There are many other
examples like this especially on the server side."
the topic is *internal* data of *local* software
you may have luck and nothing happens
This was not at all made clear from the start, it was assumed by people who understood. I
explicitly asked if I was on the same page or not. Instead of bringing me up to speed, you
decide to be condescending. Congratulations on your rudeness.
with bad luck you even won't realize that there are new mails you
never face
because of happy upgrade/downgrade internal caches are accessed with
*undefined bahvior*
Email are user documents the same as a Libreoffice document. You do not get to say that
just because it's a semi-hidden database, that its file format is allowed to change in
a downward incompatible manner. I will disagree with that position at every possible turn
as something between sloppy programming and dereliction.
any software on that planet will recognize upgrades and convert *internal* data
but nobody will give you a warranty how the same software behaves after a downgrade
Well insofar as the whole software EULA paradigm basically says for any reason, willful or
not, they can blow up your data in any direction possible and there is no liability claim
whatsoever… what you're saying doesn't even apply to upgrades.
yes, in most cases nothing bad happens
in rare cases you recognize the problem and find a solution
in some cases you even don't recognize that internal things are slightly going wrong
It's no worse a risk than a conventional reversion with a clean install. So I fail to
see how any of this relates to snapshots.
Chris Murphy
3:07 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 26.01.2014 21:56, schrieb Chris Murphy:
On Jan 26, 2014, at 1:07 PM, Reindl Harald
<h.reindl(a)thelounge.net> wrote:
>> Well, the mail servers regularly get updated by the company I pay for such
things, and I've
>> never noticed the change. It uses IMAP so I don't think the server even
cares, its just a bunch
>> of folders and files
>
> blabla - nobody talks about the mailserver
Jerk. Simo said, in the line right above this that you cut: "There are many other
examples like this especially on the server side."
be careful in which context you somebody calls a Jerk
> the topic is *internal* data of *local* software
> you may have luck and nothing happens
This was not at all made clear from the start, it was assumed by people who understood
because that people thought somebody with that much replies
to the thread would have understodd the topic
I explicitly asked if I was on the same page or not. Instead of
bringing me up to speed,
you decide to be condescending. Congratulations on your rudeness
as you can see some lines above you needed *exactly* that way of comminucation
to understand what we are talking about in this thread - this is the *dvel* list
and so technical understanding is implicit in a discussion
> with bad luck you even won't realize that there are new mails
you never face
> because of happy upgrade/downgrade internal caches are accessed with
> *undefined bahvior*
Email are user documents the same as a Libreoffice document. You do not get to say that
just
because it's a semi-hidden database, that its file format is allowed to change in a
downward
incompatible manner
what exactly did you not understand in the two words "internal caches"
frankly i faced mail clients where you needed to remove the complete
IMAP account to stop not display any new or moved message in specific
folders
> any software on that planet will recognize upgrades and convert
*internal* data
> but nobody will give you a warranty how the same software behaves after a downgrade
Well insofar as the whole software EULA paradigm basically says for any reason, willful
or not, they can blow up your data in any direction possible and there is no liability
claim whatsoever… what you're saying doesn't even apply to upgrades.
google for "undefined behavior"
> yes, in most cases nothing bad happens
> in rare cases you recognize the problem and find a solution
> in some cases you even don't recognize that internal things are slightly going
wrong
It's no worse a risk than a conventional reversion with a clean install
well, i never re-installed any linux system in my life for good reasons
the same reasons i never would restore a snapshot of my whole filesystem
or even more worse *a complete tree* alone of it
So I fail to see how any of this relates to snapshots
that you fail to see the possible impact of a snapshot-restore is obviously
and you do not need to repeat that again and again
3:10 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 26.01.2014 21:56, schrieb Chris Murphy:
No you just have reading comprehension problem. The minor versions
are compatible. The major versions aren't
one last question: what are firefox updates 25->26->27
minor, major, dunno?
more and more software has no minor/major splitting at all
systemd, firefox, chrome..........
what warranties do you have?
none!
what warranties did you ever have?
none as long the specific devloper did not make any promises
luck is no warranty as well as "until now" is not
6:28 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Jan 25, 2014, at 12:55 PM, Simo Sorce <simo(a)redhat.com> wrote:
The reason is simple: lot's of software *changes* data as part of
its
normal functioning, including and often in rollback-incompatible ways.
You cannot assume that upgrading a program that uses a database X from
version A to B can still work if you keep database X unchanged and then
rollback from B to A. Lot of applications apply changes to database at
upgrade time, either in the rpm scriplets or automatically as soon as a
new version binary is run.
I wouldn't ever expect this with a minor version or security update. I'd consider
it a complete betrayal of software versioning to do this. In fact in certain instances of
major version changes, downward compatibility of the file format is expected.
It is basically impossible to find applications that handle the case
where you downgrade, in any more graceful way than punting and failing
to start in the *good* case. In the bad case they start and trash the
database.
I guess I'm not really following. Do you have a for instance? Because off hand this
sounds like a kind of sabotage to me. If it's throw away database info that can simply
be reconstructed I'd probably tolerate it. But for that matter, such things should go
in an defined cache location so that it's not even being backed up.
But important user data having it's format updated in a way that makes it incompatible
with the previous minor version (same major version)? I'm snickering at the language
that would ensue in the proprietary software world, if I'm not totally confused about
what it is you're suggested is fair game. It'd be the sort of language you
wouldn't want your teenager or grandmother to hear.
Chris Murphy
6:45 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 26.01.2014 01:28, schrieb Chris Murphy:
> It is basically impossible to find applications that handle the
case
> where you downgrade, in any more graceful way than punting and failing
> to start in the *good* case. In the bad case they start and trash the
> database.
But important user data having it's format updated in a way that makes it
incompatible
with the previous minor version (same major version)? I'm snickering at the language
that
would ensue in the proprietary software world
you do not know what happens in case *of a bug*
you are in the area of undefined behavior
the point is that the snapshot *does not* bring you for
sure back where you came from or if it does you may regret
it because there is a timewindow between 3 steps
* snapshot / update
* continue your normal operations
* recognize there is a problem
* restore the snapshot
_________________________
* if i have /var on a seperate partition *god beware* of the idea
rollback a snapshot of the remaining rootfs because the system
is ruined -> /var contains the whole rpm-database
* if i have /home on the same FS as the rootfs -> *god beware* of restore
a snapshot because all work before "recognize there is a problem" is
ruined
_________________________
well, people already statet the solution maybe split the OS granulary
and extend the FHS -> that will *not* solve the problem, it only will
create new ones beause at the end of the day nobody except very few
people know what is hwre stored, snapshotted and can be restored with
what exactly impact leading to lose any control
a bug is a bug and in case of undefined bahvior the word *undefined*
is the really important - frankly, what happens if one of the components
used for snapshots is affected?
* nothing
* undefined system state
* all get trashed
solving problems by add more layers of complexity leads to have more
layers prone for bugs themself and the IT after 2010 tends to solve
that by wrap another layer around..... frustrating.....
Linux would not exist if Unix would not have made it a different way and
people coming up with technical complex solutions should consider how
it can be that 30 or 40 years old solutions still working perfect and
all the new ones are replaced every 2-5 years
7:38 a.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Chris Murphy wrote:
I wouldn't ever expect this with a minor version or security
update. I'd
consider it a complete betrayal of software versioning to do this. In fact
in certain instances of major version changes, downward compatibility of
the file format is expected.
The compatibility is often only one way, i.e. newer versions can read older
config files just fine, but not the other way round.
Kevin Kofler
12:35 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Sat, 2014-01-25 at 17:28 -0700, Chris Murphy wrote:
On Jan 25, 2014, at 12:55 PM, Simo Sorce <simo(a)redhat.com>
wrote:
> The reason is simple: lot's of software *changes* data as part of its
> normal functioning, including and often in rollback-incompatible ways.
>
> You cannot assume that upgrading a program that uses a database X from
> version A to B can still work if you keep database X unchanged and then
> rollback from B to A. Lot of applications apply changes to database at
> upgrade time, either in the rpm scriplets or automatically as soon as a
> new version binary is run.
I wouldn't ever expect this with a minor version or security update.
I'd consider it a complete betrayal of software versioning to do this.
In fact in certain instances of major version changes, downward
compatibility of the file format is expected.
And who ever said 'minor' version ?
However I've done this with minor versions too with internal databases.
There is no betrayal whatsoever, major versions are introduced when you
make user-visible changes or you break an API/ABI, you do not
necessarily change major version for some internal change.
> It is basically impossible to find applications that handle the
case
> where you downgrade, in any more graceful way than punting and
failing
> to start in the *good* case. In the bad case they start and trash
the
> database.
I guess I'm not really following. Do you have a for instance?
At least 3 or 4 applications I am involved with did this kind of
internal changes.
Because off hand this sounds like a kind of sabotage to me.
No, it is just normal, everyday, software development.
If it's throw away database info that can simply be
reconstructed I'd
probably tolerate it. But for that matter, such things should go in an
defined cache location so that it's not even being backed up.
In some cases it is data that can be reconstructed, so all you need to
do is to manually blow away the files and let the app rebuild them. In
some cases that also have additional inconveniences. In some cases it is
not data you can or want to throw away.
But important user data having it's format updated in a way that
makes
it incompatible with the previous minor version (same major version)?
So ?
It is only visible if you downgrade which a lot of software do not
support and explicitly so.
I'm snickering at the language that would ensue in the
proprietary
software world, if I'm not totally confused about what it is you're
suggested is fair game. It'd be the sort of language you wouldn't want
your teenager or grandmother to hear.
??
Simo.
--
Simo Sorce * Red Hat, Inc * New York
1:45 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Jan 26, 2014, at 11:35 AM, Simo Sorce <simo(a)redhat.com> wrote:
On Sat, 2014-01-25 at 17:28 -0700, Chris Murphy wrote:
> On Jan 25, 2014, at 12:55 PM, Simo Sorce <simo(a)redhat.com> wrote:
>
>> The reason is simple: lot's of software *changes* data as part of its
>> normal functioning, including and often in rollback-incompatible ways.
>>
>> You cannot assume that upgrading a program that uses a database X from
>> version A to B can still work if you keep database X unchanged and then
>> rollback from B to A. Lot of applications apply changes to database at
>> upgrade time, either in the rpm scriplets or automatically as soon as a
>> new version binary is run.
>
> I wouldn't ever expect this with a minor version or security update.
> I'd consider it a complete betrayal of software versioning to do this.
> In fact in certain instances of major version changes, downward
> compatibility of the file format is expected.
And who ever said 'minor' version ?
I'm saying it. I don't expect the behavior you describe with minor version or
security updates.
However I've done this with minor versions too with internal databases.
There is no betrayal whatsoever, major versions are introduced when you
make user-visible changes or you break an API/ABI, you do not
necessarily change major version for some internal change.
Again, are we talking user data or configuration files?
If a minor version update of LibreOffice were to read my documents and write them out in a
new format that the prior version could not read? I'm actually confused by how I'd
react - honestly. I'm not sure if I'd file a bug with "this must be a bug, is
it intended" or if I'd come out a public list with guns blazing and accuse the
development team of gross incompetency (and that, honestly, would be kind language).
>> It is basically impossible to find applications that handle
the case
>> where you downgrade, in any more graceful way than punting and
> failing
>> to start in the *good* case. In the bad case they start and trash
> the
>> database.
>
> I guess I'm not really following. Do you have a for instance?
At least 3 or 4 applications I am involved with did this kind of
internal changes.
I still really have no idea what sorts of changes you're talking about. Maybe database
people are really tolerant of format changes compared to average Joe user's
expectations of downward compatibility with music, movie, and various document formats? If
so, we're talking about different contexts.
> Because off hand this sounds like a kind of sabotage to me.
No, it is just normal, everyday, software development.
> If it's throw away database info that can simply be reconstructed I'd
> probably tolerate it. But for that matter, such things should go in an
> defined cache location so that it's not even being backed up.
In some cases it is data that can be reconstructed, so all you need to
do is to manually blow away the files and let the app rebuild them. In
some cases that also have additional inconveniences. In some cases it is
not data you can or want to throw away.
> But important user data having it's format updated in a way that makes
> it incompatible with the previous minor version (same major version)?
So ?
It is only visible if you downgrade which a lot of software do not
support and explicitly so.
The right way to do file format changes is you design the new format. And in a minor
version update, the application gains the ability to read the new file format, but still
writes the old file format. The major version upgrade of the application is enabled to
write the new file format, while it can read either old or new formats.
> I'm snickering at the language that would ensue in the
proprietary
> software world, if I'm not totally confused about what it is you're
> suggested is fair game. It'd be the sort of language you wouldn't want
> your teenager or grandmother to hear.
??
If Adobe Photoshop version n.1.0 started to write out Photoshop documents in a manner that
n.0.0 could not read, 100% of users would call it a major bug, and it would escalate into
vicious name calling - not just of Adobe the company and its shareholders, but specific
individuals would get called out like an old Western gun fight. It would be nasty.
It's a scary and hilarious mental exercise to think of what might happen but also
ridiculous because of that. It simply wouldn't happen because of the holy hellfire
that would ensue. New swear words would be invented by end users.
About the closest thing I can think of was almost 10 years ago when Nikon
"encrypted" their white balance metadata in Raw files, instigated after a camera
firmware update. The language used by professional photographers started out with
"WTF this is bullshit" and escalated from there. If I repeated some of the
common phrases from respected professional photographer colleagues of mine on this list,
I'd probably get kicked off. The language was absolutely deserved and it was caustic.
Breaking downward compatibility in file formats for regular Joe user is courting public
relations disaster. It can kill a product. Even Microsoft doesn't do this lightly.
They've had now many file format changes of Word in how many decades? I can count the
format changes on one hand. It's bad enough to make a file format proprietary to a
particular application, but for an application to effectively make the file format
proprietary to older versions of itself? Yeah, grossly incompetent from my perspective.
Such things must be planned well in advance and made as painless as possible on the user.
Chris Murphy
1:51 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 26.01.2014 20:45, schrieb Chris Murphy:
> So ?
> It is only visible if you downgrade which a lot of software do not
> support and explicitly so
The right way to do file format changes is you design the new format.
And in a minor version update, the application gains the ability to
read the new file format, but still writes the old file format.
The major version upgrade of the application is enabled to write the
new file format, while it can read either old or new formats.
please look at the hidden folders in your userhome and /var/lib/
to get a picture about what we are talking here
If Adobe Photoshop version n.1.0 started to write out Photoshop
documents
in a manner that n.0.0 could not read, 100% of users would call it a major bug,
and it would escalate into vicious name calling
nobody but you is talking about documents the user really visualizes
Breaking downward compatibility in file formats for regular Joe user
is courting
public relations disaster. It can kill a product. Even Microsoft doesn't do this
lightly
nobody but you is talking about documents the user really visualizes
1:59 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 26.01.2014 20:51, schrieb Reindl Harald:
Am 26.01.2014 20:45, schrieb Chris Murphy:
>> So ?
>> It is only visible if you downgrade which a lot of software do not
>> support and explicitly so
>
> The right way to do file format changes is you design the new format.
> And in a minor version update, the application gains the ability to
> read the new file format, but still writes the old file format.
> The major version upgrade of the application is enabled to write the
> new file format, while it can read either old or new formats.
please look at the hidden folders in your userhome and /var/lib/
to get a picture about what we are talking here
> If Adobe Photoshop version n.1.0 started to write out Photoshop documents
> in a manner that n.0.0 could not read, 100% of users would call it a major bug,
> and it would escalate into vicious name calling
nobody but you is talking about documents the user really visualizes
> Breaking downward compatibility in file formats for regular Joe user is courting
> public relations disaster. It can kill a product. Even Microsoft doesn't do this
lightly
nobody but you is talking about documents the user really visualizes
you may also read http://en.wikipedia.org/wiki/SQLite
* sqlite is used by a lot of software to store data
* a new feature in a new version may change the scheme
* you do not need care about that
* the new software version recognizes the old format and applies changes
* the old software version may ignore new tables and columns
* it continues to ignore them and don't donwgrade the version info stored somewhere
* the new version after the next update expects consistent data in the new format
* your downgrade and still work with the sqlite database may lead to regret doing so
months later
again: *nobody* is talking about documents
and the same which is done with sqlite may be *whatever* format of store internal data
but is affected by the same problematic - *nobody* really supports downgrades
they *may* work fine, there *maybe* are no changes
nobody is telling you so - why? - because what you are doing is not supported
so *please* stop talking about document formats in *that* thread
2:30 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Jan 26, 2014, at 12:51 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
Am 26.01.2014 20:45, schrieb Chris Murphy:
>> So ?
>> It is only visible if you downgrade which a lot of software do not
>> support and explicitly so
>
> The right way to do file format changes is you design the new format.
> And in a minor version update, the application gains the ability to
> read the new file format, but still writes the old file format.
> The major version upgrade of the application is enabled to write the
> new file format, while it can read either old or new formats.
please look at the hidden folders in your userhome and /var/lib/
to get a picture about what we are talking here
This sounds like FUD and there's no actual real world example. You're suggesting
that if I have gnome-shell 3.10.3 and I either yum downgrade to 3.10.1, or I do a clean
install of Fedora 20 and get gnome-shell 3.10.0, that I'm going to see explosions?
What's going to happen? Surely there aren't such significant configuration format
changes between such minor versions, and surely the development teams anticipate this very
use case where uses have a /home with such files, and have no choice but to revert to an
older system with the same major version but lower minor version.
This is why changing configuration formats is hopefully a conscientious decision and not
done willy nilly. From many years of experience I know I can reliably upgrade and
downgrade at will, within minor versions of OS X - that is all versions of 10.7.x
configuration file wise are expected to be compatible. And I'd exclude disposable
cache files which by default aren't even backed up anyway as they're expected to
be rebuilt on a restore.
Chris Murphy
2:40 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 26.01.2014 21:30, schrieb Chris Murphy:
On Jan 26, 2014, at 12:51 PM, Reindl Harald
<h.reindl(a)thelounge.net> wrote:
> Am 26.01.2014 20:45, schrieb Chris Murphy:
>>> So ?
>>> It is only visible if you downgrade which a lot of software do not
>>> support and explicitly so
>>
>> The right way to do file format changes is you design the new format.
>> And in a minor version update, the application gains the ability to
>> read the new file format, but still writes the old file format.
>> The major version upgrade of the application is enabled to write the
>> new file format, while it can read either old or new formats.
>
> please look at the hidden folders in your userhome and /var/lib/
> to get a picture about what we are talking here
This sounds like FUD and there's no actual real world example
* i do not know what *may happen* by restore a snapshot
* you do not know what *may* happen by restore a snapshot
* nobody knows
why?
because nobody *can* know what exactly packages, versions are installed
in which combination or which *user specific* data may exist on exactly
the FS which is restored *additionally* to what the system sofware knows
frankly you can have your kwallet or the files your browser stores
passwords you recently created and thought they are safe on exactly
that FS, and they *maybe* saved *between* upgrade, realize a problem
and restore the snapshot
again: *nobody* knows for sure the *complete possible impact* on the
users computer by restore a snapshot because a upgrade should be
rolled back
surely, you can do that, i and many other people won't do this now
nor in the future for good reasons and not knowing *exactly* any
possible impact of a operation is a *damned good* reason
nothing more to say about that topic because
* i *never* won't do that
* i *never* would use LVM
* i *never* use BTRFS
* so my environment even does not support that snapshots
why i won#t use BTRFS/LVM?
because my drives are a Linux RAID10 and i never re-installed my system from
scratch nor would i do that in the future and *because* not everybody is using
a storage even supports snapshots it would be a bad idea to rely on that
5:41 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Jan 26, 2014, at 1:40 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
Am 26.01.2014 21:30, schrieb Chris Murphy:
> On Jan 26, 2014, at 12:51 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
>> Am 26.01.2014 20:45, schrieb Chris Murphy:
>>>> So ?
>>>> It is only visible if you downgrade which a lot of software do not
>>>> support and explicitly so
>>>
>>> The right way to do file format changes is you design the new format.
>>> And in a minor version update, the application gains the ability to
>>> read the new file format, but still writes the old file format.
>>> The major version upgrade of the application is enabled to write the
>>> new file format, while it can read either old or new formats.
>>
>> please look at the hidden folders in your userhome and /var/lib/
>> to get a picture about what we are talking here
>
> This sounds like FUD and there's no actual real world example
* i do not know what *may happen* by restore a snapshot
* you do not know what *may* happen by restore a snapshot
* nobody knows
Great, well I'll tell you what. I will just keep living dangerously, and when I find a
real world case of this, I'll file a bug. How about that?
because nobody *can* know what exactly packages, versions are
installed
in which combination or which *user specific* data may exist on exactly
the FS which is restored *additionally* to what the system sofware knows
frankly you can have your kwallet or the files your browser stores
passwords you recently created and thought they are safe on exactly
that FS, and they *maybe* saved *between* upgrade, realize a problem
and restore the snapshot
Oh for f's sake. And *maybe* the next time I take a shower I'll slip and fall, so
I'll just decide now to not bathe ever again. This hysterical paranoia you're
going on about is even less hypothetical than slipping in the bathroom. I read this
buttkiss nonsense and feel like someone has injected my brain with novocaine.
again: *nobody* knows for sure the *complete possible impact* on the
users computer by restore a snapshot because a upgrade should be
rolled back.
Well you know what I think, is that applications should largely be self contained instead
of sneezing all kinds of crap all over my file system. It's one of the best examples
of why I prefer using OS X compared to Windows, which are drag and drop installation of
applications that don't install weird junk all over my computer. Very very easy to
rollback from this.
surely, you can do that, i and many other people won't do this now
nor in the future for good reasons and not knowing *exactly* any
possible impact of a operation is a *damned good* reason
nothing more to say about that topic because
* i *never* won't do that
* i *never* would use LVM
* i *never* use BTRFS
* so my environment even does not support that snapshots
Uh huh. So this is sort like a user coming onto this forum and talking trash about all of
linux and Fedora and what all is broken and doesn't fit their use case or workflow at
all, and then after 50 f'n emails they say they never use linux or Fedora. Even for
you this is an especially egregious waste of time and brain cells. I can even feel the rot
occurring in my brain from reading this mindnumbing nonsense you've written in this
whole thread, and the icing on the cake is that you don't even use the technologies
you're bitching about. Bitch, bitch, bitch. That's the only thing you've
accomplished. You're just bitching. It's f'n annoying.
why i won#t use BTRFS/LVM?
because my drives are a Linux RAID10 and i never re-installed my system from
scratch nor would i do that in the future and *because* not everybody is using
a storage even supports snapshots it would be a bad idea to rely on that
I think you having access to a computer with internet access is a bad idea.
Chris Murphy
5:47 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 27.01.2014 00:41, schrieb Chris Murphy:
Great, well I'll tell you what. I will just keep living
dangerously, and when I find a real world case of this, I'll file a bug. How about
that?
do that, your problem
> because nobody *can* know what exactly packages, versions are
installed
> in which combination or which *user specific* data may exist on exactly
> the FS which is restored *additionally* to what the system sofware knows
>
> frankly you can have your kwallet or the files your browser stores
> passwords you recently created and thought they are safe on exactly
> that FS, and they *maybe* saved *between* upgrade, realize a problem
> and restore the snapshot
Oh for f's sake. And *maybe* the next time I take a shower I'll slip and fall
off-topic
you do not understand anything this theard is about so why not leaves us in peace?
It's one of the best examples of why I prefer using OS X compared
to Windows
then use it
> nothing more to say about that topic because
>
> * i *never* won't do that
> * i *never* would use LVM
> * i *never* use BTRFS
> * so my environment even does not support that snapshots
Uh huh. So this is sort like a user coming onto this forum and talking trash
about all of linux and Fedora and what all is broken and doesn't fit their use
case or workflow at all, and then after 50 f'n emails they say they never use
linux or Fedora
you do not understand the intention of that thread at all
so why you don't just listen and be quite?
I think you having access to a computer with internet access is a bad
idea
must be why i get paid for server-adminstration and security for a decade...
please bother the users-lists where i am no longer subscribed because people
like you leading to get upset again and again
6:07 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Jan 26, 2014, at 4:47 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
Am 27.01.2014 00:41, schrieb Chris Murphy:
> Great, well I'll tell you what. I will just keep living dangerously, and when I
find a real world case of this, I'll file a bug. How about that?
do that, your problem
>> because nobody *can* know what exactly packages, versions are installed
>> in which combination or which *user specific* data may exist on exactly
>> the FS which is restored *additionally* to what the system sofware knows
>>
>> frankly you can have your kwallet or the files your browser stores
>> passwords you recently created and thought they are safe on exactly
>> that FS, and they *maybe* saved *between* upgrade, realize a problem
>> and restore the snapshot
>
> Oh for f's sake. And *maybe* the next time I take a shower I'll slip and
fall
off-topic
you do not understand anything this theard is about so why not leaves us in peace?
I'll add rampant hyperbole to your list of personal attributes. I'm the one who
doesn't understand anything even though you don't use or have any use for
snapshots, and you also want no one to have them or it'll make developers careless:
On Jan 25, 2014, at 6:10 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
the short version of ahwat you said could have been "forget
snapshots at all to solve
such problems" to not lead dvelopers into temptation of "i can be less caeful
because
we have snapshots"
in other words: don't work around problems by create new ones
And then you propose a ridonkulous snapshot-rollback strategy that would for certain cause
major problems if the rollback were actually done, and then use that as fait accompli for
why the entire concept of fs rollbacks are stupid. Your arguments are asinine. Your emails
belong in a kill file.
Chris Murphy
6:10 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
Am 27.01.2014 01:07, schrieb Chris Murphy:
And then you propose a ridonkulous snapshot-rollback strategy that
would for certain cause major problems
if the rollback were actually done
*the opposite is true because i WARNED of doing snapshots*
6:30 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Jan 26, 2014, at 5:10 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
Am 27.01.2014 01:07, schrieb Chris Murphy:
> And then you propose a ridonkulous snapshot-rollback strategy that would for certain
cause major problems
> if the rollback were actually done
*the opposite is true because i WARNED of doing snapshots*
Yes, you argued against the general case of snapshot-rollbacks while using bad examples of
rollback methods that would obviously cause problems.
Chris Murphy
2:50 p.m.
New subject: Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]
On Sun, 2014-01-26 at 12:45 -0700, Chris Murphy wrote:
I still really have no idea what sorts of changes you're talking
about.
I think you made it abundantly clear :-)
I am also sure what I wanted to convey to people that understand what I
am talking about is also clear. So I think the matter has been expressed
well enough for devel and I do not think we need to further pollute the
list.
Thank you,
Simo.
--
Simo Sorce * Red Hat, Inc * New York
5:42 p.m.
On Jan 25, 2014, at 9:46 AM, Tomasz Torcz <tomek(a)pipebreaker.pl> wrote:
On Fri, Jan 24, 2014 at 03:10:04PM -0700, Chris Murphy wrote:
>
> Another possible case it's /etc/ where the either a package or the user could
> make changes during the update. Btrfs allows per file snapshots with cp
> --reflink so there might be a way to carve the snapshot with a scalpel but I
> prefer doing it with subvolume granularity. Plus that granularity translates to
> LVM.
Note that this situation is perfectly handled by Offline Updates.
After reboot, there aren't collateral changes to filesystem, only upgrade-related
ones. So if there's a need for revert, the previous state is clearly defined.
I don't follow this. The realization an update is bad doesn't necessarily occur
right away. So we still need a way to separate system domain vs user domain, at least, so
that system files are rolled back separately from user files.
Chris Murphy
3:38 p.m.
On Jan 24, 2014, at 2:58 AM, Sergio Pascual <sergio.pasra(a)gmail.com> wrote:
2014/1/24 Ralf Corsepius <rc040203(a)freenet.de>
Certainly, downgrading installations which already upgraded to faulty packages would not
work.
Ralf
The situation (a broken system that cannot be upgraded) could be mitigated a little bit
by using yum + system snapshots. You can rollback to a previous sane system.
This is non-trivial for the typical user. And as far as I'm aware there's no
storage SIG to define a best practices, so at the moment there are 19 recipes per snapshot
technology. So one person can explain how they do it, then the user gets into some trouble
with a question and no other user can really answer the question because they do it a
different way.
There is a plugin yum-plugin-fs-snapshot, but it requires better documentation and system
integration.
Well I'd go a step further and ask some more basic questions how how many snapshots
should be bootable, whether systemd-journal should be persistent across snapshots or
snapshot specific, what exactly are we snapshotting, can we require /home be separate
(presently we don't require it) in order to support such bootable snapshots, on and
on.
I'm not so sure the plugin needs an update or replacing or a separate user space
program that helps manage the moving parts: the snapshot creation, the update, altering
fstab and grub.cfg as needed, and even what the bootable snapshot options should look like
and where: we have three kernels to choose from in GRUB, as soon as there's one
snapshot, we might have three identical kernel versions each with two sysroots; or
possibly there are four kernels, one only boots the old sysroot, one only boots the new
sysroot and two could boot either sysroot. And that's just with one snapshot, as soon
as there are accumulating snapshots to boot, I start looking for blood because I don't
have enough going to my brain as it is.
Currently (I don't know how current is F16 documentation) it requires running lvm by
hand
http://docs.fedoraproject.org/en-US/Fedora/16/html/System_Administrators_...
Right and there's a legitimate question how useful conventional LVM snapshots are,
just because at one time they were the only thing we had, but they're slow and
inefficient and you wouldn't want to use them for very long as that's not what
they were designed for. Whereas LVM thinp snapshots are completely different, useable,
accumulatble over the long haul, do not require preallocation, very much like Btrfs
snapshots but of course a different implementation. And then Btrfs snapshots are dead nuts
simple to create and remove compared to thinp - *and* they are directly grub2 bootable
unlike LVM thinp.
Chris Murphy
5:16 p.m.
On 01/24/2014 01:38 PM, Chris Murphy wrote:
On Jan 24, 2014, at 2:58 AM, Sergio Pascual
<sergio.pasra(a)gmail.com
> There is a plugin yum-plugin-fs-snapshot, but it requires better
> documentation and system integration.
Well I'd go a step further and ask some more basic questions how how
many snapshots should be bootable, whether systemd-journal should be
persistent across snapshots or snapshot specific, what exactly are we
snapshotting, can we require /home be separate (presently we don't
require it) in order to support such bootable snapshots, on and on.
I'd also ask where we keep these snapshots, and how do you prevent
access to them normally. IIRC, yum-plugin-fs-snapshot makes btrfs
snapshots as a subvolume directly within the filesystem, which means it
will still be accessible.
This concerns me especially in the case of security updates -- for
example, a vulnerable setuid-root binary should be locked up tight!
Josh
7:27 p.m.
On Jan 24, 2014, at 4:16 PM, Josh Stone <jistone(a)redhat.com> wrote:
On 01/24/2014 01:38 PM, Chris Murphy wrote:
> On Jan 24, 2014, at 2:58 AM, Sergio Pascual <sergio.pasra(a)gmail.com
>> There is a plugin yum-plugin-fs-snapshot, but it requires better
>> documentation and system integration.
>
> Well I'd go a step further and ask some more basic questions how how
> many snapshots should be bootable, whether systemd-journal should be
> persistent across snapshots or snapshot specific, what exactly are we
> snapshotting, can we require /home be separate (presently we don't
> require it) in order to support such bootable snapshots, on and on.
I'd also ask where we keep these snapshots, and how do you prevent
access to them normally. IIRC, yum-plugin-fs-snapshot makes btrfs
snapshots as a subvolume directly within the filesystem, which means it
will still be accessible.
It's possible the utility could mount another subvolume not in the present path,
including top level ID 5 (the first and default subvolume by mkfs.btrfs), and place
snapshots there, and then unmount.
For LVM thinp snapshots, they become completely out of tree LVs. They aren't
accessible unless explicitly mounted.
This concerns me especially in the case of security updates -- for
example, a vulnerable setuid-root binary should be locked up tight!
The organization question is valid. But sudo or root could just mount any subvolume.
However, btrfs read-only snapshots can't be written to even by root. Naturally root
could just create a rw snapshot of a ro snapshot and then delete the ro snapshot, but an
audit probably ought to show the subvolume UUIDs and creation dates involved so that
we'd know this is what happened.
Chris Murphy
10:40 p.m.
On 01/24/2014 05:27 PM, Chris Murphy wrote:
On Jan 24, 2014, at 4:16 PM, Josh Stone <jistone(a)redhat.com>
wrote:
> This concerns me especially in the case of security updates -- for
> example, a vulnerable setuid-root binary should be locked up tight!
The organization question is valid. But sudo or root could just mount
any subvolume. However, btrfs read-only snapshots can't be written to
even by root. Naturally root could just create a rw snapshot of a ro
snapshot and then delete the ro snapshot, but an audit probably ought
to show the subvolume UUIDs and creation dates involved so that we'd
know this is what happened.
My point was not about what root can do. Suppose there's a vulnerable
'sudo' binary that gives everyone a root shell. If that binary is
available on any executable path, even readonly, that's trouble.
As you say, LVM snapshots are out of view, but with btrfs it needs to be
an inaccessible subvolume path, or mounted noexec, etc.
8:03 a.m.
On Fri, Jan 24, 2014 at 20:40:28 -0800,
Josh Stone <jistone(a)redhat.com> wrote:
My point was not about what root can do. Suppose there's a vulnerable
'sudo' binary that gives everyone a root shell. If that binary is
available on any executable path, even readonly, that's trouble.
That isn't true. File systems can be mounted such that suid bits are
ignored. suid executables on such file systems are effectively just
normal executables.
12:37 p.m.
On 01/25/2014 06:03 AM, Bruno Wolff III wrote:
On Fri, Jan 24, 2014 at 20:40:28 -0800,
Josh Stone <jistone(a)redhat.com> wrote:
>
> My point was not about what root can do. Suppose there's a vulnerable
> 'sudo' binary that gives everyone a root shell. If that binary is
> available on any executable path, even readonly, that's trouble.
That isn't true. File systems can be mounted such that suid bits are
ignored. suid executables on such file systems are effectively just
normal executables.
Ok, sure, you can mount -o nosuid,noexec,nodev ... but this isn't the
default for btrfs subvolume paths AFAIK. It needs to be a conscious
decision in whatever snapshot design we choose.
1:32 p.m.
On Sat, 2014-01-25 at 10:37 -0800, Josh Stone wrote:
Ok, sure, you can mount -o nosuid,noexec,nodev ... but this isn't
the
default for btrfs subvolume paths AFAIK. It needs to be a conscious
decision in whatever snapshot design we choose.
This is definitely an issue with the OSTree design, since everything
shares a physical partition (you can choose whatever block storage you
want) - it's just hard links.
I just filed:
https://bugzilla.gnome.org/show_bug.cgi?id=722984
for this.
But really, now that KDBus is on the way, we can start using it for
system services to replace many setuid binaries, like unix_chkpwd
without losing the auditing trail and such that old indirection via
dbus-daemon required. That's a subject for a different thread though.
2:05 p.m.
On Sat, 2014-01-25 at 14:32 -0500, Colin Walters wrote:
On Sat, 2014-01-25 at 10:37 -0800, Josh Stone wrote:
> Ok, sure, you can mount -o nosuid,noexec,nodev ... but this isn't the
> default for btrfs subvolume paths AFAIK. It needs to be a conscious
> decision in whatever snapshot design we choose.
This is definitely an issue with the OSTree design, since everything
shares a physical partition (you can choose whatever block storage you
want) - it's just hard links.
I just filed:
https://bugzilla.gnome.org/show_bug.cgi?id=722984
for this.
I forgot by gnome bugzilla password (again) so before I forget:
do not use .files or such it quickly becomes a mess. If you need to
annotate this kind of things I humbly suggest you add an xattr to the
file namespaced to ostree.
Alternatively, if you do not want to touch the original file at all,
keep a separate database where you note all these things, it will make
for a faster lookup in case you need bulk operations instead of having
to troll the whole tree.
But really, now that KDBus is on the way, we can start using it for
system services to replace many setuid binaries, like unix_chkpwd
without losing the auditing trail and such that old indirection via
dbus-daemon required. That's a subject for a different thread though.
This is a good point, but a number of binaries are that way for legacy
reasons, or come from upstreams that care for portability and can't rely
on dbus (yet), so I think you need to care for the problem anyway.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
2:15 p.m.
On Jan 24, 2014, at 9:40 PM, Josh Stone <jistone(a)redhat.com> wrote:
On 01/24/2014 05:27 PM, Chris Murphy wrote:
> On Jan 24, 2014, at 4:16 PM, Josh Stone <jistone(a)redhat.com> wrote:
>> This concerns me especially in the case of security updates -- for
>> example, a vulnerable setuid-root binary should be locked up tight!
>
> The organization question is valid. But sudo or root could just mount
> any subvolume. However, btrfs read-only snapshots can't be written to
> even by root. Naturally root could just create a rw snapshot of a ro
> snapshot and then delete the ro snapshot, but an audit probably ought
> to show the subvolume UUIDs and creation dates involved so that we'd
> know this is what happened.
My point was not about what root can do. Suppose there's a vulnerable
'sudo' binary that gives everyone a root shell. If that binary is
available on any executable path, even readonly, that's trouble.
OK, so is the fact it's persistently available the problem? Because if I were to have
a persistent backup of sysroot mounted, I've got the same attack vector available. By
default for even an unprivileged user gnome-shell mounts with By default, gnome-shell
mounts volumes with rw,nosuid,nodev,relatime,seclabel,uhelper=udisks2.
As you say, LVM snapshots are out of view, but with btrfs it needs to be
an inaccessible subvolume path, or mounted noexec, etc.
To make inaccessible: mount a subvol outside of the presently mounted path, snapshot,
umount.
Seems like I can independently mount subvolumes with noexec:
49 37 0:45 /isos /mnt/isos rw,relatime shared:35 - btrfs /dev/sdb
rw,seclabel,compress=lzo,space_cache
177 37 0:45 /archive /mnt/root rw,noexec,relatime shared:159 - btrfs /dev/sdb
rw,seclabel,compress=lzo,space_cache
So another possibility is to have a "snapshots" subvolume persistently mounted,
with noexec, and always place snapshots in that subvolume.
Chris Murphy
1:55 p.m.
On 01/25/2014 12:15 PM, Chris Murphy wrote:
OK, so is the fact it's persistently available the problem?
Because
if I were to have a persistent backup of sysroot mounted, I've got
the same attack vector available. By default for even an unprivileged
user gnome-shell mounts with By default, gnome-shell mounts volumes
with rw,nosuid,nodev,relatime,seclabel,uhelper=udisks2.
Right, it's having a persistent and usable copy of a vulnerability.
So another possibility is to have a "snapshots" subvolume
persistently mounted, with noexec, and always place snapshots in that
subvolume.
That sounds good -- even might be just nosuid on that.
Josh
10:45 a.m.
Sergio Pascual wrote:
The situation (a broken system that cannot be upgraded) could be
mitigated a little bit by using yum + system snapshots. You can rollback
to a previous sane system.
The big problem with that approach (other than the granularity issue already
pointed out) is disk space. Even with a smart snapshotting technology that
really only keeps on disk exactly what changed, it still requires a lot of
extra disk space.
Kevin Kofler
6:36 a.m.
Adam Williamson wrote:
TBH this has always been the one of Kevin's Big Book Of Update
Policy
Complaints I find the most baffling. If we know you managed to screw up
your update once, why exactly would we just trust you to get it right
the *second* time without any testing?
* If the package is already so screwed that it breaks the whole system, it
cannot realistically get any worse.
* A regression fix is usually a trivial change, often reverting something to
a previous, already well-tested, state.
Kevin Kofler
11:42 a.m.
On Fri, 2014-01-24 at 13:36 +0100, Kevin Kofler wrote:
Adam Williamson wrote:
> TBH this has always been the one of Kevin's Big Book Of Update Policy
> Complaints I find the most baffling. If we know you managed to screw up
> your update once, why exactly would we just trust you to get it right
> the *second* time without any testing?
* If the package is already so screwed that it breaks the whole system, it
cannot realistically get any worse.
Sure it can. It can wipe all your data, or mail it to the NSA...
"Breaks the whole system" is high on the Pantscon Scale, sure, but it's
not the highest. Data loss and security compromise both come higher.
* A regression fix is usually a trivial change, often reverting
something to
a previous, already well-tested, state.
Sure. And what could possibly go wrong.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
10:56 a.m.
Adam Williamson wrote:
On Fri, 2014-01-24 at 13:36 +0100, Kevin Kofler wrote:
> * If the package is already so screwed that it breaks the whole system,
> it cannot realistically get any worse.
Sure it can. It can wipe all your data, or mail it to the NSA...
That's why I said "realistically". What is the probability of something like
that happening IN PRACTICE from a trivial change, especially one that
reverts something to a known previous state? Perfect QA does not exist
anyway, so it is all a matter of probabilities.
"Breaks the whole system" is high on the Pantscon Scale,
sure, but it's
not the highest. Data loss and security compromise both come higher.
And you seriously think a week (or 2) of testing can catch that? Security
vulnerabilities can take years to get noticed, creeping data loss days to
weeks. The only thing in your catastrophe scale that can be noticed in 1-2
weeks is blatant "wipes entire directories immediately" data loss, something
extremely unlikely to happen from a regression fix (but so are the other
catastrophic scenarios, unless the problem was already there before the
regression fix and is thus no reason to withhold the fix).
> * A regression fix is usually a trivial change, often reverting
something
> to a previous, already well-tested, state.
Sure. And what could possibly go wrong.
The risk of a catastrophe as you described happening needs to be estimated
(a tiny fraction of a percent) and weighed against the breakage (and denial
of service, if security terms are the only ones you understand!) of keeping
the broken update in stable for longer (and thus also letting it affect more
users). I think it is blatantly obvious that the first theoretical risk is
the much better one to take compared to the second, very practical one.
Kevin Kofler
4:54 a.m.
On Fri, Jan 24, 2014 at 12:55 AM, Kevin Kofler <kevin.kofler(a)chello.at> wrote:
So, what happened:
* We are enabling SELinux enabled (enforcing) by default, a tool designed to
prevent anything it does not like from happening. (Reread this carefully:
The ONLY thing that tool is designed to do at all is PREVENT things. It does
not have a SINGLE feature other than being a roadblock and an annoyance.)
The "feature" is called security. By your logic everyone should be
root, we should
disable other security features like ASLR and NX (both PREVENT me from running
malicious code but do not add a SINGLE feature).
So please read on how security is implemented and why.
* SELinux works by shipping a "policy" that effectively
tries to specify in
one single place (read: single point of failure!) everything any program in
Fedora (scalability disaster!) ever wants to do (second-guessing its actual
code, i.e., duplication of all logic!).
That's not how it works not how it supposed to work. Please read on MAC.
(Note the 3 (!) major antipatterns
in a single-sentence (!) description of how SELinux works!)
Not a description on how it works but your misunderstand.
* An update to that SELinux policy was shipped that BREAKS the most
critical
tools in Fedora, the ones required to update the system and thus install the
fixes for any regressions, including the very regression that caused the
breakage. And also any automated workarounds are blocked by design.
No idea what "automated workaround" means but there are other ways to
deal with it see Colin's post.
* That update made it out to the stable updates! In other words, the
draconian Update Policies that were enacted in a vain attempt to prevent
such issues from happening utterly failed at catching this bug.
Yeah so we should find out why this happened and improve the testing
procedures to not let it happen in the feature (again see Colin's mail).
So, what needs to happen:
* SELinux must be disabled (or preferably, not installed in the first place,
to avoid wasting space for nothing) by default! Just consider the benefits
(none!)
As stated above that's not true.
* The Update Policies must be repealed. This regression has shown us
that
not only they totally failed at preventing it, but they are actively
contributing to exposing MORE users to broken updates by delaying regression
fixes. (This kind of regression fixes needs to go out DIRECTLY to stable!)
This is a contradiction "our current testing didn't find the bug so
how about we do no testing at all".
6:56 a.m.
drago01 wrote:
The "feature" is called security. By your logic everyone
should be
root,
For home user machines, that wouldn't necessarily be a bad thing (but it
would mean fixing the software that special-cases the root user improperly
for no good reason).
Alternatively, the kernel could be patched to give "admin users" (either
defined as members of the "wheel" group as now, or by some additional
property that would be set for the same users by default) some strategic
capabilities such as dac_override. That would also put an end to the endless
annoyance of having to sudo all the time. (And by the way, sudo and
PolicyKit actions should be allowed with no password (rather than the user
password as now) for wheel group members by default.) That way, you still
get the benefits from different accounts, e.g., different preferences per
family member, without the current restrictions imposed to "normal" users.
The endless password prompts make a lot of sense in controlled corporate
environments with dedicated system administrators, but on home machines,
they are just an unnecessary annoyance.
> * SELinux works by shipping a "policy" that effectively
tries to specify
> in one single place (read: single point of failure!) everything any
> program in Fedora (scalability disaster!) ever wants to do
> (second-guessing its actual code, i.e., duplication of all logic!).
That's not how it works not how it supposed to work. Please read on MAC.
Uh, I know how it works. The above is how I summarize it. If you think that
is incorrect, please explain HOW.
> * An update to that SELinux policy was shipped that BREAKS the
most
> critical tools in Fedora, the ones required to update the system and thus
> install the fixes for any regressions, including the very regression that
> caused the breakage. And also any automated workarounds are blocked by
> design.
No idea what "automated workaround" means but there are other ways to
deal with it see Colin's post.
A %pretrans scriptlet that fixes the problem without manual user
intervention (other than OKing the update). But SELinux won't allow RPMs to
mess with it that way (especially without invoking an external executable,
which is blocked by the faulty policy) because it would defeat its flawed
security model.
Yeah so we should find out why this happened and improve the testing
procedures to not let it happen in the feature (again see Colin's mail).
NO amount of testing is going to prevent regressions from happening
occasionally. This means:
* we need to eliminate common sources of regressions such as SELinux, to
prevent whole classes of regressions from occurring in the first place
(prevention is better than duct tape!) and
* we have to accept that regressions can always happen and allow for fast
fixes to those regressions (direct stable pushes).
> So, what needs to happen:
> * SELinux must be disabled (or preferably, not installed in the first
> place, to avoid wasting space for nothing) by default! Just consider the
> benefits (none!)
As stated above that's not true.
As stated above, that IS true. :-)
> * The Update Policies must be repealed. This regression has shown
us that
> not only they totally failed at preventing it, but they are actively
> contributing to exposing MORE users to broken updates by delaying
> regression fixes. (This kind of regression fixes needs to go out DIRECTLY
> to stable!)
This is a contradiction "our current testing didn't find the bug so
how about we do no testing at all".
There is no contradiction. Doing away with policies that do not work is
perfectly logical, as is allowing quick regression fixes because regressions
do happen no matter how much you test.
Kevin Kofler
7:40 a.m.
Am 24.01.2014 13:56, schrieb Kevin Kofler:
Alternatively, the kernel could be patched to give "admin
users" (either
defined as members of the "wheel" group as now, or by some additional
property that would be set for the same users by default) some strategic
capabilities such as dac_override. That would also put an end to the endless
annoyance of having to sudo all the time. (And by the way, sudo and
PolicyKit actions should be allowed with no password (rather than the user
password as now) for wheel group members by default.) That way, you still
get the benefits from different accounts, e.g., different preferences per
family member, without the current restrictions imposed to "normal" users.
The endless password prompts make a lot of sense in controlled corporate
environments with dedicated system administrators, but on home machines,
they are just an unnecessary annoyance
no, they are not, they have the same reason as firefox asks
for the master-password before display stored passwords even
after you already entered it to login somewhere
they prevent that if you are not alone that while you go to
the toilet and forget to lock your screen unauthorized people
not doing things nobody wants on the machine
what you propose is the Apple way - not on a linux system please
8:25 a.m.
On Fri, 2014-01-24 at 14:40 +0100, Reindl Harald wrote:
Am 24.01.2014 13:56, schrieb Kevin Kofler:
> Alternatively, the kernel could be patched to give "admin users" (either
> defined as members of the "wheel" group as now, or by some additional
> property that would be set for the same users by default) some strategic
> capabilities such as dac_override. That would also put an end to the endless
> annoyance of having to sudo all the time. (And by the way, sudo and
> PolicyKit actions should be allowed with no password (rather than the user
> password as now) for wheel group members by default.) That way, you still
> get the benefits from different accounts, e.g., different preferences per
> family member, without the current restrictions imposed to "normal"
users.
>
> The endless password prompts make a lot of sense in controlled corporate
> environments with dedicated system administrators, but on home machines,
> they are just an unnecessary annoyance
no, they are not, they have the same reason as firefox asks
for the master-password before display stored passwords even
after you already entered it to login somewhere
they prevent that if you are not alone that while you go to
the toilet and forget to lock your screen unauthorized people
not doing things nobody wants on the machine
Worse than that, they prevent automated attacks via very vulnerable
applications like browsers. [which of course in Kevin's world are never
run in a SELinux sandbox]
So you if you get some malware to jailbreak out of the browser sandbox
all it needs to do is "sudo pwnme" if there is no password request.
Of course you need to understand at least a smidget of security to avoid
proposing ludicrous 'defaults'.
what you propose is the Apple way - not on a linux system please
It is just 'the pwn me' way, nothing more, nothing less.
--
Simo Sorce * Red Hat, Inc * New York
12:12 p.m.
Am Freitag, den 24.01.2014, 00:55 +0100 schrieb Kevin Kofler:
it is time to analyze the fallout from the following catastrophic
Fedora 20
regression:
https://bugzilla.redhat.com/show_bug.cgi?id=1054350
"rpm scriptlets are exiting with status 127"
Hey,
can't we add a default boot entry which starts the system in permissive
mode?
- fabian
12:18 p.m.
On Fri, Jan 24, 2014 at 7:12 PM, Fabian Deutsch <fabian.deutsch(a)gmx.de> wrote:
Am Freitag, den 24.01.2014, 00:55 +0100 schrieb Kevin Kofler:
> it is time to analyze the fallout from the following catastrophic
> Fedora 20
> regression:
> https://bugzilla.redhat.com/show_bug.cgi?id=1054350
> "rpm scriptlets are exiting with status 127"
Hey,
can't we add a default boot entry which starts the system in permissive
mode?
How would that help? If a user knows enough about the issue to try it
he/she could just switch to permissive mode.
12:31 p.m.
Am 24.01.2014 19:18, schrieb drago01:
On Fri, Jan 24, 2014 at 7:12 PM, Fabian Deutsch
<fabian.deutsch(a)gmx.de> wrote:
> Am Freitag, den 24.01.2014, 00:55 +0100 schrieb Kevin Kofler:
>> it is time to analyze the fallout from the following catastrophic
>> Fedora 20
>> regression:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1054350
>> "rpm scriptlets are exiting with status 127"
>
> Hey,
>
> can't we add a default boot entry which starts the system in permissive
> mode?
How would that help? If a user knows enough about the issue to try it
he/she could just switch to permissive mode
in *that* case
in a case where a broken selinux update leads in not boot at all
i can not imagine what i would to besides boot with a CD/DVD/USB
12:35 p.m.
Am 24.01.2014 19:31, schrieb Reindl Harald:
Am 24.01.2014 19:18, schrieb drago01:
> On Fri, Jan 24, 2014 at 7:12 PM, Fabian Deutsch <fabian.deutsch(a)gmx.de> wrote:
>> Am Freitag, den 24.01.2014, 00:55 +0100 schrieb Kevin Kofler:
>>> it is time to analyze the fallout from the following catastrophic
>>> Fedora 20
>>> regression:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1054350
>>> "rpm scriptlets are exiting with status 127"
>>
>> Hey,
>>
>> can't we add a default boot entry which starts the system in permissive
>> mode?
>
> How would that help? If a user knows enough about the issue to try it
> he/she could just switch to permissive mode
in *that* case
in a case where a broken selinux update leads in not boot at all
i can not imagine what i would to besides boot with a CD/DVD/USB
to be clear - *i can* edit the boot-params and put selinux=0 there
the average user can't but he may remember "uhm something with selinux
was one of the last updates" and try the however named option, keep
in mind some people own only one machine and can't google for help
1:22 p.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 01/24/2014 01:35 PM, Reindl Harald wrote:
Am 24.01.2014 19:31, schrieb Reindl Harald:
>
> Am 24.01.2014 19:18, schrieb drago01:
>> On Fri, Jan 24, 2014 at 7:12 PM, Fabian Deutsch <fabian.deutsch(a)gmx.de>
>> wrote:
>>> Am Freitag, den 24.01.2014, 00:55 +0100 schrieb Kevin Kofler:
>>>> it is time to analyze the fallout from the following catastrophic
>>>> Fedora 20 regression:
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1054350 "rpm scriptlets
>>>> are exiting with status 127"
>>>
>>> Hey,
>>>
>>> can't we add a default boot entry which starts the system in
>>> permissive mode?
>>
>> How would that help? If a user knows enough about the issue to try it
>> he/she could just switch to permissive mode
>
> in *that* case
>
> in a case where a broken selinux update leads in not boot at all i can
> not imagine what i would to besides boot with a CD/DVD/USB
to be clear - *i can* edit the boot-params and put selinux=0 there
the average user can't but he may remember "uhm something with selinux was
one of the last updates" and try the however named option, keep in mind
some people own only one machine and can't google for help
enforcing=0 in the kernel command line will boot the machine in permissive mode.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlLivWUACgkQrlYvE4MpobOCVwCg13o4l1+JL6MkQhbEOKfXqMBx
0YcAoNnetQTwZMslI7jP1To03WwC0twi
=86Sz
-----END PGP SIGNATURE-----
1:27 p.m.
Am 24.01.2014 20:22, schrieb Daniel J Walsh:
On 01/24/2014 01:35 PM, Reindl Harald wrote:
> Am 24.01.2014 19:31, schrieb Reindl Harald:
>>
>> Am 24.01.2014 19:18, schrieb drago01:
>>> On Fri, Jan 24, 2014 at 7:12 PM, Fabian Deutsch
<fabian.deutsch(a)gmx.de>
>>> wrote:
>>>> Am Freitag, den 24.01.2014, 00:55 +0100 schrieb Kevin Kofler:
>>>>> it is time to analyze the fallout from the following catastrophic
>>>>> Fedora 20 regression:
>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1054350 "rpm
scriptlets
>>>>> are exiting with status 127"
>>>>
>>>> Hey,
>>>>
>>>> can't we add a default boot entry which starts the system in
>>>> permissive mode?
>>>
>>> How would that help? If a user knows enough about the issue to try it
>>> he/she could just switch to permissive mode
>>
>> in *that* case
>>
>> in a case where a broken selinux update leads in not boot at all i can
>> not imagine what i would to besides boot with a CD/DVD/USB
> to be clear - *i can* edit the boot-params and put selinux=0 there
> the average user can't but he may remember "uhm something with selinux was
> one of the last updates" and try the however named option, keep in mind
> some people own only one machine and can't google for help
enforcing=0 in the kernel command line will boot the machine in permissive mode
please re-read what you have quoted and don't skip "average user" this time
the question was "can't we add a default boot entry which starts the system in
permissive mode?" and the first reply "If a user knows enough about the
issue"
2:13 p.m.
On Fri, Jan 24, 2014 at 7:35 PM, Reindl Harald <h.reindl(a)thelounge.net> wrote:
Am 24.01.2014 19:31, schrieb Reindl Harald:
>
> Am 24.01.2014 19:18, schrieb drago01:
>> On Fri, Jan 24, 2014 at 7:12 PM, Fabian Deutsch <fabian.deutsch(a)gmx.de>
wrote:
>>> Am Freitag, den 24.01.2014, 00:55 +0100 schrieb Kevin Kofler:
>>>> it is time to analyze the fallout from the following catastrophic
>>>> Fedora 20
>>>> regression:
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1054350
>>>> "rpm scriptlets are exiting with status 127"
>>>
>>> Hey,
>>>
>>> can't we add a default boot entry which starts the system in permissive
>>> mode?
>>
>> How would that help? If a user knows enough about the issue to try it
>> he/she could just switch to permissive mode
>
> in *that* case
>
> in a case where a broken selinux update leads in not boot at all
> i can not imagine what i would to besides boot with a CD/DVD/USB
to be clear - *i can* edit the boot-params and put selinux=0 there
the average user can't but he may remember "uhm something with selinux
was one of the last updates"
You are assuming that the "averange user" even knows what selinux is
or reviews the list of packages for every update.
I doubt either of them is true.
and try the however named option, keep
in mind some people own only one machine and can't google for help
I doubt that. Most people do have multiple ways to access the internet
(multiple computers, tablets, phones, game consoles ...) it is 2014
not 1996.
2:23 p.m.
Am 24.01.2014 21:13, schrieb drago01:
On Fri, Jan 24, 2014 at 7:35 PM, Reindl Harald
<h.reindl(a)thelounge.net> wrote:
> Am 24.01.2014 19:31, schrieb Reindl Harald:
>>
>> Am 24.01.2014 19:18, schrieb drago01:
>>> On Fri, Jan 24, 2014 at 7:12 PM, Fabian Deutsch <fabian.deutsch(a)gmx.de>
wrote:
>>>> Am Freitag, den 24.01.2014, 00:55 +0100 schrieb Kevin Kofler:
>>>>> it is time to analyze the fallout from the following catastrophic
>>>>> Fedora 20
>>>>> regression:
>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1054350
>>>>> "rpm scriptlets are exiting with status 127"
>>>>
>>>> Hey,
>>>>
>>>> can't we add a default boot entry which starts the system in
permissive
>>>> mode?
>>>
>>> How would that help? If a user knows enough about the issue to try it
>>> he/she could just switch to permissive mode
>>
>> in *that* case
>>
>> in a case where a broken selinux update leads in not boot at all
>> i can not imagine what i would to besides boot with a CD/DVD/USB
>
> to be clear - *i can* edit the boot-params and put selinux=0 there
>
> the average user can't but he may remember "uhm something with selinux
> was one of the last updates"
You are assuming that the "averange user" even knows what selinux is
or reviews the list of packages for every update.
I doubt either of them is true.
as i said often:
linux systems tend also to get way too closed
many things are hidden in the assumption "the user do not want to be disturbed
with this and that information and install as well as boot needs to be pretty
and shiny"
* rhgb
* quiet
* hidden grub-menu
hence, while you install Fedora there should be a (default enabled) checkbox
asking if you want to enable SELinux with a short description what it is
> and try the however named option, keep
> in mind some people own only one machine and can't google for help
I doubt that. Most people do have multiple ways to access the internet
(multiple computers, tablets, phones, game consoles ...) it is 2014
not 1996
technically yes
practically how much fun does somebody have to google on a smart-phone
for a solution while he is frustrated and angry - and even if - do not
assume that all users are living in your social structure, that is not
really true
for me it is no problem, on the other hand there is a guy on the CentOS
list with the thread "died again" seeking for a hardware problem and
stating he has no money to chnage his 10 or so years old computer while
you and i would have thrown out that crap by the next window weeks ago
9:13 p.m.
On 24 January 2014 20:13, drago01 <drago01(a)gmail.com> wrote:
> Am 24.01.2014 19:31, schrieb Reindl Harald:
>and try the however named option, keep
> in mind some people own only one machine and can't google for help
I doubt that. Most people do have multiple ways to access the internet
(multiple computers, tablets, phones, game consoles ...) it is 2014
not 1996.
Most well-off people in Europe and North America. Even then it's not
necessarily particularly convenient, and if net access on a machine
with a problem isn't available then you have further issues if files
are needed.
Beside, even given I have a laptop and a desktop machine, if I'm
running Fedora on both and an update breaks something in a way that
isn't immediately obvious then both could get hit. And looking up
technical instructions on a phone is pretty tedious.
--
imalone
http://ibmalone.blogspot.co.uk
12:37 p.m.
Am Freitag, den 24.01.2014, 19:18 +0100 schrieb drago01:
On Fri, Jan 24, 2014 at 7:12 PM, Fabian Deutsch
<fabian.deutsch(a)gmx.de> wrote:
> Am Freitag, den 24.01.2014, 00:55 +0100 schrieb Kevin Kofler:
>> it is time to analyze the fallout from the following catastrophic
>> Fedora 20
>> regression:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1054350
>> "rpm scriptlets are exiting with status 127"
>
> Hey,
>
> can't we add a default boot entry which starts the system in permissive
> mode?
How would that help? If a user knows enough about the issue to try it
he/she could just switch to permissive mode.
I mean, don't we have a general "save boot" / "emergency boot"
entry -
we could add enforcing=0 there.
- fabian
12:53 p.m.
On Fri, Jan 24, 2014 at 1:37 PM, Fabian Deutsch <fabian.deutsch(a)gmx.de> wrote:
> How would that help? If a user knows enough about the issue to
try it
> he/she could just switch to permissive mode.
I mean, don't we have a general "save boot" / "emergency boot"
entry -
we could add enforcing=0 there.
I like this idea. Then the solution would have been "reboot into
rescue mode and update your system".
Let's do this.
--
Konstantin Ryabitsev
LinuxFoundation.org
Montréal, Québec
1:26 p.m.
On Fri, Jan 24, 2014 at 01:53:10PM -0500, Konstantin Ryabitsev wrote:
> I mean, don't we have a general "save boot" /
"emergency boot" entry -
> we could add enforcing=0 there.
I like this idea. Then the solution would have been "reboot into
rescue mode and update your system".
Let's do this.
https://bugzilla.redhat.com/show_bug.cgi?id=1057768
--
Matthew Miller -- Fedora Project -- <mattdm(a)fedoraproject.org>
1:32 p.m.
Am Freitag, den 24.01.2014, 19:18 +0100 schrieb drago01:
On Fri, Jan 24, 2014 at 7:12 PM, Fabian Deutsch
<fabian.deutsch(a)gmx.de> wrote:
> Am Freitag, den 24.01.2014, 00:55 +0100 schrieb Kevin Kofler:
>> it is time to analyze the fallout from the following catastrophic
>> Fedora 20
>> regression:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1054350
>> "rpm scriptlets are exiting with status 127"
>
> Hey,
>
> can't we add a default boot entry which starts the system in permissive
> mode?
How would that help? If a user knows enough about the issue to try it
he/she could just switch to permissive mode.
Having the ability to revoke stable updates an a way to handle automatic
downgrades of revoked updates including a temporary switching SELinux to
permissive mode would IMHO be a better solution for the case a buggy
update went to stable and the system is still up and running. With this
way the user has nothing more to do than running a new update-check.
--
Regards,
Heiko Adams
12:26 p.m.
On Fri, 24 Jan 2014 00:55:23 +0100, Kevin Kofler wrote:
it is time to analyze the fallout from the following catastrophic
Fedora 20
regression:
https://bugzilla.redhat.com/show_bug.cgi?id=1054350
"rpm scriptlets are exiting with status 127"
* We are losing users to Ubuntu because of this issue.
Always hard to comment on without have seen numbers/statistics first.
IMO, there's some sort of "pork cycle" related to users switching
distributions forth and back after a couple of releases. Around Fedora 18
users of Ubuntu have returned. The current Anaconda is not everyone's
coup of tea, however, so that's one point where users are lost already.
* The bug now has 38 (!) duplicates in Bugzilla, plus many complaints
on
IRC, mailing lists, comments to other unrelated bugs (the fix for which
cannot be installed due to the SELinux bug) etc.
As I've searched for and closed several of the dupes, the various bug
reports are interesting. The users have misidentified the problem as
being a problem in the package/update they wanted to install. Some have
noticed many packages failing and have blamed Yum. I wonder how many
users are affected by the problem and have not done anything yet (since,
for example, they expect an update to fix it "magically").
* That update made it out to the stable updates! In other words, the
draconian Update Policies that were enacted in a vain attempt to prevent
such issues from happening utterly failed at catching this bug.
Those policies are not "draconian" enough [1]. On erroneous belief that
a +1 from three different testers would mean that the update has seen
enough testing, the test update has been published with the default karma
threshold of +3. The testers have failed. It's too simple for testers to
rush through the voting in bodhi without testing the updates
painstakingly. "The faster the better" has lead to a fatal mistake in
this case.
[1] It's up to the package maintainers to disable karma automatism or
to increase the threshold. AFAIK, the selinux maintainers are open to
doing exactly that.
1:14 p.m.
On Fri, 2014-01-24 at 19:26 +0100, Michael Schwendt wrote:
> * That update made it out to the stable updates! In other words,
the
> draconian Update Policies that were enacted in a vain attempt to prevent
> such issues from happening utterly failed at catching this bug.
Those policies are not "draconian" enough [1]. On erroneous belief that
a +1 from three different testers would mean that the update has seen
enough testing, the test update has been published with the default karma
threshold of +3. The testers have failed. It's too simple for testers to
rush through the voting in bodhi without testing the updates
painstakingly. "The faster the better" has lead to a fatal mistake in
this case.
I think that's being unnecessarily harsh on the testers. It's not at all
obvious to anyone that you ought to test update/install of another
package in order to validate an update to selinux-policy-targeted .
Hell, I don't do that.
Hate to sound like a broken record, but really the problem here is just
the complete lack of granularity in the karma system: to phrase it
theoretically, we know there are a huge spectrum of meanings for both +1
and -1:
+1
--
* I installed it and nothing blew up
* I installed it, rebooted and nothing blew up
* I installed it, ran the entire test suite, grabbed the source tarball
and inspected it line-by-line for vulnerabilities, fuzz tested all the
variable handling, then deployed it to my extensive test farm for a week
and assessed the results
* It fixes my bug, and I didn't test anything else
* It fixes my bug, and nothing blew up
* It fixes my bug, and...(you see where I'm going with this)
* It installs, it works, maybe it fixes some bugs, but it also
introduces this other regression
* I like the update text / the update submitter / candy
-1
--
* It failed to install
* I installed it, and something blew up
* I installed it, rebooted and something blew up
* (etc)
* It doesn't fix my bug (and that's the only bug the update was meant to
fix)
* It doesn't fix my bug (but the update also fixes 50 other bugs,
successfully)
* It doesn't fix this other bug I have that the update didn't even claim
to fix
* It installs, it works, maybe it fixes some bugs, but it also
introduces this other new bug (yes. this is identical to one of the +1
entries. That is the point. The same thing can also be registered as 0,
giving us the perfect set. Depending on the details of what's fixed and
what's broken, and the individual karma submitter's instincts, it can
seem 'right' to file this as any one of the three possible values.)
* It installs, it works, it doesn't exactly introduce any bugs, but I
think it is not compliant with the update policy (i.e. too drastic a
change in behaviour from the previous package)
* I don't like the update text / the update submitter / candy
The 'comment' field exists to allow people to express all these things,
but as it's just a completely free-form text field, it's intrinsically
impossible to really base any programmatic stuff or even policy on it.
In theory maintainers could submit updates without using autokarma and
then keep a careful eye on the feedback and 'tend' their updates
manually, but I think it's pretty clear that in practice, this is not
what happens: maintainers really want to be able to use the karma system
as a 'helper', they want to farm out the evaluation process to Bodhi/the
karma system. But our current system is too stupid to handle this
perfectly, so we get these breakdowns.
With a more flexible karma system we have a *lot* of opportunity to do
much cleverer stuff. We can provide presets for all the above different
things that are currently commonly expressed via +1 or -1 with a
comment. This opens up possibilities at two different levels: the distro
policy level, and the packager level. We can make the distro policy much
more fine-grained, if we want to - we can require certain of the 'karma
types' to be available in all updates, and for instance, block any
update where X people pull the 'it's completely busted' or 'it
introduces a security vulnerability' cord, regardless of how much
broadly-categorized 'positive' karma it has. At the packager level, the
packager gets the freedom to define a much more fine-grained policy for
when they're happy that updates to their package are 'good to go', but
they still don't have to sit there reading the emails and manually
interpreting what people have written. You get to define the policy that
makes the most sense for your package, within the confines of the
distro-wide policy - if you have a good package-specific test suite, you
can say to the auto-karma system 'don't send this update out until at
least one person sets the "I ran the test suite and it passed" karma
property.
Those are just examples: the point is that what we badly need here is a
more expressive and flexible system. (As well, as I've said elsewhere in
the discussion, as a good automated test for this specific and
well-known category of 'delayed action' update problems).
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
2:06 p.m.
On Fri, 2014-01-24 at 11:14 -0800, Adam Williamson wrote:
On Fri, 2014-01-24 at 19:26 +0100, Michael Schwendt wrote:
I think that's being unnecessarily harsh on the testers. It's not at all
obvious to anyone that you ought to test update/install of another
package in order to validate an update to selinux-policy-targeted .
Hell, I don't do that.
Agreed, The testers did not fail. Their issues were solved. They could
not have found this issue in reason. There was no change log entry for
it, and even if there was they would still would need to be able trace
the bug to SELinux.
The commit that caused this grief, was meant for rawhide but it ended up
in the wrong branch. (too many branches, too high volume fixes, Too much
rushing. Routine can be a killer)
Only the people that know this matter could have prevented it in reason,
anything else would have been just luck in my opinion.
In my view there are two issues that need to be addressed.
- Take your time, think twice before you commit and double check what
you commit. Do not let routine get the upper hand. Give all your changes
the attention they deserve.
- Coordinate with your team mates. Proof read each others commits (if
only skim over them)
If a team proof reads each others commits then this could have been
prevented in reason. Sure it is a structural investment but it pay's off
and its not as bad as it sounds. Five minutes daily?
2:38 p.m.
On Fri, 24 Jan 2014 21:06:29 +0100, Dominick Grift wrote:
Agreed, The testers did not fail. Their issues were solved.
That doesn't match what one can read here:
https://admin.fedoraproject.org/updates/FEDORA-2014-0806/selinux-policy-3...
They could not have found this issue in reason.
Why not? Please explain.
There was no change log entry for it,
You make it sound as if the testers have tried to skim over the several of
dozen bugzilla ticket descriptions linked at
https://admin.fedoraproject.org/updates/FEDORA-2014-0806/selinux-policy-3...
in an attempt at trying to find out _what_ the update touches.
A fundamental problem here is that even if a tester confirms that the
update fixes a _single_ bug, the other several dozens of changes could
cause regression -> reason to be careful and test this thing a bit longer.
and even if there was they would still would need to be able trace
the bug to SELinux.
That has been easy once the update arrived here on the nearby mirror.
"setenforce 0 && repeat previous command that caused strange behaviour"
is a very common troubleshooting thing, even if there haven't been any
AVC denied messages.
3:17 p.m.
On Fri, 2014-01-24 at 21:38 +0100, Michael Schwendt wrote:
On Fri, 24 Jan 2014 21:06:29 +0100, Dominick Grift wrote:
> Agreed, The testers did not fail. Their issues were solved.
That doesn't match what one can read here:
https://admin.fedoraproject.org/updates/FEDORA-2014-0806/selinux-policy-3...
I just had a quick look at the above URL. From all the testers there was
one guy that noticed the anomaly, and the biggest part of the events
weren't even related to the RPM issue
The RPM issue did not cause the normal AVC (type=AVC) denials (AFAIK)
that one would expect. Instead there were some SELINUX_ERR events
(type=SELINUX_ERR) that one might not notice if one is looking for AVC
(type=AVC) denials. (not sure if setroubleshoot would have reported
those)
The person that did notice the anomaly did some thorough testing, and
maybe there was also a little bit of luck involved there
> They could not have found this issue in reason.
Why not? Please explain.
Because you would need to run RPM to notice it, and then be able to
correlate the issue to SELinux. If you are waiting for a package that
has your fixes then you test your issues and give karma, it may take a
while before one actually runs yum again, and by then the update may
have been ended up in the repository.
And this is just in the case of RPM. There can be bugs in policy for
many components, but those are often not fatal.
> There was no change log entry for it,
You make it sound as if the testers have tried to skim over the several of
dozen bugzilla ticket descriptions linked at
https://admin.fedoraproject.org/updates/FEDORA-2014-0806/selinux-policy-3...
in an attempt at trying to find out _what_ the update touches.
A fundamental problem here is that even if a tester confirms that the
update fixes a _single_ bug, the other several dozens of changes could
cause regression -> reason to be careful and test this thing a bit longer.
Sure, what i am saying is that this could have been prevented if the
team just put a little more passion into it and also did some proof
reading/coordination. The team knows whats going on. They know the
issues and they can quickly and effortlessly identify issues like these
if only they would take some time to watch each others commits.
> and even if there was they would still would need to be able
trace
> the bug to SELinux.
That has been easy once the update arrived here on the nearby mirror.
"setenforce 0 && repeat previous command that caused strange
behaviour"
is a very common troubleshooting thing, even if there haven't been any
AVC denied messages.
If it was as common as you make it sound then maybe it might not have
come this far. It did. Again, one would have first had to identify the
issue (e.g. run RPM). There was no indication of any change related to
RPM (no change log entry).
But sure i give you that, yes thorough testing could have also prevented
this. (i still think its pretty unlikely but it could so i will take
that back)
Never the less, I think this issue could have been prevented even before
a package was spun.
3:54 p.m.
On Fri, 24 Jan 2014 22:17:20 +0100, Dominick Grift wrote:
Because you would need to run RPM to notice it,
Or Yum, DNF, Yumex, PackageKit, all tools on top of RPM would run into the
scriptlet errors. ;) Provided that you get a chance to evaluate the
installed test update for some time and the vote won't be too late.
> That has been easy once the update arrived here on the nearby
mirror.
> "setenforce 0 && repeat previous command that caused strange
behaviour"
> is a very common troubleshooting thing, even if there haven't been any
> AVC denied messages.
If it was as common as you make it sound then maybe it might not have
come this far.
Well, as mentioned before, this test update had been marked stable and
pushed into the updates repo already before appearing in updates-testing
on more mirrors. Worse if some testers fetch packages from koji and vote
in bodhi too early. By the time the first testers noticed the scriptlet
errors it was too late, since stable updates cannot be withdrawn.
It did. Again, one would have first had to identify the
issue (e.g. run RPM). There was no indication of any change related to
RPM (no change log entry).
Unconvincing. A similar thing has been prevented in a Yum Test Update some
weegs ago only because some _more_ testers have _not_ voted +1 before
actually using the updated Yum for some time. That is a lesson to
learn. Watch the votes:
https://admin.fedoraproject.org/updates/FEDORA-2013-22706/yum-3.4.3-119.fc20
4:05 p.m.
On Fri, 2014-01-24 at 22:54 +0100, Michael Schwendt wrote:
On Fri, 24 Jan 2014 22:17:20 +0100, Dominick Grift wrote:
> >
https://admin.fedoraproject.org/updates/FEDORA-2014-0806/selinux-policy-3...
> >
> Because you would need to run RPM to notice it,
Or Yum, DNF, Yumex, PackageKit, all tools on top of RPM would run into the
scriptlet errors. ;) Provided that you get a chance to evaluate the
installed test update for some time and the vote won't be too late.
You're right, i suppose there is room for improvement there as well,
just like there is room for improvement when it comes to how the
maintainers work.
Thanks for reading, i just wanted to shine a little light on the other
end of the story, for the record.
12:17 p.m.
Michael Schwendt wrote:
By the time the first testers noticed the scriptlet errors it was
too
late, since stable updates cannot be withdrawn.
That is also not a law of Physics. In the early days of Bodhi, one could
actually unpush stuff from stable. Having stable updates become immutable is
purely a policy decision. Withdrawing faulty updates has been done in the
past (even after Bodhi stopped allowing it in the normal case; the pulling
has then been done by an admin) and should be done again. Of course it won't
fix the systems that already got upgraded, but it will (within mirroring
delays) stop MORE systems from getting affected (and those that did already
get the faulty update won't notice the difference, unless they distro-sync,
in which case withdrawing the update actually fixes them, so in no case does
it make things worse for them).
And I don't see any valid reason why stable updates cannot simply be
withdrawn or sent back to testing by the maintainer. The update notes should
also remain editable, so that bug references can be added when the bug was
only found to be fixed after the stable push, errors in the update
description can be fixed, etc.
Kevin Kofler
1:28 p.m.
On Sat, 25 Jan 2014 19:17:14 +0100, Kevin Kofler wrote:
> By the time the first testers noticed the scriptlet errors it
was too
> late, since stable updates cannot be withdrawn.
That is also not a law of Physics. In the early days of Bodhi, one could
actually unpush stuff from stable.
Pointing that out doesn't make a difference. Obviously, I don't refer
to technical contraints. Even before bodhi, e.g., the Fedora Extras signers
could modify the master repo in an emergency situation.
Having stable updates become immutable is purely a policy decision.
Sure.
Withdrawing faulty updates has been done in the
past (even after Bodhi stopped allowing it in the normal case; the pulling
has then been done by an admin) and should be done again. Of course it won't
fix the systems that already got upgraded, but it will (within mirroring
delays) stop MORE systems from getting affected (and those that did already
get the faulty update won't notice the difference, unless they distro-sync,
in which case withdrawing the update actually fixes them, so in no case does
it make things worse for them).
Not sure that can be generalised. Distro-sync may downgrade packages.
We don't test downgrades of packages (scriptlets e.g.), and we don't test
downgrades of software either. We can't be sure downgraded software can
restore state at runtime after a previous upgrade may have touched
(= converted, renamed or replaced) config files or database files.
Downgrades could also affect dependencies and may make it necessary
to have a system update tool run distro-sync automatically. There are
enough users already, who play too much with --skip-broken instead of
reporting uninstallable updates/packages quickly.
12:10 p.m.
Dominick Grift wrote:
Sure, what i am saying is that this could have been prevented if the
team just put a little more passion into it and also did some proof
reading/coordination. The team knows whats going on. They know the
issues and they can quickly and effortlessly identify issues like these
if only they would take some time to watch each others commits.
Looking at the history of the involved bugs, using manual pushes rather than
the broken karma automatism and taking into account Bugzilla comments, not
just Bodhi comments, would probably also have prevented this fiasco. One of
the bugs (not the one that ended up becoming the canonical bug, but an
earlier one) was reassigned to selinux-policy fairly quickly.
One of the major flaws in the Bodhi karma system is that it cannot possibly
see what happens in Bugzilla.
Never the less, I think this issue could have been prevented even
before
a package was spun.
Yes, by disabling SELinux by default. :-)
Kevin Kofler
1 p.m.
On Sat, 2014-01-25 at 19:10 +0100, Kevin Kofler wrote:
> Never the less, I think this issue could have been prevented
even before
> a package was spun.
Yes, by disabling SELinux by default. :-)
No, that is a different discussion. Disabling SELinux does nothing to
solve this. If anything, to me this is confirmation of why we need a
good SELinux implementation. If this would happen to any other component
then a good SELinux implementation could have contained the damage
caused by issues just like this one.
The SELinux experience can, in my view be improved, and i believe your
problem is not with SELinux itself but with how it is
configured/implemented by default.
I just believe that a little team coordination, and a little more care
can go a long way, and that that is likely more efficient than trying to
create tests that would catch all of the bugs which sounds like utopia
to me.
I am not saying that the tests can't be improved or that they should not
be improved. It's just that in this case a little bit more care and a
double check by another involved party would probably have prevent this,
and similar other issues, in my view.
2:51 p.m.
Dominick Grift wrote:
No, that is a different discussion.
Nonsense. That SELinux should be disabled is the whole point of this thread
(I know, I have started it!), all the suggestions (in the various
subthreads) of how to paper over the problem are off topic.
Disabling SELinux does nothing to solve this.
Oh sure it does. It eliminates this whole class of breakage (critical
components unable to do their job because SELinux gets in their way) once
and for all. This type of breakage keeps occurring, in fact one instance is
still ongoing in Rawhide while we're discussing this:
https://bugzilla.redhat.com/show_bug.cgi?id=1052317
Disable SELinux and nothing will keep those components (RPM, display
managers, etc.) from doing their work.
If anything, to me this is confirmation of why we need a good
SELinux
implementation. If this would happen to any other component then a good
SELinux implementation could have contained the damage caused by issues
just like this one.
You don't seem to understand at all what SELinux is (it is not a tool
magically able to fix bugs, its only purpose is to keep programs from doing
their work)…
The SELinux experience can, in my view be improved, and i believe
your
problem is not with SELinux itself but with how it is
configured/implemented by default.
… nor what it can or cannot do. (No amount of configuration can make SELinux
do anything other than block (i.e. break) things.)
I just believe that a little team coordination, and a little more
care
can go a long way, and that that is likely more efficient than trying to
create tests that would catch all of the bugs which sounds like utopia
to me.
What "coordination"?
I am not saying that the tests can't be improved or that they
should not
be improved. It's just that in this case a little bit more care and a
double check by another involved party would probably have prevent this,
and similar other issues, in my view.
Sure, dropping autokarma could help (and should be done in any case, that
Bodhi "feature" never made any sense), but ultimately there's no way around
disabling SELinux. Enabling it by default in Fedora has always been a
mistake!
Kevin Kofler
3:08 p.m.
On Sat, 2014-01-25 at 21:51 +0100, Kevin Kofler wrote:
Dominick Grift wrote:
> No, that is a different discussion.
Nonsense. That SELinux should be disabled is the whole point of this thread
(I know, I have started it!), all the suggestions (in the various
subthreads) of how to paper over the problem are off topic.
Sorry, I must have misinterpreted the subject: "Drawing lessons from
fatal SELinux bug"
> Disabling SELinux does nothing to solve this.
Oh sure it does. It eliminates this whole class of breakage (critical
components unable to do their job because SELinux gets in their way) once
and for all. This type of breakage keeps occurring, in fact one instance is
still ongoing in Rawhide while we're discussing this:
https://bugzilla.redhat.com/show_bug.cgi?id=1052317
Disable SELinux and nothing will keep those components (RPM, display
managers, etc.) from doing their work.
> If anything, to me this is confirmation of why we need a good SELinux
> implementation. If this would happen to any other component then a good
> SELinux implementation could have contained the damage caused by issues
> just like this one.
You don't seem to understand at all what SELinux is (it is not a tool
magically able to fix bugs, its only purpose is to keep programs from doing
their work)…
I did not mean to suggest that. I meant to suggest that SELinux would be
able to contain the damage, referring to "fatal" in: "Drawing lessons
from fatal SELinux bug"
> The SELinux experience can, in my view be improved, and i
believe your
> problem is not with SELinux itself but with how it is
> configured/implemented by default.
… nor what it can or cannot do. (No amount of configuration can make SELinux
do anything other than block (i.e. break) things.)
Actually it is the other way around. SELinux blocks everything by
default. Everything needs to be explicitly allowed by means of
"configuration"
> I just believe that a little team coordination, and a little
more care
> can go a long way, and that that is likely more efficient than trying to
> create tests that would catch all of the bugs which sounds like utopia
> to me.
What "coordination"?
For example coordinate who does what where and when.
> I am not saying that the tests can't be improved or that they should not
> be improved. It's just that in this case a little bit more care and a
> double check by another involved party would probably have prevent this,
> and similar other issues, in my view.
Sure, dropping autokarma could help (and should be done in any case, that
Bodhi "feature" never made any sense), but ultimately there's no way around
disabling SELinux. Enabling it by default in Fedora has always been a
mistake!
Kevin Kofler
7:50 a.m.
Dominick Grift wrote:
I did not mean to suggest that. I meant to suggest that SELinux would
be
able to contain the damage, referring to "fatal" in: "Drawing lessons
from fatal SELinux bug"
And by what magic would it do that? SELinux cannot by its nature contain any
damage of the "the system is broken" type, no matter in what component. The
ONLY type of damage it can possibly contain is a security hole (and even
there, not all classes of security holes). All those fatal regressions where
basic system functionality such as upgrading or even logging in is non-
functional can absolutely NOT be fixed by SELinux.
Actually it is the other way around. SELinux blocks everything by
default. Everything needs to be explicitly allowed by means of
"configuration"
Yes. This "default deny" attitude is a big part of the problem. (Whenever a
program covered by a strict policy gets a new feature, the SELinux policy
has to be updated to allow it, i.e. a duplication of efforts and a
maintainability nightmare.) But no matter what you configure SELinux to
allow, it will only work if the program is coded to do it in the first
place, so you cannot use SELinux to fix a regression in another critical
component. The only regressions you can possibly fix with an SELinux update
are regressions in SELinux itself, i.e., the ones that can trivially be
avoided by disabling SELinux in the first place.
So I still don't follow when you claim SELinux can fix regressions
elsewhere. That argument just doesn't make sense, sorry.
Kevin Kofler
2:17 p.m.
On Fri, 24 Jan 2014 11:14:50 -0800, Adam Williamson wrote:
It's not at all
obvious to anyone that you ought to test update/install of another
package in order to validate an update to selinux-policy-targeted .
Hell, I don't do that.
Amazing.
https://admin.fedoraproject.org/updates/FEDORA-2014-0806/selinux-policy-3...
| selinux-policy-3.12.1-116.fc20 critical path bugfix update
https://fedoraproject.org/wiki/Critical_path_package#Actions
| Packages within the critical path are required to perform the
| most fundamental actions on a system. Those actions include:
|
| [...]
| get updates
| [...]
How to understand that?
Especially with regard to downloading builds from koji, installing them
manually and voting +1 even before a test update has entered the repo.
The fast people, who do that regularly in addition to a daily yum update,
could not escape from this bug. On the contrary, other users who don't
update often, have skipped the bad selinux policy update.
I consider it likely that the testers would have noticed Yum/RPM update
errors, if only they had used their updated systems normal for let's say
one or two days and at least one reboot.
There's also
fedora-easy-karma --installed-min-days=4
which is can very helpful, since you won't be asked for updates installed
just a few minutes ago.
Also let's not forget, for testing an selinux-policy-targeted update,
you ought to run with SELinux in enforcing mode.
The 'comment' field exists to allow people to express all
these things,
but as it's just a completely free-form text field,
... and even can be left empty :( so a packager doesn't get any
explicit feedback from the tester other than the +1.
Those are just examples: the point is that what we badly need here is
a
more expressive and flexible system. (As well, as I've said elsewhere in
the discussion, as a good automated test for this specific and
well-known category of 'delayed action' update problems).
Is it so hard for testers to slow down a bit until such a system will be
available? ;-)
2:39 p.m.
On Fri, 2014-01-24 at 21:17 +0100, Michael Schwendt wrote:
On Fri, 24 Jan 2014 11:14:50 -0800, Adam Williamson wrote:
> It's not at all
> obvious to anyone that you ought to test update/install of another
> package in order to validate an update to selinux-policy-targeted .
> Hell, I don't do that.
Amazing.
https://admin.fedoraproject.org/updates/FEDORA-2014-0806/selinux-policy-3...
| selinux-policy-3.12.1-116.fc20 critical path bugfix update
https://fedoraproject.org/wiki/Critical_path_package#Actions
| Packages within the critical path are required to perform the
| most fundamental actions on a system. Those actions include:
|
| [...]
| get updates
| [...]
How to understand that?
It also says:
graphical network install
post-install booting
decrypt encrypted filesystems
graphics
login
networking
get updates
minimal buildroot
compose new trees
compose live
Are we to be expected to re-test every single one of those actions with
every single critical path update? That seems unreasonable. If that were
the approach, I think Kevin would have an actual apoplexy while waiting
for his updates to get released. :)
Especially with regard to downloading builds from koji, installing
them
manually and voting +1 even before a test update has entered the repo.
The fast people, who do that regularly in addition to a daily yum update,
could not escape from this bug. On the contrary, other users who don't
update often, have skipped the bad selinux policy update.
I consider it likely that the testers would have noticed Yum/RPM update
errors, if only they had used their updated systems normal for let's say
one or two days and at least one reboot.
There's also
fedora-easy-karma --installed-min-days=4
which is can very helpful, since you won't be asked for updates installed
just a few minutes ago.
Yup, indeed. Of course, this is another area where we could improve the
tooling: it doesn't seem like it'd be difficult for maintainers to be
allowed to set a minimum timeframe before their update goes stable, but
at present this isn't possible.
Also let's not forget, for testing an selinux-policy-targeted
update,
you ought to run with SELinux in enforcing mode.
This is already generally understood, I think.
> The 'comment' field exists to allow people to express
all these things,
> but as it's just a completely free-form text field,
... and even can be left empty :( so a packager doesn't get any
explicit feedback from the tester other than the +1.
> Those are just examples: the point is that what we badly need here is a
> more expressive and flexible system. (As well, as I've said elsewhere in
> the discussion, as a good automated test for this specific and
> well-known category of 'delayed action' update problems).
Is it so hard for testers to slow down a bit until such a system will be
available? ;-)
Well, at present, it kind of is, because we have this Bodhi-Bugzilla
interaction where when an update is submitted that claims to fix a given
bug, Bodhi posts a comment on the bug that says 'try this update and
leave positive karma if it fixes your bug'.
I mean, we could tweak that process, but I still think the Grand Bodhi
2.0 Vision is much more interesting, because if we have multiple karma
types we can still take and use fast 'this fixed my bug' feedback
however we feel it to be appropriate, while having _much_ more
flexibility to do 'valuation' of feedback on a 'type of feedback' and
'package being updated' basis. Again, hate to sound like a broken
record, but it's just hard to get enthusiastic about trying to twiddle
the edges of the process when the process is fundamentally inadequate.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
3:36 p.m.
On Fri, 24 Jan 2014 12:39:55 -0800, Adam Williamson wrote:
> https://fedoraproject.org/wiki/Critical_path_package#Actions
>
> | Packages within the critical path are required to perform the
> | most fundamental actions on a system. Those actions include:
> |
> | [...]
> | get updates
> | [...]
>
>
> How to understand that?
It also says:
graphical network install
post-install booting
decrypt encrypted filesystems
graphics
login
networking
get updates
minimal buildroot
compose new trees
compose live
Are we to be expected to re-test every single one of those actions with
every single critical path update? That seems unreasonable. If that were
the approach, I think Kevin would have an actual apoplexy while waiting
for his updates to get released. :)
It would be unreasonable for a single tester, but "the more testers, the
better".
Which doesn't mean the group of testers will perform _all_ of the
(re-)tests. It means: the more testers get a chance to apply Test Updates
*and* continue using their systems normally for a few days, the more
likely it could get that some will notice issues at run-time, boot-time,
compile-time, to mention a few.
That's why I think there's reason to be very careful and sometimes even
prefer a +0 (with a comment) over a very early over-ambitious +1.
And guess what happens in non-critpath updates after 7 days and _no_
feedback. Packagers push the update manually. Sometimes with broken
deps. Sometimes the testing starts no sooner than when the update arrives
in the stable updates repo and the first real user becomes the "guinea
pig".
> Is it so hard for testers to slow down a bit until such a system
will be
> available? ;-)
Well, at present, it kind of is, because we have this Bodhi-Bugzilla
interaction where when an update is submitted that claims to fix a given
bug, Bodhi posts a comment on the bug that says 'try this update and
leave positive karma if it fixes your bug'.
Good point. Raises the question why an update that links so many bugzilla
tickets can be marked stable automatically after a +3, which may be even
about a single bz ticket.
I mean, we could tweak that process, but I still think the Grand
Bodhi
2.0 Vision is much more interesting, because if we have multiple karma
types we can still take and use fast 'this fixed my bug' feedback
however we feel it to be appropriate, while having _much_ more
flexibility to do 'valuation' of feedback on a 'type of feedback' and
'package being updated' basis. Again, hate to sound like a broken
record, but it's just hard to get enthusiastic about trying to twiddle
the edges of the process when the process is fundamentally inadequate.
That's understood, of course.
4:35 p.m.
On 24 January 2014 14:36, Michael Schwendt <mschwendt(a)gmail.com> wrote:
On Fri, 24 Jan 2014 12:39:55 -0800, Adam Williamson wrote:
> > https://fedoraproject.org/wiki/Critical_path_package#Actions
> >
> > | Packages within the critical path are required to perform the
> > | most fundamental actions on a system. Those actions include:
> > |
> > | [...]
> > | get updates
> > | [...]
> >
> >
> > How to understand that?
>
> It also says:
>
> graphical network install
> post-install booting
> decrypt encrypted filesystems
> graphics
> login
> networking
> get updates
> minimal buildroot
> compose new trees
> compose live
>
> Are we to be expected to re-test every single one of those actions with
> every single critical path update? That seems unreasonable. If that were
> the approach, I think Kevin would have an actual apoplexy while waiting
> for his updates to get released. :)
It would be unreasonable for a single tester, but "the more testers, the
better".
Looking at the number of people who respond to the qa list at times.. I am
going to say there are probably 6-10 active testers during non-release
times. It comes and it goes, but that is about the number who seem active
at least on lists (and it seems to be that way going over archives for the
last couple of years.) So any policy would need to take into account of
that limitation.
--
Stephen J Smoogen.
4:46 p.m.
On Fri, 24 Jan 2014 15:35:24 -0700, Stephen John Smoogen wrote:
Looking at the number of people who respond to the qa list at times..
I am
going to say there are probably 6-10 active testers during non-release
times. It comes and it goes, but that is about the number who seem active
at least on lists (and it seems to be that way going over archives for the
last couple of years.) So any policy would need to take into account of
that limitation.
Any ideas how to attract more testers?
How to make the updates-testing repo more sexy?
More lessons to learn:
https://admin.fedoraproject.org/updates/FEDORA-2013-23627
Karma: 17
Stable karma: 16 (!)
It has reached the karma threshold 16 after ~5 days.
And those have not been all testers.
5:21 p.m.
Am 24.01.2014 23:46, schrieb Michael Schwendt:
On Fri, 24 Jan 2014 15:35:24 -0700, Stephen John Smoogen wrote:
> Looking at the number of people who respond to the qa list at times.. I am
> going to say there are probably 6-10 active testers during non-release
> times. It comes and it goes, but that is about the number who seem active
> at least on lists (and it seems to be that way going over archives for the
> last couple of years.) So any policy would need to take into account of
> that limitation.
Any ideas how to attract more testers?
How to make the updates-testing repo more sexy?
More lessons to learn:
https://admin.fedoraproject.org/updates/FEDORA-2013-23627
Karma: 17
Stable karma: 16 (!)
It has reached the karma threshold 16 after ~5 days.
And those have not been all testers
but where do you see a problem with that package?
hreindl - 2013-12-20 11:07:28
works for me
[root@srv-rhsoft:~]$ rpm -q yum
yum-3.4.3-132.fc20.noarch
Jan 22 12:48:00 Updated: yum-3.4.3-132.fc20.noarch
________________________________________
built explicitly to test the yum package:
Jan 22 12:52:25 Updated: lounge-rhsoft-workstation-20.0-2.fc20.20140122.rh.noarch
updates after that:
Jan 22 14:09:10 Updated: ffmpeg-latest-manpages-2.1.3-4.fc20.20140122.rh.noarch
Jan 22 14:09:11 Updated: ffmpeg-latest-2.1.3-4.fc20.20140122.rh.x86_64
Jan 22 14:28:03 Installed: iftop-1.0-0.7.pre4.fc20.x86_64
Jan 22 14:28:03 Updated: lounge-rhsoft-workstation-20.0-3.fc20.20140122.rh.noarch
Jan 23 11:29:10 Updated: apcupsd-3.14.10-14.fc20.x86_64
Jan 23 11:29:10 Updated: apcupsd-gui-3.14.10-14.fc20.x86_64
Jan 23 16:32:25 Updated: glibc-2.18-12.fc20.x86_64
Jan 23 16:32:30 Updated: glibc-common-2.18-12.fc20.x86_64
Jan 23 16:32:30 Updated: pulseaudio-libs-4.0-12.gitf81e3.fc20.x86_64
Jan 23 16:32:30 Updated: libtool-ltdl-2.4.2-23.fc20.x86_64
Jan 23 16:32:31 Updated: pulseaudio-4.0-12.gitf81e3.fc20.x86_64
Jan 23 16:32:31 Updated: pulseaudio-utils-4.0-12.gitf81e3.fc20.x86_64
Jan 23 16:32:32 Updated: glibc-headers-2.18-12.fc20.x86_64
Jan 23 16:32:33 Updated: glibc-devel-2.18-12.fc20.x86_64
Jan 23 16:32:33 Updated: pulseaudio-module-x11-4.0-12.gitf81e3.fc20.x86_64
Jan 23 16:32:33 Updated: pulseaudio-libs-glib2-4.0-12.gitf81e3.fc20.x86_64
Jan 23 16:38:22 Updated: 12:dhcp-libs-4.2.6-0.1.b1.fc20.x86_64
Jan 23 16:38:22 Updated: 12:dhcp-common-4.2.6-0.1.b1.fc20.x86_64
Jan 23 16:38:23 Updated: 12:dhcp-4.2.6-0.1.b1.fc20.x86_64
Jan 23 16:38:23 Updated: 12:dhclient-4.2.6-0.1.b1.fc20.x86_64
Jan 23 21:03:57 Updated: tzdata-java-2013i-2.fc20.noarch
Jan 23 21:03:57 Updated: crypto-utils-2.4.1-46.fc20.x86_64
Jan 23 21:03:58 Updated: krb5-libs-1.11.3-39.fc20.x86_64
Jan 23 21:03:59 Updated: google-crosextra-caladea-fonts-1.002-0.3.20130214.fc20.noarch
Jan 23 21:04:01 Updated: webkitgtk-2.2.4-1.fc20.x86_64
Jan 23 21:04:02 Updated: tzdata-2013i-2.fc20.noarch
Jan 24 17:41:58 Updated: fltk-1.3.2-3.fc20.x86_64
Jan 24 17:41:58 Updated: libnl3-3.2.24-1.fc20.x86_64
Jan 24 17:41:58 Updated: freeglut-2.8.1-3.fc20.x86_64
Jan 24 17:41:58 Updated: pango-1.36.1-2.fc20.x86_64
6:22 p.m.
On Sat, 25 Jan 2014 00:21:13 +0100, Reindl Harald wrote:
> More lessons to learn:
>
> https://admin.fedoraproject.org/updates/FEDORA-2013-23627
> Karma: 17
> Stable karma: 16 (!)
>
> It has reached the karma threshold 16 after ~5 days.
> And those have not been all testers
but where do you see a problem with that package?
With the bodhi ticket? I don't. :)
I've pointed at this because it's another good example of an
increase karma threshold.
Of course, and at the risk of repeating it too often, the better
example (due to the early votes) is this:
https://admin.fedoraproject.org/updates/FEDORA-2013-22706/yum-3.4.3-119.fc20
5:34 p.m.
Am 24.01.2014 23:46, schrieb Michael Schwendt:
On Fri, 24 Jan 2014 15:35:24 -0700, Stephen John Smoogen wrote:
> Looking at the number of people who respond to the qa list at times.. I am
> going to say there are probably 6-10 active testers during non-release
> times. It comes and it goes, but that is about the number who seem active
> at least on lists (and it seems to be that way going over archives for the
> last couple of years.) So any policy would need to take into account of
> that limitation.
Any ideas how to attract more testers?
How to make the updates-testing repo more sexy?
* i am running updates testing 365/24 over the last 3 years
* i am running "yum --enablerepo=updates-testing --security" in production
* i test this packages in "near production" (test-vm-mirrors)
the only ones i leave out are *real* server packages because i
build them in general at my own inlcuding major-updates Fedora
not see at all or the other direction (PHP 5.4 first time with F18)
More lessons to learn
yes here: https://bugzilla.redhat.com/show_bug.cgi?id=1019251
Joe Orton 2013-11-18 07:15:12 EST
Upstream is gearing up for 2.4.7 RSN - ECC support will get
picked up automagically when we do a new f19 build
where is the 2.4.7 build for F19?
more than 3 months ago i had httpd-2.4.6 with ECDHE *in production* on F18
and so confirmed that httpd works fine as expected with the new openssl
https://bugzilla.redhat.com/show_bug.cgi?id=319901#c108
12:29 p.m.
Michael Schwendt wrote:
More lessons to learn:
https://admin.fedoraproject.org/updates/FEDORA-2013-23627
Karma: 17
Stable karma: 16 (!)
It has reached the karma threshold 16 after ~5 days.
And those have not been all testers.
That can work for yum, but if I set the stable karma to 16 for kfloppy, the
release will reach its EOL without it getting there (unless somebody targets
testers specifically at kfloppy to win the bet ;-) … and buys them floppy
drives, too ;-) ). Heck, even if Fedora releases didn't have an EOL, you or
me probably wouldn't live to see it go stable.
Kevin Kofler
1:28 p.m.
On Sat, 25 Jan 2014 19:29:12 +0100, Kevin Kofler wrote:
> https://admin.fedoraproject.org/updates/FEDORA-2013-23627
> Karma: 17
> Stable karma: 16 (!)
>
> It has reached the karma threshold 16 after ~5 days.
> And those have not been all testers.
That can work for yum, but if I set the stable karma to 16 for kfloppy,
Nobody suggested doing that for kfloppy or other packages, which hardly
ever get feedback in bodhi.
If a test update doesn't get any feedback in bodhi, what does that imply?
If you mark it stable after 7 days, you've tested it yourself for some
days, correct?
If the update doesn't refer to any bugzilla tickets, what does that mean?
the
release will reach its EOL without it getting there (unless somebody targets
testers specifically at kfloppy to win the bet ;-) … and buys them floppy
drives, too ;-) ). Heck, even if Fedora releases didn't have an EOL, you or
me probably wouldn't live to see it go stable.
Almost funny, if it weren't possible to mark test updates as stable after
7 days.
It could be that nobody uses the package at all, so it would not a big
deal if an update (or upgrade?) took 7+ days to enter the updates repo. ;-p
3 p.m.
Michael Schwendt wrote:
If the update doesn't refer to any bugzilla tickets, what does
that mean?
In that particular case, it means that we are updating all the KDE software
compilation and so there's a new release of KFloppy too, which most likely
doesn't even contain any actual changes from upstream (just a new version
number on the tarball), but the updates are scripted, and the version bump
is also needed to keep our metapackages (kdeutils in this case) working. :-)
That said, in practice, we file those as grouped updates and so there's a
chance that the update actually gets some karma. Surely not because of
KFloppy though. ;-)
Almost funny, if it weren't possible to mark test updates as
stable after
7 days.
Right, but you were proposing to wait until it reaches a karma of +16.
It could be that nobody uses the package at all, so it would not a
big
deal if an update (or upgrade?) took 7+ days to enter the updates repo.
;-p
But then the right solution is to disable karma automatism entirely, not to
set it to some ridiculously high value.
Those meaningless thresholds need to go away (and really, the whole concept
of Bodhi karma and the policies that depend on it).
Kevin Kofler
3:11 p.m.
Am 25.01.2014 22:00, schrieb Kevin Kofler:
But then the right solution is to disable karma automatism entirely,
not to
set it to some ridiculously high value.
Those meaningless thresholds need to go away (and really, the whole concept
of Bodhi karma and the policies that depend on it)
i am not entirely sure how that is meant
* disable the automatism to push to stable
* forget the whole karma system at all
in case of "disable the automatism to push to stable" i agree
in my opinion karma is a indication for the maintainer but not
the decision - the karma has to be handeled differently for the
same package and different updates and only the maintainer can
decide that *as person*
why?
because it depends on the change itself
speaking with my developer hat on: there are updates on software
inside our company where i do not hestitate a single seconds deploy
the new CMS version to some hundrets of customers without tell anybody
there was a update at all because *i know* there can be no bad impact
on the other hand there are updates and changes which needs to prepare
any singel webhost, rollout a small update to prepare the real one by
add database colums not used currently but need to be there in the time
window files are replaced and database scheme can be updated
the second case is for not have any single request going wrong
and there is another category where all the work above has to be done
and tested thousands of times but still need a "keep your eyes open"
after it is done because you can't test and verify every single action
a complex software may do with every possible input data
7:54 a.m.
Reindl Harald wrote:
i am not entirely sure how that is meant
* disable the automatism to push to stable
* forget the whole karma system at all
in case of "disable the automatism to push to stable" i agree
Even just doing that would be an improvement, but I still think the whole
karma system should go away entirely and the maintainers should have the
call.
in my opinion karma is a indication for the maintainer but not
the decision - the karma has to be handeled differently for the
same package and different updates and only the maintainer can
decide that *as person*
why?
because it depends on the change itself
I totally agree that the maintainer should be the one making the call!
That's why I want the karma stuff removed. :-)
What's the point of keeping that number if we drop the silly Update
Policies? Shouldn't the maintainer actually READ the comments instead of
basing the decision on an arbitrary algebraic sum of unweighted +1 and -1
terms?
speaking with my developer hat on: there are updates on software
inside our company where i do not hestitate a single seconds deploy
the new CMS version to some hundrets of customers without tell anybody
there was a update at all because *i know* there can be no bad impact
on the other hand there are updates and changes which needs to prepare
any singel webhost, rollout a small update to prepare the real one by
add database colums not used currently but need to be there in the time
window files are replaced and database scheme can be updated
the second case is for not have any single request going wrong
and there is another category where all the work above has to be done
and tested thousands of times but still need a "keep your eyes open"
after it is done because you can't test and verify every single action
a complex software may do with every possible input data
+1
Kevin Kofler
3:25 p.m.
On Sat, 25 Jan 2014 22:00:02 +0100, Kevin Kofler wrote:
Right, but you were proposing to wait until it reaches a karma of
+16.
Certainly not. That Yum update is only a good example where a high
karma threshold has been reached in less than a week, and even without
a vote from all available/active testers. Sure, Yum is widely-used, which
cannot be said about niche-market packages.
I'm proposing that Test Updates are offered for a minimum duration
to allow for the time it takes to push them into the repo *and*
be picked up by world-wide mirrors. That increases the chance that
available testers get a chance of evaluating an update and giving
feedback _before_ it gets marked stable because of reaching +3 very
early (based on koji downloads, for example).
That has worked rather well for the more interesting update with a karma
threshold of 5: https://admin.fedoraproject.org/updates/FEDORA-2013-22706
The -1 votes have not been too late.
On the contrary, if a tester is available and finds a bug in a new
update as offered on a nearby mirror, but in bodhi the update has been
marked stable in less than a day already or has skipped updates-testing
even, the tester cannot help anymore. That's a case that would be
avoidable.
> It could be that nobody uses the package at all, so it would not
a big
> deal if an update (or upgrade?) took 7+ days to enter the updates repo.
> ;-p
But then the right solution is to disable karma automatism entirely, not to
set it to some ridiculously high value.
Yes, debatable. But we shouldn't argue about it. '5' or '10' for Yum
updates isn't ridiculously high while still giving testers the opportunity
to vote cleverly and trigger the automatic push.
9:18 a.m.
On Fri, Jan 24, 2014 at 11:46:41PM +0100, Michael Schwendt wrote:
Any ideas how to attract more testers?
How to make the updates-testing repo more sexy?
* More automated smoke-tests, so that:
a) you know the most boring tests have been handled so you
can focus on more interesting apsects of the update
a) it's less likely that something broken will slip through,
so it's more safe to keep the testing repo active
* possibly adding a "what should users test?" field to the update info.
I know that there's a "notes" field in the update, but maybe it'd
help
to explicitly include testing instructions?
Each package in the pkgdb (or in git, or wherever) could have
a standard list included in each update as the default (for example, for
'calc', it might be to try `calc -q read /usr/share/calc/regress.cal`.
That would duplicate a likely smoke-test, though, so maybe also "run
interactively and make sure basic math works".
Then, each update could also optionally (and this would be presented in
bold if it were used) say something like "New release adds log() function;
please test that it works", or "Severe bug where 1+1=3 corrected; please
test that the answer now corresponds with consensus reality."
* badges! We already have this, but making them more visible would help.
People on Stack Exchange get crazily obsessed with quality control all
in exchange for digital gold stars.
* In fact, this ties into the general problem that various bits of Fedora
are disconnected and not particularly discoverable. You can find them if
you're looking, but mostly out-of-site, out-of-mind. Unless one is already
engaged, how would one even know that this is really an easy way to help?
* Present pending updates in a more informative way.
Let's say I'm in the mood to be helpful and test some stuff and earn some
badges. I go to https://admin.fedoraproject.org/updates/ (because I have
that bookmarked; see the previous point). I see a list of package names,
and update types. Maybe I recognize something, maybe I don't. I don't
right now, it happens, so I think "okay, there's xflr5. Maybe that's
interesting". I click on
https://admin.fedoraproject.org/updates/xflr5-6.09.06-1.fc20, and am
immediately shown that this is only a package I should care about if I
already know what it is -- that is, there's no description at all. The
update note is reasonably helpful as these things go (it's an update to a
new upstream version), but the URL isn't hyperlinked, and when I go there,
Sourceforge and makes me wait and then makes the file download instead of
displaying it. And then I see that this program is very a technical
airplane engineering program and I'm not qualified to test it. Okay, there
went five minutes of my life. Shall I guess another one? Hmmm, maybe
"alleyoop". Will there be cavemen? Okay, also no description, so I'll do
"yum info" in a shell. Ah, okay, memory debugger GUI. Okay, I can test
that one but it's not necessarily gonna be quick....
Anyway. The list could be more informative. Maybe I could even star
particular packages I'm interested in, and future updates would show up
first. And after choosing something, the above idea of having a quick
description of what to test would help here too.
* Silly, but... remember my login in bodhi longer, so there are fewer
steps.
--
Matthew Miller -- Fedora Project -- <mattdm(a)fedoraproject.org>
4:11 p.m.
On Mon, 27 Jan 2014 10:18:56 -0500
Matthew Miller <mattdm(a)fedoraproject.org> wrote:
....snip....
* possibly adding a "what should users test?" field to the
update
info.
I know that there's a "notes" field in the update, but maybe it'd
help to explicitly include testing instructions?
Each package in the pkgdb (or in git, or wherever) could have
a standard list included in each update as the default (for
example, for 'calc', it might be to try `calc -q
read /usr/share/calc/regress.cal`. That would duplicate a likely
smoke-test, though, so maybe also "run interactively and make sure
basic math works".
Then, each update could also optionally (and this would be
presented in bold if it were used) say something like "New release
adds log() function; please test that it works", or "Severe bug where
1+1=3 corrected; please test that the answer now corresponds with
consensus reality."
I could have sworn we already had something like this where bodhi would
add a link to a wiki page for test plan on a package if that wiki page
existed. I can't seem to find it now, so perhaps it was just something
we talked about, but never implemented.
kevin
8:39 p.m.
On Mon, 2014-01-27 at 15:11 -0700, Kevin Fenzi wrote:
On Mon, 27 Jan 2014 10:18:56 -0500
Matthew Miller <mattdm(a)fedoraproject.org> wrote:
....snip....
> * possibly adding a "what should users test?" field to the update
> info.
>
> I know that there's a "notes" field in the update, but maybe
it'd
> help to explicitly include testing instructions?
>
> Each package in the pkgdb (or in git, or wherever) could have
> a standard list included in each update as the default (for
> example, for 'calc', it might be to try `calc -q
> read /usr/share/calc/regress.cal`. That would duplicate a likely
> smoke-test, though, so maybe also "run interactively and make sure
> basic math works".
> Then, each update could also optionally (and this would be
> presented in bold if it were used) say something like "New release
> adds log() function; please test that it works", or "Severe bug where
> 1+1=3 corrected; please test that the answer now corresponds with
> consensus reality."
I could have sworn we already had something like this where bodhi would
add a link to a wiki page for test plan on a package if that wiki page
existed. I can't seem to find it now, so perhaps it was just something
we talked about, but never implemented.
Nope, you're right. :)
https://github.com/fedora-infra/bodhi/blob/develop/bodhi/model.py#L191
For example, the test case pages for the package 'foo' need to be added
to the 'Category:Package foo test cases' category.
There's also an option in the config file which must be switched on for
this to work: 'query_wiki_test_cases'
And here is an update where this is, in fact, actually used:
https://admin.fedoraproject.org/updates/FEDORA-2014-1465/
--
Mathieu
9:24 p.m.
On Tue, 2014-01-28 at 10:39 +0800, Mathieu Bridon wrote:
On Mon, 2014-01-27 at 15:11 -0700, Kevin Fenzi wrote:
> On Mon, 27 Jan 2014 10:18:56 -0500
> Matthew Miller <mattdm(a)fedoraproject.org> wrote:
>
> ....snip....
>
> > * possibly adding a "what should users test?" field to the update
> > info.
> >
> > I know that there's a "notes" field in the update, but maybe
it'd
> > help to explicitly include testing instructions?
> >
> > Each package in the pkgdb (or in git, or wherever) could have
> > a standard list included in each update as the default (for
> > example, for 'calc', it might be to try `calc -q
> > read /usr/share/calc/regress.cal`. That would duplicate a likely
> > smoke-test, though, so maybe also "run interactively and make sure
> > basic math works".
>
> > Then, each update could also optionally (and this would be
> > presented in bold if it were used) say something like "New release
> > adds log() function; please test that it works", or "Severe bug
where
> > 1+1=3 corrected; please test that the answer now corresponds with
> > consensus reality."
>
> I could have sworn we already had something like this where bodhi would
> add a link to a wiki page for test plan on a package if that wiki page
> existed. I can't seem to find it now, so perhaps it was just something
> we talked about, but never implemented.
Nope, you're right. :)
https://github.com/fedora-infra/bodhi/blob/develop/bodhi/model.py#L191
For example, the test case pages for the package 'foo' need to be added
to the 'Category:Package foo test cases' category.
There's also an option in the config file which must be switched on for
this to work: 'query_wiki_test_cases'
And here is an update where this is, in fact, actually used:
https://admin.fedoraproject.org/updates/FEDORA-2014-1465/
This is the package test plan thing - the QA docs on it are at
https://fedoraproject.org/wiki/QA:SOP_package_test_plan_creation . I
usually use xorg-x11-drv-nouveau updates as a handy example of it in
action. You can create a test case for a package and add it to the
category Category:Package_(packagename)_test_cases (where 'packagename'
is the .src.rpm name), and it will show up in Bodhi like this.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
1:55 p.m.
On Mon, 2014-01-27 at 19:24 -0800, Adam Williamson wrote:
On Tue, 2014-01-28 at 10:39 +0800, Mathieu Bridon wrote:
> On Mon, 2014-01-27 at 15:11 -0700, Kevin Fenzi wrote:
> > On Mon, 27 Jan 2014 10:18:56 -0500
> > Matthew Miller <mattdm(a)fedoraproject.org> wrote:
> >
> > ....snip....
> >
> > > * possibly adding a "what should users test?" field to the
update
> > > info.
> > >
> > > I know that there's a "notes" field in the update, but
maybe it'd
> > > help to explicitly include testing instructions?
> > >
> > > Each package in the pkgdb (or in git, or wherever) could have
> > > a standard list included in each update as the default (for
> > > example, for 'calc', it might be to try `calc -q
> > > read /usr/share/calc/regress.cal`. That would duplicate a likely
> > > smoke-test, though, so maybe also "run interactively and make sure
> > > basic math works".
> >
> > > Then, each update could also optionally (and this would be
> > > presented in bold if it were used) say something like "New release
> > > adds log() function; please test that it works", or "Severe bug
where
> > > 1+1=3 corrected; please test that the answer now corresponds with
> > > consensus reality."
> >
> > I could have sworn we already had something like this where bodhi would
> > add a link to a wiki page for test plan on a package if that wiki page
> > existed. I can't seem to find it now, so perhaps it was just something
> > we talked about, but never implemented.
>
> Nope, you're right. :)
>
> https://github.com/fedora-infra/bodhi/blob/develop/bodhi/model.py#L191
>
> For example, the test case pages for the package 'foo' need to be added
> to the 'Category:Package foo test cases' category.
>
> There's also an option in the config file which must be switched on for
> this to work: 'query_wiki_test_cases'
>
> And here is an update where this is, in fact, actually used:
> https://admin.fedoraproject.org/updates/FEDORA-2014-1465/
This is the package test plan thing - the QA docs on it are at
https://fedoraproject.org/wiki/QA:SOP_package_test_plan_creation . I
usually use xorg-x11-drv-nouveau updates as a handy example of it in
action. You can create a test case for a package and add it to the
category Category:Package_(packagename)_test_cases (where 'packagename'
is the .src.rpm name), and it will show up in Bodhi like this.
Actually, if it wouldn't be _too_ much work, a handy extension to this
suggests itself: it might be nice to be able to use a similar mechanism
to mark a test case as being relevant to an entire group of packages -
whether that means 'any comps group' or we just special-case critpath,
or whatever.
It'd be nice to write a 'install the update, then try installing and
removing a package' test case, and mark it to show up on all critpath
updates, maybe.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
2:19 p.m.
On Tue, Jan 28, 2014 at 10:39:32AM +0800, Mathieu Bridon wrote:
> I could have sworn we already had something like this where
bodhi would
> add a link to a wiki page for test plan on a package if that wiki page
> existed. I can't seem to find it now, so perhaps it was just something
> we talked about, but never implemented.
Nope, you're right. :)
https://github.com/fedora-infra/bodhi/blob/develop/bodhi/model.py#L191
For example, the test case pages for the package 'foo' need to be added
to the 'Category:Package foo test cases' category.
Huh, lookit that. That's great! So I guess my suggestion changes to "make
that more visible".
Also my head is exploding a little bit, because it's yet another case of the
wiki doing a lot of different types of work. I wonder if it would help both
of these things to move the test plans to live in package git (right next to
the future taskotron scripts?
And, while clearly helpful, I think it suffers from a little bit of tl;dr.
Maybe we could embed summaries of each right into the Bodhi page? Ooh,
possibly with checkboxes by each? After the other stuff Adam wants done, of
course. :)
The other thing is that I think that in addition to the per-package test
cases, it would be good to encourage updates to include at least a line
about what is of particular interest to test in _this_ release. Sometimes,
that's the same as the update Details, but not always.
--
Matthew Miller -- Fedora Project -- <mattdm(a)fedoraproject.org>
8:42 p.m.
On Tue, 2014-01-28 at 15:19 -0500, Matthew Miller wrote:
On Tue, Jan 28, 2014 at 10:39:32AM +0800, Mathieu Bridon wrote:
> > I could have sworn we already had something like this where bodhi would
> > add a link to a wiki page for test plan on a package if that wiki page
> > existed. I can't seem to find it now, so perhaps it was just something
> > we talked about, but never implemented.
> Nope, you're right. :)
> https://github.com/fedora-infra/bodhi/blob/develop/bodhi/model.py#L191
> For example, the test case pages for the package 'foo' need to be added
> to the 'Category:Package foo test cases' category.
Huh, lookit that. That's great! So I guess my suggestion changes to "make
that more visible".
Also my head is exploding a little bit, because it's yet another case of the
wiki doing a lot of different types of work.
That's because I came up with the design, and I'm an idiot monkey who
knows how wikis work and not much else. :P
Also, tbf, it's because we use the wiki as our TCMS (test case
management system - look, I'm using jargon to pretend I have some kind
of formal knowledge about my job!), and as long as we're doing that,
it's probably no *more* hideous to use wiki categories for this kind of
purpose as it is to weld on some kind of external method for
categorizing test cases in this way.
Using the wiki as a TCMS is of course a gross hack, but it comes with a
surprising number of advantages - see
https://fedoraproject.org/wiki/Tcms_Comparison . It's very
https://xkcd.com/1172/ , but goddamnit, my spacebar heating works. We're
not opposed to moving to some better in principle, but it would be a
moderate amount of work and we do at least want the 'better' thing to be
really and truly and concretely better and not just 'at least it's
actually called a TCMS' :P
I wonder if it would help both
of these things to move the test plans to live in package git (right next to
the future taskotron scripts?
Well, what would they look like, and how would you read them? What we're
calling a 'test plan' for these purposes is basically just a category.
Sure, we could stick some kind of metadata in some format in package git
that just included the list of test cases associated with that package,
but that doesn't seem like a fantastic design either, and it comes with
the rather substantial downside that you now need commit rights to a
package in order to write a test plan for it (or the co-operation of
someone who has those rights). Of the relatively few test plans we have
so far, most were written by QA folks, not package maintainers.
And, while clearly helpful, I think it suffers from a little bit of
tl;dr.
Maybe we could embed summaries of each right into the Bodhi page?
I'm not quite clear what you mean here - what's tl? Summaries of what?
The *user experience* here is indeed that you just see the test cases
right in the Bodhi page (or in whatever other interface you're using to
view the update - easy-karma, gooey-karma, whatever). The documents
describing how the system works and how to create test plans etc are
somewhat long-winded, yeah, in the latter case because I wrote it. :P
Are you saying you'd like a summary of how to create a test plan right
in the Bodhi page for an update? that seems a bit inappropriate.
The other thing is that I think that in addition to the per-package
test
cases, it would be good to encourage updates to include at least a line
about what is of particular interest to test in _this_ release. Sometimes,
that's the same as the update Details, but not always.
It would indeed be nice if maintainers did this, though of course it's
not relevant to the case we're currently considering (as the broken
content in the update was a mistake and the maintainer didn't even know
it was there, he'd hardly have been likely to highlight it for testing).
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
8:26 a.m.
On Tue, Jan 28, 2014 at 06:42:52PM -0800, Adam Williamson wrote:
Using the wiki as a TCMS is of course a gross hack, but it comes with
a
surprising number of advantages - see
Makes sense to me. I'm actually not opposed to using the wiki in this way,
but I think it highlights the need for _something else_ to be our
short-quick-easy documentation solution.
> I wonder if it would help both
> of these things to move the test plans to live in package git (right next to
> the future taskotron scripts?
Well, what would they look like, and how would you read them? What we're
They would be short text files. Possibly markdown. Something simple, though.
I hear you on the downside of having to have commit rights to the package in
order to work on test plans in git. That applies to the taskotron tests too,
though, doesn't it? Maybe we could use git submodules for this. (Where each
fedora package has a submodule consisting of its tests, both automated and
test plans.)
But like I said I'm not really opposed to continuing to use the wiki. Maybe
we just need an awareness campaign. (Some text in the bodhi.template from
fedpkg update would be a good start, I think. And on the New Update page in
the bodhi web interface. And definitely in
https://fedoraproject.org/wiki/Package_update_HOWTO -- I'll do that, as soon
as I have that coffee I didn't have when I said I was going to at the end of
the last message I sent.)
> And, while clearly helpful, I think it suffers from a little bit
of tl;dr.
> Maybe we could embed summaries of each right into the Bodhi page?
I'm not quite clear what you mean here - what's tl? Summaries of what?
I mean the links to the test cases shown in bodhi. In the context of trying
to get better feedback in bodhi, it would be nice if there were a summary of
each test right in the bodhi screen (not just its title). Then, there could
be a little dropdown to choose between "didn't try this aspect",
"looked at
the thing in the summary; seemed okay" and "actually went through this full
test case". That would make it easy for people to be descriptive and honest
about what they actually tried.
Don't get me wrong -- the way it works currently is pretty awesome and just
making that more used would be great.
> The other thing is that I think that in addition to the
per-package test
> cases, it would be good to encourage updates to include at least a line
> about what is of particular interest to test in _this_ release. Sometimes,
> that's the same as the update Details, but not always.
It would indeed be nice if maintainers did this, though of course it's
not relevant to the case we're currently considering (as the broken
content in the update was a mistake and the maintainer didn't even know
it was there, he'd hardly have been likely to highlight it for testing).
Yes, in this case the general test plans would have covered it. Hopefully.
:)
--
Matthew Miller -- Fedora Project -- <mattdm(a)fedoraproject.org>
12:31 p.m.
On Wed, 2014-01-29 at 09:26 -0500, Matthew Miller wrote:
On Tue, Jan 28, 2014 at 06:42:52PM -0800, Adam Williamson wrote:
> Using the wiki as a TCMS is of course a gross hack, but it comes with a
> surprising number of advantages - see
Makes sense to me. I'm actually not opposed to using the wiki in this way,
but I think it highlights the need for _something else_ to be our
short-quick-easy documentation solution.
> > I wonder if it would help both
> > of these things to move the test plans to live in package git (right next to
> > the future taskotron scripts?
> Well, what would they look like, and how would you read them? What we're
They would be short text files. Possibly markdown. Something simple, though.
I hear you on the downside of having to have commit rights to the package in
order to work on test plans in git. That applies to the taskotron tests too,
though, doesn't it? Maybe we could use git submodules for this. (Where each
fedora package has a submodule consisting of its tests, both automated and
test plans.)
Yeah it does - there's an argument to be made that you need a lot more
(or, y'know, any at all) in the way of dev skills to write a taskotron
test than a manual test case, but still, you could be the world's
foremost expert on automated testing and not have provenpackager rights,
so it's still a problem. Submodules with broader access rights doesn't
seem like a bad design...but I guess if this all gets hooked up to
Bodhi, you should still need package commit privs to decide which
automated tests can cause an update to be rejected or whatever (or else
anyone can DoS a package, accidentally or intentionally, by adding a
broken automated test).
But like I said I'm not really opposed to continuing to use the
wiki. Maybe
we just need an awareness campaign. (Some text in the bodhi.template from
fedpkg update would be a good start, I think. And on the New Update page in
the bodhi web interface. And definitely in
https://fedoraproject.org/wiki/Package_update_HOWTO -- I'll do that, as soon
as I have that coffee I didn't have when I said I was going to at the end of
the last message I sent.)
OK, that makes sense - you're talking about 'publicising' this system in
the places *packagers* see when interacting with Bodhi. Absolutely, good
idea, no objections.
> > And, while clearly helpful, I think it suffers from a
little bit of tl;dr.
> > Maybe we could embed summaries of each right into the Bodhi page?
> I'm not quite clear what you mean here - what's tl? Summaries of what?
I mean the links to the test cases shown in bodhi. In the context of trying
to get better feedback in bodhi, it would be nice if there were a summary of
each test right in the bodhi screen (not just its title).
Ahh, I see. I'd worry that would make the Bodhi pages a bit long, with
the way test cases are laid out - they tend to be vertical space hungry
- but it'd be interesting to mock it up and see.
Then, there could
be a little dropdown to choose between "didn't try this aspect",
"looked at
the thing in the summary; seemed okay" and "actually went through this full
test case". That would make it easy for people to be descriptive and honest
about what they actually tried.
*This* bit is part of the Glorious Bodhi 2.0 Vision, yep - if you go and
find that post from forever ago in the archives, it describes something
very like this. It is Bodhi 2.0-dependent, though, I think.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
5:32 p.m.
On Fri, 2014-01-24 at 15:35 -0700, Stephen John Smoogen wrote:
Looking at the number of people who respond to the qa list at times..
I am going to say there are probably 6-10 active testers during
non-release times. It comes and it goes, but that is about the number
who seem active at least on lists (and it seems to be that way going
over archives for the last couple of years.) So any policy would need
to take into account of that limitation.
QA list posters isn't a good proxy for feedback submitters, but you can
get fairly good numbers on active update feedback submitters from Bodhi
itself. Mike Ruckman has taken over the role of posting the 'heroes of
Fedora' numbers each quarter and Fedora release, which include these
statistics:
http://roshi.fedorapeople.org/heroes-of-fedora-quarter-3-2013-statistics....
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
5:24 p.m.
On Fri, 2014-01-24 at 22:36 +0100, Michael Schwendt wrote:
That's why I think there's reason to be very careful and
sometimes even
prefer a +0 (with a comment) over a very early over-ambitious +1.
And guess what happens in non-critpath updates after 7 days and _no_
feedback. Packagers push the update manually. Sometimes with broken
deps. Sometimes the testing starts no sooner than when the update arrives
in the stable updates repo and the first real user becomes the "guinea
pig".
Good point. Raises the question why an update that links so many
bugzilla
tickets can be marked stable automatically after a +3, which may be even
about a single bz ticket.
See, this is what happens when we have a fundamentally inadequate
process: an eternal tug-of-war between the tendency to prioritize
'safety' (in a very dumb and insufficiently granular way) and the
tendency to prioritize 'getting updates out' (in a very dumb and
insufficiently granular way). There are reasonable arguments in favour
of both sides.
I incline to the view that any time there is a situation like this -
where there are two alternative ways of doing something, both bad in
different ways, and roughly equally strong arguments on either side -
it's not a great use of anyone's time to keep tweaking things to one end
of the continuum or the other; _we need a better process_.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
5:29 p.m.
On Fri, 2014-01-24 at 22:36 +0100, Michael Schwendt wrote:
Good point. Raises the question why an update that links so many
bugzilla
tickets can be marked stable automatically after a +3, which may be even
about a single bz ticket.
Because that's how the maintainer configured it. (It is also the default
configuration, of course, but no-one is obliged to accept the default
configuration).
In Fedora we effectively define a *baseline* requirement for updates of
various types to go from testing to stable, and beyond that, we grant to
maintainers the power to choose the policy that will apply to their
updates (and, therefore, the responsibility to choose wisely).
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
12:22 p.m.
Adam Williamson wrote:
Yup, indeed. Of course, this is another area where we could improve
the
tooling: it doesn't seem like it'd be difficult for maintainers to be
allowed to set a minimum timeframe before their update goes stable, but
at present this isn't possible.
Why do we need to keep polishing that karma turd instead of just flushing it
away? Especially "karma automatism" is totally broken by design.
Again, hate to sound like a broken record, but it's just hard to
get
enthusiastic about trying to twiddle the edges of the process when the
process is fundamentally inadequate.
Oh yes, it is! So let's do away with it!
Kevin Kofler
4:43 a.m.
On Fri, Jan 24, 2014 at 11:14:50AM -0800, Adam Williamson wrote:
On Fri, 2014-01-24 at 19:26 +0100, Michael Schwendt wrote:
> > * That update made it out to the stable updates! In other words, the
> > draconian Update Policies that were enacted in a vain attempt to prevent
> > such issues from happening utterly failed at catching this bug.
>
> Those policies are not "draconian" enough [1]. On erroneous belief that
> a +1 from three different testers would mean that the update has seen
> enough testing, the test update has been published with the default karma
> threshold of +3. The testers have failed. It's too simple for testers to
> rush through the voting in bodhi without testing the updates
> painstakingly. "The faster the better" has lead to a fatal mistake in
> this case.
I think that's being unnecessarily harsh on the testers. It's not at all
obvious to anyone that you ought to test update/install of another
package in order to validate an update to selinux-policy-targeted .
Hell, I don't do that.
Doesn't / can't AutoQA (or whatever we're calling it these days) pick
up the new package, install it in a VM, and run through some automated
tests:
- Does Fedora still boot with this package added?
- Does GNOME still come up?
- Does yum still work?
At least the third one might have automatically found this bug.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
10:37 a.m.
On Sat, 2014-01-25 at 10:43 +0000, Richard W.M. Jones wrote:
> I think that's being unnecessarily harsh on the testers.
It's not at all
> obvious to anyone that you ought to test update/install of another
> package in order to validate an update to selinux-policy-targeted .
> Hell, I don't do that.
Doesn't
No, it doesn't.
/ can't AutoQA (or whatever we're calling it these days)
pick
up the new package, install it in a VM, and run through some automated
tests:
- Does Fedora still boot with this package added?
- Does GNOME still come up?
- Does yum still work?
I answered precisely that question a couple of days ago in the other
thread:
https://lists.fedoraproject.org/pipermail/devel/2014-January/194155.html
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
12:03 p.m.
Adam Williamson wrote:
The 'comment' field exists to allow people to express all
these things,
but as it's just a completely free-form text field, it's intrinsically
impossible to really base any programmatic stuff or even policy on it.
In theory maintainers could submit updates without using autokarma and
then keep a careful eye on the feedback and 'tend' their updates
manually, but I think it's pretty clear that in practice, this is not
what happens: maintainers really want to be able to use the karma system
as a 'helper', they want to farm out the evaluation process to Bodhi/the
karma system. But our current system is too stupid to handle this
perfectly, so we get these breakdowns.
It's not that they WANT to be able to do that, it's that the system is
rigged to encourage that broken practice. Autokarma (ab)use was much lower
before the enactment of the Update Policies. And really, autokarma needs to
just go away entirely. Having an intelligent being interpret the free-form
text field is the only way to make sane decisions (which also implies that
we should not arbitrarily impose any restrictions that, by their nature,
cannot take the free-form text into account).
With a more flexible karma system we have a *lot* of opportunity to
do
much cleverer stuff. We can provide presets for all the above different
things that are currently commonly expressed via +1 or -1 with a
comment. This opens up possibilities at two different levels: the distro
policy level, and the packager level. We can make the distro policy much
more fine-grained, if we want to - we can require certain of the 'karma
types' to be available in all updates, and for instance, block any
update where X people pull the 'it's completely busted' or 'it
introduces a security vulnerability' cord, regardless of how much
broadly-categorized 'positive' karma it has. At the packager level, the
packager gets the freedom to define a much more fine-grained policy for
when they're happy that updates to their package are 'good to go', but
they still don't have to sit there reading the emails and manually
interpreting what people have written. You get to define the policy that
makes the most sense for your package, within the confines of the
distro-wide policy - if you have a good package-specific test suite, you
can say to the auto-karma system 'don't send this update out until at
least one person sets the "I ran the test suite and it passed" karma
property.
To me, this just screams "OVERENGINEERED!!!". :-(
You are introducing a lot of complexity, that will ultimately always only be
an approximation to reality. You just cannot reliably quantify all the
details. E.g. "this introduces a regression, but the regression is that you
sometimes have to click OK twice (instead of once) to format a 5.25" floppy
in KFloppy, whereas it fixes a critical bug in KDE Plasma Desktop where all
your data was sent to the NSA and then securely wiped from your hard disk".
:-) (Yes, of course that is an exaggerated example. ;-) I sure hope we don't
have bugs like that. ;-) ) If you only have "fixes a bug, but introduces a
regression" as a feedback type, you probably end up making the wrong
decision. If you try to get more fine-grained, then you again need numbers
to quantify the severity of one issue vs. the other, and those will
inherently be subjective. (Users always think THEIR bug is the end of the
world whereas regressions that don't affect them are entirely unimportant.)
The complexity also means there are a lot more arbitrary parameters to deal
with. The current stable/unstable thresholds are already bad enough, and
often end up set to the wrong value. A decision process tends to be the
worse the more arbitrary parameters it needs.
And of course, more complexity means less transparency. It becomes harder
and harder to understand what really needs to be satisfied for an update to
be allowed to go stable.
So I can only advocate for the KISS approach: The update is stable when the
maintainer says so, period. We do not need any karma, be it a simple ±1 or a
long (and inherently non-exhaustive) list of all the things that can
possibly happen. So let's just do away with the 3 radio buttons and use a
free-form text field only. We just need somebody able to read comments, and
that is what we have maintainers for!
Kevin Kofler
3748
days inactive
3754
days old
150 comments
28 participants
participants (28)
-
Adam Williamson -
Bruno Wolff III -
Chris Murphy -
Colin Walters -
Daniel J Walsh -
David Beveridge -
Dominick Grift -
drago01 -
Eric Sandeen -
Fabian Deutsch -
Heiko Adams -
Ian Malone -
Josh Stone -
Kevin Fenzi -
Kevin Kofler -
Konstantin Ryabitsev -
Mathieu Bridon -
Matthew Miller -
Michael Schwendt -
Rahul Sundaram -
Ralf Corsepius -
Reindl Harald -
Richard W.M. Jones -
Sergio Pascual -
Simo Sorce -
Stephen John Smoogen -
Sérgio Basto -
Tomasz Torcz