New bodhi release in production

[Kevin Kofler]

Adam Williamson wrote:
> To me, that reads more like a problem with the update submission system
> than anything. I'd like to see far fewer restrictions on it (just like
> I'd like for koji), so you could edit the existing update to add your
> packages. This same issue exists even without feedback requirements.

Editing isn't what I want though, I want to have a newer update in testing 
while keeping the old one not obsolete because it should go out to stable 
ASAP. Bodhi only allows this if it's already queued for stable when the new 
update is submitted, otherwise it automatically obsoletes the old one.

>> The non-critpath packages still have a hard +1 requirement.
> 
> Indeed, but that's not a difficult standard to meet.

That's what I dispute. Especially if you expect people to actually meet the 
testing criteria for a +1. (It's easy and fast to just "swap +1 votes" or 
something like that, but it defeats the purpose of the process.)

>>  where 1 must be a proventester (which is really
>> ridiculous, why isn't a proventester enough?).
> 
> A single tester can often miss an issue, it's good to have at least two
> pairs of eyes.

Only 2 testers can often miss an issue, it's good to have at least 3 pairs 
of eyes.
Only 3 testers can often miss an issue, it's good to have at least 4 pairs 
of eyes.
Only 4 testers can often miss an issue, it's good to have at least 5 pairs 
of eyes.
… (ad infinitum)

Where do we stop? NO amount of testers will catch ALL issues.

In addition, the maintainer certainly won't submit something he thinks is 
broken, so the proventester is already a second pair of eyes.

Each time we decide that one person is not enough (but n are needed), we 
multiply the required manpower to do any work by a factor n! This is not 
something to be taken lightly. We DO NOT HAVE the manpower for this kind of 
QA. It takes VERY LONG to get any kind of positive karma for anything. 
(Negative one can sometimes be gotten quite fast, just screw up blatantly 
enough. ;-) )

And FWIW, I also dislike the fact that our QA process focuses almost 
exclusively on testing (well, there will be AutoQA too soon, but the manual 
checks will still be limited to just testing). I have found proofreading 
(even self-proofreading, but proofreading by another person can be even more 
effective) to be a much more effective form of QA than testing. I have found 
many issues when proofreading changes (both mine and other people's, and 
both for RPM specfiles and actual code) which would almost certainly never 
have been caught through testing (but were still real issues). I always 
proofread my changes before I commit them and I also proofread the specfile 
changes done by comaintainers. IMHO, that should already be sufficient 
ground for validating the resulting update, but according to our process I'm 
not supposed to +1 an update I have "only" proofread the changes of and not 
also tested. (That also reminds me of Knuth's "Beware of bugs in the above 
code; I have only proved it correct, not tried it.". :-) ) It often takes 
much less time to proofread the spec changes than to do testing with any 
kind of reasonable coverage (especially for something like KDevelop), and it 
is much more likely to find issues (because you just can't test everything: 
For example, how many testers actually check for unowned directories? A 
trained proofreader catches these quite reliably, especially if the change 
being proofread is the one making the directory unowned. And do you really 
think you can exhaustively test ALL the features of a powerful application? 
Proofreading can evidence that you broke one (e.g. by dropping a required 
BuildRequires), or that you made only trivial changes extremely unlikely to 
break anything at all). Also a FYI: Often it is also useful to have a look 
at the build.log in addition to the specfile, that will evidence complaints 
by the package's build system (CMake, autoconf-generated configure or 
whatever) about missing optional build requirements (and thus missing 
features). Those are again issues which testing generally does not catch.

> AFAIK, as long as we keep pushing builds through to updates-testing while
> the main repo (which is filled from the dist-f14 tag) is frozen for RC
> composes, there isn't really a problem, as you can keep pushing fixes into
> updates-testing, build things on top of other things there, however you
> like.

Well, now that we basically force everything through testing (though there 
can theoretically still be stuff which gets queued for stable before 
reaching testing if karma comes really quickly), it is not such a big 
problem anymore, but before, we had the paradoxical situation where urgent 
fixes queued directly for stable did not become available whereas 
unimportant stuff went out to testing just fine. :-/ Banning direct stable 
pushes entirely strikes me as the entirely wrong solution for this issue, 
basically solving the wrong problem.

> except that systemd-sysvinit and upstart-sysvinit conflict, but that's
> fine as there's no way you can actually cause upstart-sysvinit to be
> selected for installation, it only gets pulled in because mash pulls in
> everything that can potentially satisfy dependencies, and they both
> provide the 'sysvinit-userspace' dependency

Why do we even still ship upstart at all? It should get Obsoleted by systemd 
and EOLed.

> As I said above, as long as you can fix dependency issues in
> updates-testing, the fact that the 'stable' repo is frozen shouldn't
> really be an issue, right?

Well, it is, because the stable repo is what shows up in the Branched 
reports, what people will be using by default etc. Now I know that we've 
started enabling updates-testing by default for prereleases with F13, but 
that strikes me as an awfully bad idea, because updates-testing is like the 
Red Pill: once you're on it, you're stuck with it! We should NEVER enable 
updates-testing by default for ANY user. This is again a completely wrong 
solution which misses the essence of the problem.

        Kevin Kofler