update (and questions) on "urgent fixes urgently" solution

[Matthew Miller]

See https://fedorahosted.org/rel-eng/ticket/5886 for background. We
reccently had a meeting with release engineering about actually
implementing this. See the notes in comment #33.

This resulted in several questions for the security team.

First: we agreed that we don't want just any update using this -- it
should be for updates that are both critical and urgent. (Remote root
ssh exploit, anyone? *knock on wood*) So, we think it makes sense for
updates to be selected for this process by the security team. The
question is: who should that be exactly, and how should that group be
defined. (An existing FAS group? A special new one? Something else?)

Second, a more technical matter. In order for an update to be treated
specially, it will need something special in Bodhi. One relatively easy
way is to have the security person (from question #1) submit the
update, instead of the packager. (Or, possibly, in *addition* to the
packager submitting the update for the regular repositories.) Since
this is something that hopefully will happen a couple of times a year
(or less!), is that workable? Or, would it be better for the packager
to submit the update as normal, but provide some button or checkbox
available to the security team to escalate the update to the urgent
repo?

Third, any other questions or concerns?

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[P J P]

Hello,

>  First: we agreed that we don't want just any update using this --
> it
>  should be for updates that are both critical and urgent. (Remote root
>  ssh exploit, anyone? *knock on wood*) So, we think it makes sense for
>  updates to be selected for this process by the security team. The
>  question is: who should that be exactly, and how should that group be
>  defined. (An existing FAS group? A special new one? Something else?) 

  FST members could handle this.


>  Second, a more technical matter. In order for an update to be
> treated
>  specially, it will need something special in Bodhi. One relatively easy
>  way is to have the security person (from question #1) submit the
>  update, instead of the packager. (Or, possibly, in *addition* to the
>  packager submitting the update for the regular repositories.) Since
>  this is something that hopefully will happen a couple of times a year
>  (or less!), is that workable? Or, would it be better for the packager
>  to submit the update as normal, but provide some button or checkbox
>  available to the security team to escalate the update to the urgent
>  repo? 

  I think it should be okay for a packager to push such updates. Important
is to push them in time.


>  Third, any other questions or concerns? 
  Is this going to be a separate repository? How do we handle the testing/qa part
on urgent basis?


---  -P J P
http://feedmug.com

[Matthew Miller]

On Thu, Feb 25, 2016 at 12:05:19PM +0000, P J P wrote:
>  > Third, any other questions or concerns?
>    Is this going to be a separate repository? How do we handle the
>  testing/qa part on urgent basis? 
Yeah, the idea is a separate repository which:

a) can be checked more frequently
b) doesn't need to go through the very slow "push" process, which now
   takes several hours best case — for shellshock, it took something
   like 18 hours, although that was partly due to a series of disasters
c) will be provided through our normal web cache mechanism instead of
   mirrored, to reduce delay there

QA would happen as normal through bodhi. Getting that organized and
orchestrated is separate-but-related ticket
<https://fedorahosted.org/fesco/ticket/1278>;.

In the case of previous high-urgency updates, we had plenty of people
pulling all-nighters to get the packages built and tested, but then
_after_ that, it took a whole day (or more, depending on mirrors) to
actually get the tested fix to users, which was frustrating.



-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[P J P]

>  On Thursday, 25 February 2016 9:03 PM, Matthew Miller wrote:
>  Yeah, the idea is a separate repository which:
>  QA would happen as normal through bodhi. Getting that organized and
>  orchestrated is separate-but-related ticket
>  <https://fedorahosted.org/fesco/ticket/1278>;. 
  I see. I guess we'd need some check to ensure the same fix/package
is not pushed to -stable/other repositories later. (thinking aloud)

>  In the case of previous high-urgency updates, we had plenty of
> people
>  pulling all-nighters to get the packages built and tested, but then
>  _after_ that, it took a whole day (or more, depending on mirrors) to
>  actually get the tested fix to users, which was frustrating. 


  Yes, I can imagine.


Thank you.
---
  -P J P
http://feedmug.com

[Matthew Miller]

On Thu, Feb 25, 2016 at 05:50:49PM +0000, P J P wrote:
>  > On Thursday, 25 February 2016 9:03 PM, Matthew Miller wrote:
>  > Yeah, the idea is a separate repository which:
>  > QA would happen as normal through bodhi. Getting that organized and
>  > orchestrated is separate-but-related ticket
>  > <https://fedorahosted.org/fesco/ticket/1278>;.
>    I see. I guess we'd need some check to ensure the same fix/package
>  is not pushed to -stable/other repositories later. (thinking aloud) 
Actually, it's the opposite. The idea is to push the package in both
channels. It's okay if there are duplicate sources — dnf/yum/packagekit
won't care. In fact, once a few days (weeks?) have passed, and the
update is in the normal update repositories on mirrors everywhere, we'd
*remove* it from the critical-urgent repo, to keep the size of that
small.


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader