developing a SOP for critical updates (the Fedora Bat Signal)

[Matthew Miller]

FESCo has asked me to bring this back up, and this seems like the right
place for it. See https://fedorahosted.org/fesco/ticket/1278, and the
very basic outline of a SOP from Paul Frields at
https://fedoraproject.org/wiki/User:Pfrields/Critical_security_update_SOP.

To paste from the ticket:

 We need to have responders for

    coordination (it helps when one person has the "incident lead"
       baton; can be passed around as needed)

    communications (drafting and sending community messages; email,
       web, social media)

    package fixing (ideally package maintainer is security expert,
       second best is package maintainer + security expert, third is security
       expert with provenpackager privileges or assistance from someone who
       has them, or last resort, provenpackager alone)

    quality assurance (again, ideally someone with security expertise
       to advise and coordinate, but fast widespread testing at all levels
       helps)

    release engineering (lots of work getting an update out as an
       exception to normal flow) 

 and the ability to get at least one person in each role out of bed in
 the event of an emergency. 

I expect that in many cases, there are also roles like "communication
with $otherproject security team", and possible handoff from whereever
we learned about the vulnerability.


Security Team, are you interested in helping develop this procedure
(and putting it somewhere so we know what to do in a fire drill)?



-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Eric Christensen]

On Wednesday, October 21, 2015 02:19:33 PM Matthew Miller wrote:
>  FESCo has asked me to bring this back up, and this seems like the
> right
>  place for it. See https://fedorahosted.org/fesco/ticket/1278, and the
>  very basic outline of a SOP from Paul Frields at
>  https://fedoraproject.org/wiki/User:Pfrields/Critical_security_update_SOP. 
We've been talking about this [informally] for the past few weeks and I think 
what Paul has written up makes a lot of sense.  There are a few pieces that 
need to be moved around the board or approved by someone above my paygrade 
(you?).

>      coordination (it helps when one person has the "incident
> lead"
>         baton; can be passed around as needed) 
Traditionally this has been Red Hat Product Security.  I say this because they 
are the ones that are handling incoming and are notified of a security threat.  
The problem with letting Product Security handle coordination is that they 
don't really care about Fedora (well, don't care isn't really the correct 
words to use but Fedora really doesn't have the proper tooling for doing what 
Red Hat does with staging security fixes).

>      communications (drafting and sending community messages; email,
>         web, social media) 
Does Fedora have a PIO?

>      package fixing (ideally package maintainer is security expert,
>         second best is package maintainer + security expert, third is
>  security expert with provenpackager privileges or assistance from someone
>  who has them, or last resort, provenpackager alone) 
We have a couple of provenpackagers on our team.  I suspect we'd be able to 
pull someone in from the "civilian" side to help as needed.

>      quality assurance (again, ideally someone with security
> expertise
>         to advise and coordinate, but fast widespread testing at all levels
>         helps) 
This is another piece of the puzzle.  We'd need to be able to pull in someone 
from QA to verify the fixes.

>      release engineering (lots of work getting an update out as an
>         exception to normal flow) 
This is why I, and others, have argued for a separate channel in which to send 
out high-priority security fixes.  We shouldn't have to run around finding the 
correct person to do a special push.  We should be able to dump them into the 
security channel and make it available sooner.

>   and the ability to get at least one person in each role out of bed
> in
>   the event of an emergency. 
Or qualified folks around the globe.

>  I expect that in many cases, there are also roles like
> "communication
>  with $otherproject security team", and possible handoff from whereever
>  we learned about the vulnerability. 
I suspect in all cases as there will likely be few issues that affect 
Fedora/EPEL that doesn't affect RHEL.

>  Security Team, are you interested in helping develop this procedure
>  (and putting it somewhere so we know what to do in a fire drill)? 
Yep.

--Eric

[Matthew Miller]

On Wed, Oct 21, 2015 at 03:45:23PM -0400, Eric Christensen wrote:
>  > FESCo has asked me to bring this back up, and this seems like
> the right
>  > place for it. See https://fedorahosted.org/fesco/ticket/1278, and the
>  > very basic outline of a SOP from Paul Frields at
>  > https://fedoraproject.org/wiki/User:Pfrields/Critical_security_update_SOP.
>  We've been talking about this [informally] for the past few weeks and I think 
>  what Paul has written up makes a lot of sense.  There are a few pieces that 
>  need to be moved around the board or approved by someone above my paygrade 
>  (you?). 
Sure — probably Council, and I can help with that.


>  >     coordination (it helps when one person has the
> "incident lead"
>  >        baton; can be passed around as needed)
>  
>  Traditionally this has been Red Hat Product Security.  I say this
>  because they are the ones that are handling incoming and are notified
>  of a security threat. The problem with letting Product Security
>  handle coordination is that they don't really care about Fedora
>  (well, don't care isn't really the correct words to use but Fedora
>  really doesn't have the proper tooling for doing what Red Hat does
>  with staging security fixes). 
Yeah, I think, traditionally, RH product security has had their own
coordiation, and Fedora tries to follow along as best we can.
Hopefully, having a Fedora-focused coordination role will actually make
that easier for the RH side too, because it'll be clear from _that_
side who to coordinate with and how.


>  >     communications (drafting and sending community messages;
> email,
>  >        web, social media)
>  Does Fedora have a PIO? 
I don't know because I don't know what that means. :)


>  This is why I, and others, have argued for a separate channel in
>  which to send out high-priority security fixes. We shouldn't have to
>  run around finding the correct person to do a special push. We should
>  be able to dump them into the security channel and make it available
>  sooner. 
+1. That's https://fedorahosted.org/rel-eng/ticket/5886.


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader