I think we did a pretty good job in responding to CVE-2014-0160, but there's also room for improvement.
One particular need is the ability to get in touch with owners of core components, or if they are not available, provenpackagers with particular security expertise -- and in either case, also _testers_ with a security background.
Maybe we need to have some sort of (opt-in) Fedora Bat Signal for extra-critical and urgent security issues in core packages. We would promise not to use it unless the internet were actually on fire, as it appears to be in this case, and then have (escrowed somewhere?) private 24/7 contact information (phone numbers, SMS).
What do you think? Anyone interested in developing this idea further?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 4/8/14, 8:11, Matthew Miller wrote:
Maybe we need to have some sort of (opt-in) Fedora Bat Signal for extra-critical and urgent security issues in core packages. We would promise not to use it unless the internet were actually on fire, as it appears to be in this case, and then have (escrowed somewhere?) private 24/7 contact information (phone numbers, SMS).
What do you think? Anyone interested in developing this idea further?
This is a great idea and would really be valuable in the types of situations we had yesterday. I ended up jumping on Twitter/G+ to spread the news about package updates. Having a team dedicated to the fixing and the communications would help keep people better informed.
With that said, I'd be glad to help. I'm sure we can come up with some technologies and processes relatively quickly. Something as simple as a call to join #fedora-eoc (emergency operations center) might be a good stopgap.
- -- Major Hayden
Cant tell if I did reply all on my reply. No need meeting for srt we caught up offline.
-----Original Message----- From: Major Hayden [major@mhtx.net] Received: Tuesday, 08 Apr 2014, 2:34PM To: security@lists.fedoraproject.org Subject: Re: Developing a security Bat Signal?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 4/8/14, 8:11, Matthew Miller wrote:
Maybe we need to have some sort of (opt-in) Fedora Bat Signal for extra-critical and urgent security issues in core packages. We would promise not to use it unless the internet were actually on fire, as it appears to be in this case, and then have (escrowed somewhere?) private 24/7 contact information (phone numbers, SMS).
What do you think? Anyone interested in developing this idea further?
This is a great idea and would really be valuable in the types of situations we had yesterday. I ended up jumping on Twitter/G+ to spread the news about package updates. Having a team dedicated to the fixing and the communications would help keep people better informed.
With that said, I'd be glad to help. I'm sure we can come up with some technologies and processes relatively quickly. Something as simple as a call to join #fedora-eoc (emergency operations center) might be a good stopgap.
- -- Major Hayden
-- security mailing list security@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/security
On 08/04/14 14:57, Mark Cox wrote:
Cant tell if I did reply all on my reply. No need meeting for srt we caught up offline.
-----Original Message----- From: Major Hayden [major@mhtx.net] Received: Tuesday, 08 Apr 2014, 2:34PM To: security@lists.fedoraproject.org Subject: Re: Developing a security Bat Signal?
On 4/8/14, 8:11, Matthew Miller wrote:
Maybe we need to have some sort of (opt-in) Fedora Bat Signal for extra-critical and urgent security issues in core packages. We would promise not to use it unless the internet were actually on fire, as it appears to be in this case, and then have (escrowed somewhere?) private 24/7 contact information (phone numbers, SMS).
What do you think? Anyone interested in developing this idea further?
This is a great idea and would really be valuable in the types of situations we had yesterday. I ended up jumping on Twitter/G+ to spread the news about package updates. Having a team dedicated to the fixing and the communications would help keep people better informed.
With that said, I'd be glad to help. I'm sure we can come up with some technologies and processes relatively quickly. Something as simple as a call to join #fedora-eoc (emergency operations center) might be a good stopgap.
-- security mailing list security@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/security
-- security mailing list security@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/security
We should be calling it the hat signal though, preferably with three distinct colours. Red for RHEL issues. Blue for Fedora. And green, purple, orange and blue for CentOS. ;-p
Regards,
Tristan
Hmm ignore that, touchdown replied to wrong message sigh. Mark
-----Original Message----- From: Mark Cox [mjc@redhat.com] Received: Tuesday, 08 Apr 2014, 2:57PM To: security@lists.fedoraproject.org, major@mhtx.net Subject: RE: Developing a security Bat Signal?
Cant tell if I did reply all on my reply. No need meeting for srt we caught up offline.
-----Original Message----- From: Major Hayden [major@mhtx.net] Received: Tuesday, 08 Apr 2014, 2:34PM To: security@lists.fedoraproject.org Subject: Re: Developing a security Bat Signal?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 4/8/14, 8:11, Matthew Miller wrote:
Maybe we need to have some sort of (opt-in) Fedora Bat Signal for extra-critical and urgent security issues in core packages. We would promise not to use it unless the internet were actually on fire, as it appears to be in this case, and then have (escrowed somewhere?) private 24/7 contact information (phone numbers, SMS).
What do you think? Anyone interested in developing this idea further?
This is a great idea and would really be valuable in the types of situations we had yesterday. I ended up jumping on Twitter/G+ to spread the news about package updates. Having a team dedicated to the fixing and the communications would help keep people better informed.
With that said, I'd be glad to help. I'm sure we can come up with some technologies and processes relatively quickly. Something as simple as a call to join #fedora-eoc (emergency operations center) might be a good stopgap.
- -- Major Hayden
-- security mailing list security@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/security
-- security mailing list security@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/security
On Tue, Apr 08, 2014 at 08:34:26AM -0500, Major Hayden wrote:
This is a great idea and would really be valuable in the types of situations we had yesterday. I ended up jumping on Twitter/G+ to spread the news about package updates. Having a team dedicated to the fixing and the communications would help keep people better informed.
With that said, I'd be glad to help. I'm sure we can come up with some technologies and processes relatively quickly. Something as simple as a call to join #fedora-eoc (emergency operations center) might be a good stopgap.
I created https://fedorahosted.org/fesco/ticket/1278 to help track this idea. It's more a security SIG thing than FESCo, but I think it's important enough that it deserves tracking somewhere.
Copying from that:
We need to have responders for
* coordination (it helps when one person has the "incident lead" baton; can be passed around as needed)
* communications (drafting and sending community messages; email, web, social media)
* package fixing (ideally package maintainer is security expert, second best is package maintainer + security expert, third is security expert with provenpackager privileges or assistance from someone who has them, or last resort, provenpackager alone)
* quality assurance (again, ideally someone with security expertise to advise and coordinate, but fast widespread testing at all levels helps) release engineering (lots of work getting an update out as an exception to normal flow)
and the ability to get at least one person in each role out of bed in the event of an emergency.
On 04/09/2014 11:35 AM, Matthew Miller wrote:
- quality assurance (again, ideally someone with security expertise to advise and coordinate, but fast widespread testing at all levels helps) release engineering (lots of work getting an update out as an exception to normal flow)
You can forget including QA in this since maintainers dont provide the testing community with test cases so testers cant quickly through test cases for the affected package and provide the necessary karma.
JBG
On Wed, Apr 09, 2014 at 11:44:12AM +0000, "Jóhann B. Guðmundsson" wrote:
On 04/09/2014 11:35 AM, Matthew Miller wrote:
- quality assurance (again, ideally someone with security expertise to advise and coordinate, but fast widespread testing at all levels helps)
You can forget including QA in this since maintainers dont provide the testing community with test cases so testers cant quickly through test cases for the affected package and provide the necessary karma.
I would say the _exact opposite_. We need to emphasize building those test cases so they are there when needed.
It's also why I noted that someone with security expertise is helpful -- they can provide guidance on what to check. Since each security vulnerability is different, that's probably always going to be valuable. But you are right that having more test cases in advance would help our ability to respond quickly.
On 04/09/2014 12:14 PM, Matthew Miller wrote:
On Wed, Apr 09, 2014 at 11:44:12AM +0000, "Jóhann B. Guðmundsson" wrote:
On 04/09/2014 11:35 AM, Matthew Miller wrote:
- quality assurance (again, ideally someone with security expertise to advise and coordinate, but fast widespread testing at all levels helps)
You can forget including QA in this since maintainers dont provide the testing community with test cases so testers cant quickly through test cases for the affected package and provide the necessary karma.
I would say the_exact opposite_. We need to emphasize building those test cases so they are there when needed.
I agree with you but...
Few years back I initiated effort trying to improve reporting and testing and general efficiently and communication between reporters and maintainers and actually put some value in the karma process ( have reporters actually go through some testing process not just fire up the app and give it karma based on just that ).
That effort involved how to debug and how to test pages in the wiki as well as finding it's way into the feature process where it still is amongst other things.
At that time the total size of Fedora was 5k - 6k components ( now we are 14k - 15k in size ).
And at that time I had requested that it would be a part of packaging review process and a must for acceptance of packages in the distribution, which would have allowed us in QA to work with existing maintainers and slowly gradually play catchup with those existing components.
That did not fly with FESCo/FPC due to it being to much of a burden on potential maintainers and a must was changed to a should or rather it was optional to provide this and now several years later double in size I can tell without a shadow of a doubt that a zero maintainer has provided either proper debugging information for the components he maintained nor test cases.
We are precisely today at the same place with that process as I abandoned it after realizing that nobody would provide that information.
So unless you come up with a way for maintainers *themselves* to provide test cases and debugging information for the component *they* maintain, all this will is remain wishful thinking.
Now I want you to bear the above in mind that everytime that you make decisions in any governing body in Fedora that is responsible for making system wide decisions and you serve on, the devastation the outcome of your vote can lead to and make life more difficult for others in various service sub-community and the project workflows and result in lower quality of our distribution hence I ask of you to always thoroughly familiarise yourself with the topic at hand and what the outcome of it will be in the long run before casting your vote on it.
But you are right that having more test cases in advance would help our ability to respond quickly.
Not only that but provide smoother and more reliable transaction of packages to the hands of our end user bases.
JBG
On Wed, Apr 09, 2014 at 12:54:38PM +0000, "Jóhann B. Guðmundsson" wrote:
I would say the_exact opposite_. We need to emphasize building those test cases so they are there when needed.
I agree with you but... Few years back I initiated effort trying to improve reporting and testing and general efficiently and communication between reporters and maintainers and actually put some value in the karma process ( have reporters actually go through some testing process not just fire up the app and give it karma based on just that ).
Yeah. This is a hard problem. It's hard to get maintainers to do more work, and basically impossible to *mandate* that volunteers do anything. I know you are familiar with that :).
The package review process is already very heavyweight, and I don't think we can accomplish anything by adding more to it. We should try to find other points where we can make maintainers aware of the importance and benefits of setting these test cases up in advance, and encourage/motivate/reward them for doing so.
I'm not sure if it helps to make this part of the "bat-signal" idea, but I'm certainly interested in supporting it however I can. The easier we can make it, the better. And the more *visible* we can make it, the better. I think a lot of package maintainers aren't even aware. Maybe there is something that can be done with the taskotron work. I had some ideas on this last month https://lists.fedoraproject.org/pipermail/devel/2014-January/194856.html. I didn't really follow up, but it's still on my mind and I'd love more ideas, action, whatever.
Now I want you to bear the above in mind that everytime that you make decisions in any governing body in Fedora that is responsible for making system wide decisions and you serve on, the devastation the outcome of your vote can lead to and make life more difficult for others in various service sub-community and the project workflows and result in lower quality of our distribution hence I ask of you to always thoroughly familiarise yourself with the topic at hand and what the outcome of it will be in the long run before casting your vote on it.
I certainly do try to put exactly this thought into all of the decisions and votes I make. And I think everyone in FESCo and other Fedora leadership positions takes their role seriously as well. We are all human and sometimes make mistakes, or make judgments without realizing some detail we should have, but when there are real problems, we can always revisit. Sometimes it is a painful process and not as fast as everyone would like, or not as careful as others would, and not everything that should get done gets done as quickly as would be ideal, but on the whole, I think we work things out pretty well (*points to quote in signature*).
2014-04-08 15:11 GMT+02:00 Matthew Miller mattdm@fedoraproject.org:
I think we did a pretty good job in responding to CVE-2014-0160, but there's also room for improvement.
One particular need is the ability to get in touch with owners of core components, or if they are not available, provenpackagers with particular security expertise -- and in either case, also _testers_ with a security background.
Maybe we need to have some sort of (opt-in) Fedora Bat Signal for extra-critical and urgent security issues in core packages. We would promise not to use it unless the internet were actually on fire, as it appears to be in this case, and then have (escrowed somewhere?) private 24/7 contact information (phone numbers, SMS).
I suppose this is mainly playing devil's advocate...
Looking back, how many times in the past years would we have used that signal? Once in 3 years? 5 years? If we now collect the contact information and volunteers, is it at all likely that the information will still be correct and relevant by the time we need to use it again? Mirek
On Thu, Apr 10, 2014 at 03:55:17PM +0200, Miloslav Trmač wrote:
Looking back, how many times in the past years would we have used that signal? Once in 3 years? 5 years? If we now collect the contact information and volunteers, is it at all likely that the information will still be correct and relevant by the time we need to use it again?
Good question. I think "at least once" is sufficient answer, though. :)
Maybe the system could come with a reminder to keep info current in some way?
On 10/04/14 14:58, Matthew Miller wrote:
On Thu, Apr 10, 2014 at 03:55:17PM +0200, Miloslav Trmač wrote:
Looking back, how many times in the past years would we have used that signal? Once in 3 years? 5 years? If we now collect the contact information and volunteers, is it at all likely that the information will still be correct and relevant by the time we need to use it again?
Good question. I think "at least once" is sufficient answer, though. :)
Maybe the system could come with a reminder to keep info current in some way?
I think the most important thing is to keep not only the maintainers informed, but also our general users. Including giving them. mitigation advice and explaining exactly what the problem is or was.
Regarding the OpenSSL issue. This was such a serious security breach, even admins should have been given advice. I would really like to see upstream investigate too, how this breach could occur in the first place. Who made those commits and what review they have for commits. I had a look around to see what upstream does or where to get/see commits. Not exactly easy to find these things out.
Another issue this raises is, now that everyone uses github, what security record does github have ? Can they be influenced in any way by government agencies/departments or other groups ? Who vets committers or commits to projects.
With regards to our handling of the issue, I thought it was great Robyn sent out emails on the announce list. Maybe something we should have done is make a prominent security section for notices on the fp.o site.
I must also say, the response time was quite good too, looking at package build times, compared to the time when I was informed of the issue. The only problem were the mirrors not syncing up fast enough, which makes me wonder if we should dump security fixes into a sub-directory in updates, which mirrors could sync up faster.
Just a few thoughts here.
Regards,
Tristan
On Thu, Apr 10, 2014 at 03:12:41PM +0100, Tristan Santore wrote:
Maybe the system could come with a reminder to keep info current in some way?
I think the most important thing is to keep not only the maintainers informed, but also our general users. Including giving them. mitigation advice and explaining exactly what the problem is or was.
Yes, this would be the role of the communications team outlined in the proposal. When it is crunch time, it is *incredibly* important to have people other than the active responders doing this, because a) it frees them up to concentrate on getting fixes out and b) they're probably frazzled enough to make it hard to communicate clearly.
I must also say, the response time was quite good too, looking at package build times, compared to the time when I was informed of the issue. The only problem were the mirrors not syncing up fast enough, which makes me wonder if we should dump security fixes into a sub-directory in updates, which mirrors could sync up faster.
Yes. https://fedorahosted.org/rel-eng/ticket/5886
Help wanted. :)
security@lists.fedoraproject.org
-
Jóhann B. Guðmundsson -
Major Hayden -
Mark Cox -
Matthew Miller -
Matthew Miller -
Miloslav Trmač -
Tristan Santore