(Josh Bressers suggested I send my questions here rather than asking him or someone else directly)
Yesterday you folks released an enormous number of security updates. While I could selfishly complain about it being done on a Wednesday, my real issues are the following:
- it seems deliberate that the same alert ID tag was reused (FEDORA-2008-1435 and FEDORA-2008-1535), it would seem to be a bit confusing to refer to multiple alerts with the same ID, take a peek at:
to see what I mean.
- those were all related to the same gecko vulnerabilites, which is what (I presume) motivated reusing the same IDs, but at least one (perhaps two, I can't remember for sure) of those, ruby-gnome2 also fixed a separate CVE that was unrelated to the mozilla pile
- How is it that so many packages were affected by these mozilla vulns? Are they statically linked? Reusing the code? Have very restrictive dynamic library version numbers? It just seems that a vulnerability in a component shouldn't necessarily have this kind of cascading effect.
- Overall, we have been noticing a decline in the quality of Fedora security alerts. They are often missing basic information about what bug they are fixing (other than perhaps a reference to bugzilla, sometimes a link to the CVE). I think if you read a lot of those alerts as if you were just a plain old user, you would find that some provide very little useful information to try and determine what problem is being fixed. I can provide examples if necessary. Is there something that can be done to standardize the format a bit?
thanks!
jake
On Thu, 14 Feb 2008 08:25:19 -0700 Jake Edge jake@lwn.net wrote:
- those were all related to the same gecko vulnerabilites, which is
what (I presume) motivated reusing the same IDs, but at least one (perhaps two, I can't remember for sure) of those, ruby-gnome2 also fixed a separate CVE that was unrelated to the mozilla pile
- How is it that so many packages were affected by these mozilla
vulns? Are they statically linked? Reusing the code? Have very restrictive dynamic library version numbers? It just seems that a vulnerability in a component shouldn't necessarily have this kind of cascading effect.
Here is what happens. There are a /ton/ of packages in Fedora that build against gecko libs, or otherwise link to them. In Fedora 8, the gecko lib location would change every time gecko is rebuilt. So in order for these packages to continue working, they have to be rebuilt for the new location. So when a gecko security issue happens, a big pile of packages have to be rebuilt so that they'll work with the new gecko. In order to ensure that they all get pushed at the same time, all the builds are attached to a single update request in our update tool, bodhi. See https://admin.fedoraproject.org/updates/F8/FEDORA-2008-1535
As for ruby-gnome2's other CVE fix, that was released earlier in a different update, https://admin.fedoraproject.org/updates/F8/FEDORA-2007-4216
- Overall, we have been noticing a decline in the quality of Fedora
security alerts. They are often missing basic information about what bug they are fixing (other than perhaps a reference to bugzilla, sometimes a link to the CVE). I think if you read a lot of those alerts as if you were just a plain old user, you would find that some provide very little useful information to try and determine what problem is being fixed. I can provide examples if necessary. Is there something that can be done to standardize the format a bit?
Recently the Fedora project granted the Security team the task of reviewing all updates that are to go out tagged as security. It's the Security's responsibility to ensure that the messaging is correct. This is a fairly recent change so it may take a little bit to start to notice an overall trend back into the consistent messaging. Prior to this, all Fedora maintainers were able to generate whatever message they wanted to for security updates.
On Thu, 14 Feb 2008, Jesse Keating wrote:
There are a /ton/ of packages in Fedora that build against gecko libs, or otherwise link to them. In Fedora 8, the gecko lib location would change every time gecko is rebuilt.
This is rather silly, isn't it?
--Pavel Kankovsky aka Peak [ Boycott Microsoft--http://www.vcnet.com/bms ] "Resistance is futile. Open your source code and prepare for assimilation."
On Sun, 17 Feb 2008 19:07:57 +0100 (CET) Pavel Kankovsky peak@argo.troja.mff.cuni.cz wrote:
This is rather silly, isn't it?
I didn't say that it wasn't. Blame mozilla.org. They're "fixing" it somewhat with xulrunner though.
Hi Jake,
On Thu, 2008-02-14 at 08:25 -0700, Jake Edge wrote:
(Josh Bressers suggested I send my questions here rather than asking him or someone else directly)
Yesterday you folks released an enormous number of security updates. While I could selfishly complain about it being done on a Wednesday, my real issues are the following:
- it seems deliberate that the same alert ID tag was reused
(FEDORA-2008-1435 and FEDORA-2008-1535), it would seem to be a bit confusing to refer to multiple alerts with the same ID, take a peek at:
to see what I mean.
Basically there are to be considered just two updates, FEDORA-2008-1535 for Fedora 8 gecko-libs issues and FEDORA-2008-1435 for Fedora 7 gecko-libs issues.
What is confusing here is that the announcement was split across separate mails for each package. We are currently tracking the problem for the the update system [1].
[1] https://fedorahosted.org/fedora-infrastructure/ticket/392
- those were all related to the same gecko vulnerabilites, which is what
(I presume) motivated reusing the same IDs, but at least one (perhaps two, I can't remember for sure) of those, ruby-gnome2 also fixed a separate CVE that was unrelated to the mozilla pile
- How is it that so many packages were affected by these mozilla vulns? Are they statically linked? Reusing the code? Have very restrictive
dynamic library version numbers? It just seems that a vulnerability in a component shouldn't necessarily have this kind of cascading effect.
Due to upstream (Mozilla) policy on ABI stability, all packages that are dynamically linked to gecko libraries need to be rebuilt. (So basically you were correct, it's the "restrictive dynamic library version numbers"). This is definitely not ideal, but also not our fault -- situation is expected to improve a lot with advent of xulrunner in Fedora 9 though. I'm not expert on this, I might redirect you to our Mozilla guys if you need more information.
They are all pushed as a single update to prevent dependency breakage and if the update contains a security fix it is marked as security update. It is possible that attack vectors don't exist for many of the packages.
- Overall, we have been noticing a decline in the quality of Fedora
security alerts. They are often missing basic information about what bug they are fixing (other than perhaps a reference to bugzilla, sometimes a link to the CVE). I think if you read a lot of those alerts as if you were just a plain old user, you would find that some provide very little useful information to try and determine what problem is being fixed. I can provide examples if necessary. Is there something that can be done to standardize the format a bit?
We are attempting to concentrate the detail of fixed issues in bugzilla, while using descriptive titles of bugs. The update description relies upon decision of the maintainers. I was personally convinced that it is nod needed provided references to bugzilla are good enough.
What can be done is to motivate the maintainers to provide useful descriptions. Luke: Would it be possible to complement "Notes:" in bodhi with something like: "Please provide 2-3 sentences to briefly describe nature of each of problems being fixed" or something like that?
Thanks,
On Thu, Feb 14, 2008 at 04:53:09PM +0100, Lubomir Kundrak wrote:
Hi Jake,
On Thu, 2008-02-14 at 08:25 -0700, Jake Edge wrote:
(Josh Bressers suggested I send my questions here rather than asking him or someone else directly)
Yesterday you folks released an enormous number of security updates. While I could selfishly complain about it being done on a Wednesday, my real issues are the following:
- it seems deliberate that the same alert ID tag was reused
(FEDORA-2008-1435 and FEDORA-2008-1535), it would seem to be a bit confusing to refer to multiple alerts with the same ID, take a peek at:
to see what I mean.
Basically there are to be considered just two updates, FEDORA-2008-1535 for Fedora 8 gecko-libs issues and FEDORA-2008-1435 for Fedora 7 gecko-libs issues.
This behavior has existed for while now, but seems to be confusing when we have updates that contain a ton of builds. I'm in the process of revamping a good chunk of bodhi's model, so if we wanted to make the updateID<->build relationship 1-to-1, now would be the time.
What is confusing here is that the announcement was split across separate mails for each package. We are currently tracking the problem for the the update system [1].
[1] https://fedorahosted.org/fedora-infrastructure/ticket/392
Suggestions welcome for how you want the multi-package update notification template to look. I'd be glad to implement it.
- Overall, we have been noticing a decline in the quality of Fedora
security alerts. They are often missing basic information about what bug they are fixing (other than perhaps a reference to bugzilla, sometimes a link to the CVE). I think if you read a lot of those alerts as if you were just a plain old user, you would find that some provide very little useful information to try and determine what problem is being fixed. I can provide examples if necessary. Is there something that can be done to standardize the format a bit?
We are attempting to concentrate the detail of fixed issues in bugzilla, while using descriptive titles of bugs. The update description relies upon decision of the maintainers. I was personally convinced that it is nod needed provided references to bugzilla are good enough.
What can be done is to motivate the maintainers to provide useful descriptions. Luke: Would it be possible to complement "Notes:" in bodhi with something like: "Please provide 2-3 sentences to briefly describe nature of each of problems being fixed" or something like that?
I agree that are security notices are lacking in detail.
Encouraging developers to elaborate a bit more on the update notes may help, but that still doesn't give us any sort of standard advisory format to try and live up to.
Right now our update notices don't give any hint as to the severity of any given issue, as well as any details such as if it is remotely/locally exploitable, etc. At the moment some of this data exists in the bugzilla, but it's probably not obvious to our end users. If we want to keep this data in bugzilla, that's fine, but we need to make sure our users know where to find it.
Maybe we could encourage developers / security team to elaborate a little on the impact of the issues as well in the description ? We could possibly add more fields other than just "Update Details", such as "Synopsis", "Impact", etc?
I'm open to anything, really. Suggestions welcome.
luke
On Thu, 14 Feb 2008 13:05:16 -0500 Luke Macken lmacken@redhat.com wrote:
This behavior has existed for while now, but seems to be confusing when we have updates that contain a ton of builds. I'm in the process of revamping a good chunk of bodhi's model, so if we wanted to make the updateID<->build relationship 1-to-1, now would be the time.
For security update with bunch of rebuilds because of soname change, it may be possible to have field for "packages that needed to be rebuild because of changed dependencies". Those may be either announced as separate bugfix updates or included in security announcement mail in section that would indicate rebuild is the reason why packages are updated. But on the other hand, I'm not sure this is worth the effort because of one or two packages where such rebuilds are needed (firefox, maybe clamav sometimes, any other?).
There's another question - are we going to continue pushing security fix + rebuilds in one request? We already know this is confusing. Additionally, such rebuilds may further delay push of fixed packages. And here is another RelEng vs. Security question - do we prefer to push update FF when all rebuilds are not yet ready to provide fixes to users without epiphany/galeon/liferea/blam/.../<whatever> as soon as possible or let them wait just that users with <long list here again> installed won't see yum error message?
And another note not related to security updates at all - there are some updates where multiple nvrs in one request may make more sense, e.g. KDE or xfce updates.
What is confusing here is that the announcement was split across separate mails for each package. We are currently tracking the problem for the the update system [1].
[1] https://fedorahosted.org/fedora-infrastructure/ticket/392
Suggestions welcome for how you want the multi-package update notification template to look. I'd be glad to implement it.
Assuming special handling of security fix + rebuilds cases briefly described above:
Following package(s) address security issues:
firefox-<ver>-<rel>
Additionally, following packages were rebuilt to work correctly with updated packages listed above:
epiphany-<ver>-<rel> galeon-<ver>-<rel> [...]
Which can probably be turned to more general concept (for both security and non-security requests) - have field for nvrs that really fix some bugs and other field for nvrs that are just rebuilds.
Right now our update notices don't give any hint as to the severity of any given issue, as well as any details such as if it is remotely/locally exploitable, etc.
Correct. Do we want to have that? Maybe yes. But it can hardly be achieved without developer wanting to provide such info in the updates. Current approval process will not help much unless reviewer has time to fix up description and if (s)he has something to fix up at the beginning.
Maybe we could encourage developers / security team to elaborate a little on the impact of the issues as well in the description ? We could possibly add more fields other than just "Update Details", such as "Synopsis", "Impact", etc?
Some of those fields might make sense. But will those be filled-in right?
Moreover, we should try to make our security updates CVE compatible. We may need to have possibility to edit CVE id list even for older updates, as CVE id may not be available at the time of submission or may change later (splits / duplicate rejections / ...). Do we want to delay pushing of updates until we have CVE id for the issues addressed? It's not that rare that updates are pushed without CVE id there days (which has pros in terms of fixes getting out quickly for the price of inconsistency).
Sorry for long and boring mail ;).
On Fri, Feb 15, 2008 at 05:10:22PM +0100, Tomas Hoger wrote:
On Thu, 14 Feb 2008 13:05:16 -0500 Luke Macken lmacken@redhat.com wrote:
This behavior has existed for while now, but seems to be confusing when we have updates that contain a ton of builds. I'm in the process of revamping a good chunk of bodhi's model, so if we wanted to make the updateID<->build relationship 1-to-1, now would be the time.
For security update with bunch of rebuilds because of soname change, it may be possible to have field for "packages that needed to be rebuild because of changed dependencies". Those may be either announced as separate bugfix updates or included in security announcement mail in section that would indicate rebuild is the reason why packages are updated. But on the other hand, I'm not sure this is worth the effort because of one or two packages where such rebuilds are needed (firefox, maybe clamav sometimes, any other?).
There's another question - are we going to continue pushing security fix
- rebuilds in one request? We already know this is confusing.
Some developers push updates differently than others. Some put all packages and deps in 1 update, some submit individual requests. It's really just a matter of what fits their workflow the best (web submission, cli, etc).
If a security update contains other packages in the same bodhi "update", each build still gets a separate email generated and sent to the list. If we're planning on generating multi-build update notices, then having a section for 'rebuilt dependencies' might be a good thing.
Additionally, such rebuilds may further delay push of fixed packages. And here is another RelEng vs. Security question - do we prefer to push update FF when all rebuilds are not yet ready to provide fixes to users without epiphany/galeon/liferea/blam/.../<whatever> as soon as possible or let them wait just that users with <long list here again> installed won't see yum error message?
We've gotten a little better at this over time, but it's still an issue that needs to be addressed. If we push a security fix without it's rebuilt dependencies, then isn't there a good chance that people won't be able to install this due to broken deps? Sounds like an issue that is best resolved server-side, instead of pushing brokenness on to our users.
And another note not related to security updates at all - there are some updates where multiple nvrs in one request may make more sense, e.g. KDE or xfce updates.
Yeah. Again, that's really up to the developer. I'm thinking it might be useful to have bodhi keep track of which groups of packages have been pushed together, and allow developers to say "update the gnome stack", or something of the sort.
What is confusing here is that the announcement was split across separate mails for each package. We are currently tracking the problem for the the update system [1].
[1] https://fedorahosted.org/fedora-infrastructure/ticket/392
Suggestions welcome for how you want the multi-package update notification template to look. I'd be glad to implement it.
Assuming special handling of security fix + rebuilds cases briefly described above:
Following package(s) address security issues:
firefox-<ver>-<rel>
Additionally, following packages were rebuilt to work correctly with updated packages listed above:
epiphany-<ver>-<rel> galeon-<ver>-<rel> [...]
Which can probably be turned to more general concept (for both security and non-security requests) - have field for nvrs that really fix some bugs and other field for nvrs that are just rebuilds.
Sounds good, but integrating this with our current template may be a little bit more difficult. Right now we have:
Header (date, update ID) RPM Details (name, product, version, url, summary, description) Update Information (what the developer typed into bodhi) RPM ChangeLog References (bugzillas) Foot notes
So, how do we want to handle multiple builds with the above template? We could possibly cut down the RPM Details to only display the nvr and summary, and then just stuff each builds ChangeLog together.
Right now our update notices don't give any hint as to the severity of any given issue, as well as any details such as if it is remotely/locally exploitable, etc.
Correct. Do we want to have that? Maybe yes. But it can hardly be achieved without developer wanting to provide such info in the updates. Current approval process will not help much unless reviewer has time to fix up description and if (s)he has something to fix up at the beginning.
Yeah, true. It'll require more effort from the developers as well as the security team. Probably not the best use of our time at the moment.
Maybe we could encourage developers / security team to elaborate a little on the impact of the issues as well in the description ? We could possibly add more fields other than just "Update Details", such as "Synopsis", "Impact", etc?
Some of those fields might make sense. But will those be filled-in right?
Yeah, probably not. Again, this will require at least another set of eyes to revise this data before it gets approved.
Moreover, we should try to make our security updates CVE compatible. We may need to have possibility to edit CVE id list even for older updates, as CVE id may not be available at the time of submission or may change later (splits / duplicate rejections / ...). Do we want to delay pushing of updates until we have CVE id for the issues addressed? It's not that rare that updates are pushed without CVE id there days (which has pros in terms of fixes getting out quickly for the price of inconsistency).
I don't think we're actually not too far away from being "CVE Compatible" as it is.
1. CVE Searchable: A user can search using a CVE name to find related information. 2. CVE Output: Information is presented which includes the related CVE name(s). 3. Mapping: The repository owner has provided a mapping relative to a specific version of CVE, and has made a good faith effort to ensure accuracy of that mapping. 4. Documentation: The organization's standard documentation includes a description of CVE, CVE compatibility, and the details of how its customers can use the CVE-related functionality of its product or service.
1 and 2 are pretty much done ( may need a little bit of tweaking ). Not sure where we are with 3, but 4 should be pretty easy.
Bodhi used to keep track of CVEs in it's own db table, but that recently changed as we now track them using Bugzilla, but it will still be easy to get bodhi to find & recognize them.
Regarding delaying updates for CVE ids, I'm not quite sure where we stand with this. Sounds like an issue that needs to be addressed with our security update policy.
luke
On Thu, 2008-02-14 at 08:25 -0700, Jake Edge wrote:
(Josh Bressers suggested I send my questions here rather than asking him or someone else directly)
Yesterday you folks released an enormous number of security updates. While I could selfishly complain about it being done on a Wednesday, my real issues are the following:
(...lot of stuff)
https://fedorahosted.org/fedora-infrastructure/ticket/392#comment:2 We're eager to hear your comments.
Lubomir Kundrak wrote:
https://fedorahosted.org/fedora-infrastructure/ticket/392#comment:2 We're eager to hear your comments.
I think my questions were answered. I like what I see in the template for security reports and the fact that y'all are giving them some attention at the moment. I definitely agree that changelogs are only interesting if they reflect the changes in the package for that release (unlike they sometimes have in the past).
If it is 'easy', it would be helpful to update readers to have the CVE references be links to CVE or NVD rather than just link to the redhat bugzilla ...
jake
On Sun, 2008-02-24 at 14:09 -0700, Jake Edge wrote:
Lubomir Kundrak wrote:
https://fedorahosted.org/fedora-infrastructure/ticket/392#comment:2 We're eager to hear your comments.
I think my questions were answered. I like what I see in the template for security reports and the fact that y'all are giving them some attention at the moment. I definitely agree that changelogs are only interesting if they reflect the changes in the package for that release (unlike they sometimes have in the past).
If it is 'easy', it would be helpful to update readers to have the CVE references be links to CVE or NVD rather than just link to the redhat bugzilla ...
Our decision was not to, because:
1.) Sometimes we get the CVE name after we ship the update, and unlike the update mails, we can easily update bugzilla.
2.) In most cases our bugzilla contains verbatim copy of the CVE text, and in all cases it has links to CVE, NVD and alias that is equal to the CVE name. Our bugzilla even substitutes the CVE names with links to CVE.
Regards,
Lubomir Kundrak wrote:
On Sun, 2008-02-24 at 14:09 -0700, Jake Edge wrote:
If it is 'easy', it would be helpful to update readers to have the CVE references be links to CVE or NVD rather than just link to the redhat bugzilla ...
Our decision was not to, because:
1.) Sometimes we get the CVE name after we ship the update, and unlike the update mails, we can easily update bugzilla.
2.) In most cases our bugzilla contains verbatim copy of the CVE text, and in all cases it has links to CVE, NVD and alias that is equal to the CVE name. Our bugzilla even substitutes the CVE names with links to CVE.
Ok, I am looking at today's (or maybe late yesterday's) report for qemu for F7: FEDORA-2008-2001
It doesn't list the CVE number, so I click through to bugzilla, which does list the CVE number (as an Alias), but doesn't link to CVE/NVD (which is just a placeholder at this point anyway, but will presumably be updated soon).
Does the changelog reflect the changes in this release? Which would imply that there are fixes for other, non-security bugs in the release.
It just strikes me as difficult for people receiving the advisories (or reading them on our or other sites) to figure out the *exact* bug being fixed without a CVE reference in the advisory. Maybe the timing is too tight, but that is very unfortunate.
jake
On Tue, Feb 26, 2008 at 12:19:06PM -0700, Jake Edge wrote:
Lubomir Kundrak wrote:
On Sun, 2008-02-24 at 14:09 -0700, Jake Edge wrote:
If it is 'easy', it would be helpful to update readers to have the CVE references be links to CVE or NVD rather than just link to the redhat bugzilla ...
Our decision was not to, because:
1.) Sometimes we get the CVE name after we ship the update, and unlike the update mails, we can easily update bugzilla.
2.) In most cases our bugzilla contains verbatim copy of the CVE text, and in all cases it has links to CVE, NVD and alias that is equal to the CVE name. Our bugzilla even substitutes the CVE names with links to CVE.
Ok, I am looking at today's (or maybe late yesterday's) report for qemu for F7: FEDORA-2008-2001
It doesn't list the CVE number, so I click through to bugzilla, which does list the CVE number (as an Alias), but doesn't link to CVE/NVD (which is just a placeholder at this point anyway, but will presumably be updated soon).
The summary of security bugs are *supposed* to begin with the CVE id, according to the security bug tracking procedure[0]. It looks like this update got added to our updates system before the bug summary was properly updated.
Does the changelog reflect the changes in this release? Which would imply that there are fixes for other, non-security bugs in the release.
Yes, the ChangeLog /should/ be the changes in that package, for that release, from the last update of that package.
It looks like the F7 qemu update[1] pulled in a bit too much of the changelog. The F8 notice looks fine, but the F7 changelog mentions qemu-0.9.0-3.fc7[2], which was pushed as a bugfix update in October.
As far as I can tell, it looks like Lubomir is proposing[3] to remove the RPM ChangeLogs all together from our security notices, which would help mitigate the inconsistencies mentioned above. However, I have a feeling that many people would complain if the changelogs disappeared.
I'm all for removing it, but I think it may be worth assessing what kind of value we want these changelogs to provide vs. the value they are actually providing to the end user. With all of the proper bugs listed, and fairly informative update details, I'm not sure what value the RPM changelog provide alongside of them ?
It just strikes me as difficult for people receiving the advisories (or reading them on our or other sites) to figure out the *exact* bug being fixed without a CVE reference in the advisory. Maybe the timing is too tight, but that is very unfortunate.
I agree, I think we should be linking to CVEs somewhere, at *least* from the bodhi-view of updates[4]. I can easily make all CVE ids in the bug titles linkable back to mitre within bodhi, but whether or not we put the CVE urls along with the Bugzillas in the references in our advisories is still up for discussion.
Thanks for your input on this, Jake. It's always good to have someone on the outside to let us know when stuff doesn't make sense :)
luke
[0]: http://fedoraproject.org/wiki/Security/TrackingBugs [1]: https://www.redhat.com/archives/fedora-package-announce/2008-February/msg008... [2]: https://www.redhat.com/archives/fedora-package-announce/2007-October/msg0004... [3]: https://fedorahosted.org/fedora-infrastructure/ticket/392#comment:2 [4]: https://admin.fedoraproject.org/updates/FEDORA-2008-2001
security@lists.fedoraproject.org
-
Jake Edge -
Jesse Keating -
Lubomir Kundrak -
Luke Macken -
Pavel Kankovsky -
Tomas Hoger