devel December 2013

devel@lists.fedoraproject.org

170 participants
203 discussions

Fedora 20 release day FedUp bug: post-mortem
by Adam Williamson 09 Jan '14

09 Jan '14

Hi, folks. Now things have calmed down a bit in Fedora 20 and Rawhide, I have time to write this mail! Many of you may already know that there was a significant issue with upgrades to Fedora 20 around release day - 2013-12-17. Summary of the issue -------------------- Upgrading to Fedora 20 using version 0.7 of the FedUp tool does not work. Upgrading with version 0.8 works (in the main - of course there are bugs, there are always bugs). At the time Fedora 20 was released, version 0.7 of FedUp was present in the Fedora 18 and Fedora 19 'updates' repositories. Version 0.8 of FedUp was present in 'updates-testing' for both Fedora 18 and Fedora 19 at the time. Immediate response to the issue ------------------------------- We realized quite quickly during the course of release day support that this was the case, though at first we thought perhaps only some upgrades were failing. Once it became clear that all 0.7-based upgrades would fail, several folks worked hard at communicating this to as many users in as many places as possible, including via IRC, mailing lists, the Common Bugs page (https://fedoraproject.org/wiki/Common_F20_bugs#fedup-07-fail ), the forums, and social network sites like G+. We advised using fedup 0.8 from updates-testing to upgrade. We rapidly ensured 0.8 was submitted for stable push for both F18 and F19. It was submitted for F19 at 2013-12-17 21:12:18 (I believe Bodhi timestamps are UTC, so that was mid-afternoon on release day in NA) and for F18 at 2013-12-18 11:51:47 (early morning on the day after release). However, release engineering complications (there were some problems with stable pushes at the time) meant it wasn't finally pushed until 2013-12-19 07:23:09 UTC for F19 (late on the day after release NA time) and 2013-12-19 14:05:50 UTC for F18 (early morning two days after release) and wouldn't have made it to most mirrors until 2013-12-19, two days after release, and probably 2013-12-20 in 'early' timezones in Europe and Asia. Proximate cause of the issue ---------------------------- We have not yet identified the direct (proximate) cause of the bug; doing so did not seem especially important in comparison to ensuring news of the issue was spread as widely as possible, ensuring 0.8 was sent stable as soon as possible, and resolving some related issues (see later). However, QA's current inference is that there is some incompatibility between how fedup 0.7 modifies the initramfs used by the upgrade process and/or how it configures the upgrade boot environment, and the expectations of the upgrade environment as it exists within the final shipped upgrade initramfs. The upgrade initramfs is generated as part of the release compose process, and is dependent on factors including the versions of dracut and fedup-dracut used to build it. Broadly, we suspect that an upgrade run with fedup 0.7 which uses an upgrade initramfs generated with fedup-dracut 0.8 will not work, for reasons not yet identified. Indirect causes of the issue ---------------------------- We could perhaps make a very broad characterization of the 'indirect causes' of the issue as follows: an upgrade using fedup depends on several moving parts, and neither our development nor testing processes are sufficiently robust to ensure that we cover all possible combinations of those parts. fedup / fedup-dracut interdependencies ++++++++++++++++++++++++++++++++++++++ So far as I can discern, there is not at present any policy (whether written or enforced by some kind of mechanism) with regard to the inter-dependencies between the 'fedup' package side of the fedup process and the 'fedup-dracut' side of the process which involves release engineering generating an upgrade initramfs via fedup-dracut. As this issue suggests that not all 'fedups' work with all 'fedup-dracuts', perhaps this is something that might be required, but we leave that to the superior knowledge and expertise of the FedUp maintainer. Test procedure inadequacies +++++++++++++++++++++++++++ Similarly, QA's upgrade testing process clearly did not sufficiently carefully consider the same issue. This is something we have now moved to address. Prior to Fedora 20's release, the test cases for fedup recommended testing the latest version of fedup from updates-testing against the upgrade initramfs from the development/20 tree. This procedure was a holdover from the very early days of FedUp, when it was changing daily and testing anything older was uninteresting, and when procedures for the generation and publishing of the upgrade initramfs had not yet been clearly established (and TC/RC trees did not contain one). However, it is no longer appropriate for the more mature state of FedUp development at this point in time, and it should have been changed earlier. We in QA apologize to the project for this oversight. Other factors +++++++++++++ Additionally, various parties have noted in discussion of this issue that we would have been more likely to notice it, even with our imperfect testing procedures, if a couple of other factors had been different: * The lifetime of the final release candidate * The timing of changes to fedup In recent Fedora releases it has become something of a habit (for which I personally bear rather a large share of the blame) for us to reach RC stage late, iterate RCs rapidly, and often ship an RC that was built only days or even hours before the Go/No-Go decision. This has allowed us to fix bugs we might not otherwise have fixed and to avoid release delays. However, it has the obvious danger that testing of the final release bits may not be as comprehensive as it could be. We always ensure the formal validation testing is sufficiently complete, but an issue like this highlights that a few more days of testing are likely to catch things the formal validation testing process may miss for various reasons, including the kind of deficiency noted above. Even though our test procedure for fedup was outdated, if the final RC had lived for two or three days before being signed off, someone would likely have happened across this issue in time for something to be done about it. The fact that fedup 0.8 and fedup-dracut 0.8 landed quite late in the cycle is also relevant. fedup 0.8 was submitted for updates-testing on 2013-12-11; fedup-dracut was submitted on 2013-12-06, but the first compose which used it was RC1, built on 2013-12-12 (there was a delay of several days between TC5 and RC1, as blocker bugs kept appearing and needing to be fixed before an RC1 could be spun). RC1.1 was signed off for release on 2013-12-12. Even the mathematically-challenged will note that this left us extremely limited time to spot the problem. (I had tested upgrades with the updated fedup-dracut using a 'scratch built' upgrade initramfs rather earlier, but I must have used fedup 0.8 rather than 0.7 for my tests). Obviously, if these fairly significant version bumps had arrived earlier, we may have had more time to identify issues in them. If you're wondering how they were allowed to land so late, the answer is that they fixed blocker bugs we had identified in earlier upgrade testing, and so were allowed through the freeze. As we all know, it is difficult to adhere strictly to 'best practices' with Fedora's extremely short release cycles and ambitious pace of development, but of course it would be best in future if we can manage to avoid landing significant changes to fedup so late, a goal to which both QA and development groups can contribute by identifying and fixing issues at an earlier stage. As a 'meta' note, I think a factor that contributes to all of the above factors may be a lack of understanding outside a very few people as to precisely how the entire fedup process works: speaking personally, I certainly wasn't acquainted with all the subtleties until investigating this and other issues (not that I'd confidently claim to be an expert even now!) I think beyond Will Woods (obviously) and possibly Tim Flink (who did a lot of early fedup testing) and Dennis Gilmore (who tends to be the one generating the upgrade initramfs), possibly no-one really entirely understood the whole process. Related issues -------------- It is probably worth noting a somewhat-related issue at this point. fedup 0.8's major change compared to fedup 0.7 was that it introduced checking of GPG signatures on update packages. To facilitate this, the signing key for the release to which one is upgrading must be available to fedup running on the release from which one is upgrading. Again, we did not have this fully in place at the time of Fedora 20's release. The fedora-release-19-5 update added Fedora 20's key to Fedora 19: https://admin.fedoraproject.org/updates/FEDORA-2013-21411/fedora-release-19… . It was submitted on 2013-11-14 and pushed stable on 2013-12-03, so this was in place ahead of release. However, for Fedora 18, the relevant update - https://admin.fedoraproject.org/updates/FEDORA-2013-23598/fedora-release-18… - was submitted on 2013-12-18 and pushed stable on 2013-12-22 (and then we had to add a signed .treeinfo file to the Fedora 18 repositories or things *still* didn't work, which I think we did late on 2013-12-22 or on 2013-12-23). The fact that the keys weren't available for F18 was known around F20 release time, but was not considered urgent by the parties involved as we were not aware that fedup 0.7 simply would not work and consequently that it would be an urgent matter to make fedup 0.8 available and functional, and release engineering considered it a delicate operation to add the keys for Fedora 19 and 20 to Fedora 18, and one which they were not inclined to rush. Post-release reports also make it clear that fedup will abort if GPG keys for *any* repository fedup finds available for the target release cannot be found. i.e., if you have RPM Fusion or another popular third party repository configured, it's quite likely your upgrade will fail, because third party repos didn't have the signing key issue lined up (not surprising if we couldn't even entirely manage it ourselves). We were not sufficiently aware of this behaviour before release, and did not communicate it very well. The underlying causes of this are much the same as the underlying causes of the main issue - the fedup which enabled GPG checking landing very late, inadequate/incorrect test procedures, and limited knowledge of the details of fedup operation outside a small group of people. Addressing the problems ----------------------- I've noted above that so far as specific code responses to any of these issues go, we should probably defer to the wisdom of the maintainer. However, I've filed a couple of intentionally vague and open-ended tickets on fedup to provide a forum for action: https://github.com/wgwoods/fedup/issues/42 https://github.com/wgwoods/fedup/issues/43 In terms of QA test procedures, we (QA) have already taken action that should help guard against a repeat of this kind of issue in future. The FedUp test cases - for instance, https://fedoraproject.org/wiki/QA:Testcase_upgrade_fedup_cli_previous_deskt… - have been adjusted to recommend testing the latest fedup from stable or updates (not from updates-testing), and to test against the current TC/RC tree (not the daily-updated development/ tree), now TC/RC trees contain the upgrade initramfs image. The FedUp and Upgrading wiki pages (https://fedoraproject.org/wiki/FedUp and https://fedoraproject.org/wiki/Upgrading ) have also been updated to be more consistent and correct for the current state of fedup, and the Installation Guide's section on upgrading has also been updated. Our test procedures and upgrade documentation should now be much more coherent and consistent than they were just prior to Fedora 20's release. In wider terms, this issue is another indicator on top of several previous ones that we should redouble our efforts to get 'releaseable' RCs built days ahead of go/no-go, rather than hours. That's a whole story in itself, but this is something the parties involved are all aware of and working on. Of course, the whole release process may look somewhat different in a Fedora.next world, but as long as we have our current release schedule and freeze policies, this issue is likely to exist at least in essence. It's also another good indicator that we should do whatever we can to try and land major changes much earlier in the release cycle. This is hardly a new observation, of course, nor an issue of which many relevant people were previously unaware, and there are always good reasons why we wind up landing the kitchen sink a week before release, but it's always good to have another reminder. On the positive side, the simple fact that this issue occurred has probably led to a wider understanding of at least some of the details of how fedup operates, and the fact that more people in the project have that knowledge should aid us in future fedup development and testing: we should be careful to keep that knowledge in mind as we build and test future releases. Conclusion ---------- Er, thanks for reading this far? :) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net

6 8

pacemaker in Fedora and RHEL
by Christopher Meng 08 Jan '14

08 Jan '14

Hi all, I wonder if there are some reasonable explanations of package "pacemaker" in the repo. I checked the package and the result returned is: Fedora: pacemaker-1.1.9-3.fc20 in rawhide. RHEL: pacemaker-1.1.10-14.el6. The latest version is 1.1.11RC1(1.1.10 Final stable). Why is the package in Fedora behind the one in RHEL in spite of the maintainer of these two are the same people? Thanks. -- Yours sincerely, Christopher Meng Fєdоґa ї₴ al$о a кїпd оf нaт lїкє Яёd Haт. http://cicku.me

3 2

Unannounced ABI change without soname bump in libevdev-0.6 in Rawhide (and F19 and F20...) breaks GNOME, probably other consumers
by Adam Williamson 08 Jan '14

08 Jan '14

Time for another PSA... It appears libevdev 0.6 breaks the library's ABI without bumping the soname (and without an announcement here or anywhere else I can find, but an ABI change without an soname bump is just flat out wrong whether announced or not). 0.6 was sent to Rawhide, F19 and F20 simultaneously. The ABI change breaks GNOME in F20 and Rawhide (because clutter was using the calls that disappeared in the 0.6 build): undefined symbol: LIBEVDEV_READ_NORMAL (/lib/libclutter-1.0.so.0) undefined symbol: LIBEVDEV_READ_SYNC (/lib/libclutter-1.0.so.0) Dec 26 02:02:34 pressanykey gnome-session[1320]: gnome-shell: symbol lookup error: /lib64/libclutter-1.0.so.0: undefined symbol: LIBEVDEV_READ_NORMAL Reported as https://bugzilla.redhat.com/show_bug.cgi?id=1046426 . Rebuilding clutter is not trivial at present, and in fact builds of clutter appear to fail on this same libevdev issue. So right now, GNOME in Rawhide is broken if you install libevdev 0.6, and this monkey doesn't feel comfortable trying to fix it (I don't want to start randomly whacking things in clutter's code to try and make it build against the new libevdev). I recommend not installing this build, and downgrading to 0.5 if you currently have 0.6 installed. If anyone from the desktop team is able to get together with Peter and resolve this SNAFU for Rawhide ASAP, that'd be great, though I realize we're all supposed to be on vacation. Happily, the update for F20 has already been negative-karma'ed out of existence, and the F19 update is well on its way there. Take note, those who say the update karma system is useless: without it, we would likely have broken the world's F20 systems shortly after release...(F19's clutter doesn't seem to use libevdev so GNOME on F19 doesn't break, apparently, but the libevdev update still shouldn't go out). http://koji.fedoraproject.org/koji/buildinfo?buildID=486593 (Rawhide) https://admin.fedoraproject.org/updates/FEDORA-2013-23934/libevdev-0.6-1.fc… (F20) https://admin.fedoraproject.org/updates/FEDORA-2013-23961/libevdev-0.6-1.fc… (F19) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net

11 25

Agenda for today's Env-and-Stacks WG meeting (2013-12-17)
by Marcela Mašláňová 08 Jan '14

08 Jan '14

WG meeting will be at 16:00 UTC in #fedora-meeting on Freenode. == Next Steps == - I suggest copying first two paragrahps for our so called PRD from https://fedoraproject.org/wiki/Cloud_PRD as was discussed two weeks ago. - PRD - based on discussion on env-and-stacks mailing list (some discussion on pad: http://piratepad.net/PwUiH4MEPR) If we don't have any other input from other areas, we could sum up what we discussed and create list of tasks for future work (Review tool, taskotron, ...). == Open Floor ==

2 4

PSA: If you are C/C++ developer, use cppcheck
by Rahul Sundaram 07 Jan '14

07 Jan '14

Hi In the last few days, I have been running cppcheck on quite a few programs including systemd, transmission, libvirt, ndjbdns etc and cppcheck has found real and potential bugs (null pointer dereferences, uninitialized variables, memory & resource leaks etc) in each of them. I have reported the ones I found and several developers have already fixed the issues. A couple of examples http://cgit.freedesktop.org/systemd/systemd/commit/?id=e985665d2d226cb42b52… https://github.com/pjps/ndjbdns/commit/ee4112a702e22d447d9cd7bd31b880eacfe5… You might also consider add a build target for regular checks http://cgit.freedesktop.org/systemd/systemd/commit/?id=16f4efb4150c65e3c61a… That would be all. Thanks Rahul

12 28

Problem changing symlink to directory with %pretrans scriptlet
by Richard Fearn 07 Jan '14

07 Jan '14

Hi all, I'm trying to change a symlink to a directory but it's not working. I've reduced the problem down to a simple test case. This package: http://richardfearn.fedorapeople.org/test/1-1/test.spec creates the following: * a directory /usr/share/javadoc/test-1 (named after the package version) * a file within the test-1 directory * /usr/share/javadoc/test, a symlink to test-1 In the next version: http://richardfearn.fedorapeople.org/test/1-2/test.spec the Javadoc directory is unversioned, so the 'test' symlink is now a directory (containing one file). As expected the 1-1 to 1-2 upgrade fails: file /usr/share/javadoc/test from install of test-1-2.fc20.noarch conflicts with file from package test-1-1.fc20.noarch So in the next version: http://richardfearn.fedorapeople.org/test/1-3/test.spec a %pretrans scriptlet deletes /usr/share/javadoc/test, if it's a symlink. However during the 1-1 to 1-3 upgrade, the old test-1 directory isn't deleted. The verbose rpm output from this upgrade: http://richardfearn.fedorapeople.org/test/1-3-upgrade.txt shows that rpm doesn't erase /usr/share/javadoc/test-1/hello. It does seem to try to erase the /usr/share/javadoc/test-1 directory, but can't because it still contains a file. Does anyone have any idea what's going on here? Regards, Rich -- Richard Fearn richardfearn(a)gmail.com

4 4

Unannounced soname bump: tracker
by Adam Williamson 07 Jan '14

07 Jan '14

Sigh. Yes, another of these. On 2013-12-18, tracker was bumped to 0.7.0: http://koji.fedoraproject.org/koji/buildinfo?buildID=485698 the sonames of libtracker-extract, libtracker-miner and libtracker-sparql were bumped to 0.18.so.0 (from 0.16.so.0) without announcement, and without all dependent packages being successfully rebuilt. At least the following still depend on the old sparql library: bijiben-0:3.11.1-1.fc21.x86_64 brasero-0:3.11.3-1.fc21.x86_64 grilo-plugins-0:0.2.9-2.fc21.x86_64 media-explorer-0:0.4.4-5.fc21.x86_64 rygel-tracker-0:0.21.1-1.fc21.x86_64 I'll see if I can get any of them rebuilt. What does it take for people to handle soname bumps properly? -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net

6 8

Updating hdf5 to 1.8.12
by Orion Poplawski 05 Jan '14

05 Jan '14

I will be updating hdf5 to 1.8.12 shortly in rawhide and rebuilding dependent packages: Field3D-1.3.2-14.fc21.src.rpm R-hdf5-1.6.9-22.fc20.src.rpm ViTables-2.1-7.fc21.src.rpm cgnslib-3.2-3.fc20.src.rpm gdl-0.9.4-1.fc21.src.rpm hdf5-1.8.11-6.fc21.src.rpm jhdf5-2.9-3.fc20.src.rpm matio-1.5.1-4.fc21.src.rpm netcdf-4.3.0-7.fc21.src.rpm octave-3.6.4-9.fc21.src.rpm paraview-4.0.1-3.fc21.src.rpm scilab-5.5.0-0.2.beta1.fc21.1.src.rpm vtk-6.0.0-9.fc21.src.rpm -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion(a)nwra.com Boulder, CO 80301 http://www.nwra.com

2 2

Fwd: [pkgdb] python-boto ownership changed
by Orion Poplawski 02 Jan '14

02 Jan '14

Is anyone interested in taking on python-boto, please? -------- Original Message -------- Subject: [pkgdb] python-boto ownership changed Date: Thu, 26 Dec 2013 00:19:48 +0000 From: Fedora PackageDB <pkgdb(a)fedoraproject.org> To: orion(a)cora.nwra.com Package python-boto in Fedora 18 was orphaned by robert To make changes to this package see: https://admin.fedoraproject.org/pkgdb/acls/name/python-boto

3 3

Review swap - rename
by Orion Poplawski 01 Jan '14

01 Jan '14

This is a package rename, pretty simple: https://bugzilla.redhat.com/show_bug.cgi?id=1029197 Rename Request: gshhg-gmt-nc4 - Global Self-consistent Hierarchical High-resolution Geography (GSHHG) Needed for GMT 5. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion(a)nwra.com Boulder, CO 80301 http://www.nwra.com

2 2

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

devel December 2013