Continuous delivery of release 132 failed and recovery - cockpit-devel - Fedora mailing-lists

24 Feb 2017


      If you've been following on IRC, you may have noticed that releasing
Cockpit 132 failed yesterday. Here's why, and how it's been solved.
In general we want the entire release process to be completely
automatic, with all failures detected before hand.
Earlier this week I made a pull request that passed integration tests,
but broke the continuous delivery:
https://github.com/cockpit-project/cockpit/pull/5924
The symptom was that certain files we installed in 'make install' when
they shouldn't have been installed there. But the root cause was that we
were building RPMs differently during integration tests, and later
during release:
https://fedorapeople.org/groups/cockpit/logs/release-132/log
To solve this problem I added patches to the release-132 branch to fix
this problem, and then ran the continuous delivery scripts manually from
that branch:
https://github.com/cockpit-project/cockpit/tree/release-132
... in the cockpit/infra-release container ...
$ git fetch origin
$ git checkout release-132
$ RELEASE_SINK= RELEASE_TAG=132 release-runner -v
/usr/local/bin/major-cockpit-release
What this does is use the 132 git tag to create a tarball, and then any
commits after that are included in the SRPM as patches.
So how do we prevent this from happening again?
1. Build RPMs during integration more like release:
https://github.com/cockpit-project/cockpit/pull/5942
2. Cleanup the code related to 'make install' and 'make install-tests'
https://github.com/cockpit-project/cockpit/pull/5941
3. Document how to solve this in the future (this email).
4. I noticed that the release-dsc file does not consider patches the
    way that release-srpm does. This needs a fix, any takers?
https://github.com/cockpit-project/cockpituous/blob/master/release/release-d...
This wasn't a problem this time around because the patches did
    not affect debian builds.
5. Figure out how to test upgrading. Martin Pitt caught it this time,
    but it should have failed before going in:
https://github.com/cockpit-project/cockpit/pull/5937
Cheers,
Stef