Rawhide plans

[Kevin Fenzi]

Greetings. 

We had some nice discussions about rawhide in my friday workshop at
flock. I thought I would post here to get input from folks not there,
and also allow people who were there to comment more now that it's not
after 5pm on a friday of the 3rd day of flock. ;) 

* The problems/issues: 

First we talked about current issues we see. 

I think most everyone was in agreement that once you had a rawhide
install and were just applying updates, you were in pretty good shape.
Much better than you would have been a few years ago. There are bugs,
but they are often easy to workaround with downgrades until they are
fixed, etc. Also dnf tries very hard to not install anything thats part
of a set of broken deps, so while it might take longer to get some
packages you won't break your local install. 

I think everyone also agreed that the pain points were around getting a
rawhide install in the first place: 

1. boot/netinstall iso is often broken or not produced. Live media
likewise.

2. Upgrading from a stable release is also often hard due to broken
deps, resulting in having to remove packages just to get upgraded. 

There was some mention of rawhide not being fully signed (which we
cannot do until we have gating of some kind and the sigul batch signing
bug fixed).

There was mention of larger rebuilds that didn't use side tags. It's
pretty easy to request a side tag from releng and use it to do all your
rebuilds and then get it tagged back in. When larger stacks don't do
this, they break things for a few days while they sort out all the
packages. 

Finally there was mention of the baggage associated with the name
"rawhide". To this day I see people telling others that "rawhide will
eat your babies" or "rawhide is a methlab and will blow up in your face
every day". 

Do you all see other issues ?

Next I divided things up into 3 time periods. What could we do to make
things better short term (in the next few weeks), medium term (next few
months), longer term (next year).

Short term:

* Hook up openqa to run on rawhide every night and give us data on how
  often things are broken on the install path. This could be a seperate
  report, or we could hook it up to the rawhide compose. 

* Hook up taskotron to run checks on packages as they are built and
  send at least to the maintainer about the broken deps. If the
  maintainer sees their build causing lots of broken deps they can
  untag it or get someone to rebuild all the other packages. 

* pungi4 landing. pungi4 is a big re-write of our pungi tool. When we
  enable this in rawhide it's going to make rawhide look a lot more
  like release trees. Instead of having a rawhide/ dir with just trees
  and boot isos, we will have Everything tree like we do at release
  time, and server/cloud/workstation trees each with their own branded
  netinstall isos, the server dvd and such. Also, pungi4 emits json
  logs with exactly what was produced, so we can easily parse from day
  to day and know what broke, etc. 

* Matt opened a thread on the marketing list about renaming rawhide. It
  sounds like most people would prefer us to make the changes first,
  then and only then look at renaming. 

Medium term: 

* Setup some gating. I think it should work like this: 

today you build a rawhide build and it tags into f24 tag, which is used
by the daily compose. 

I'd like to change that to build into a f24-candidate tag. At that
point taskotron or other tests are run. If the package passes it goes
to f24-pending tag. When a compose happens we run taskotron or other
tests on the f24-pending packages and if they pass, they tag into f24
and the compose happens. 

This gives us two points to check/test things. If someone needs to
override due to a check thats wrong or some other need, they can simply
use the existing koji command line client to tag their package over and
it will go out in the compose anyhow. Note that we will then have a log
of who did this. ;) 

This will also hopefully allow us to enable autosign for rawhide
packages (if we can get the batch signing bug fixed) since we can sign
them at one of the above points. 

Longer term:

* Put in place something so we can tell if a just rebuilt package is: 

- in the mock init root
- in the deps for pungi4/mock/etc
- on the boot/netinstall isos
- on live media/images
- not on/used by anything in composes

If it is, then fire off a test rebuild of that thing, test it and only
let the new thing in if it doesn't fail. We can also then relax
requirements for leaf node type packages if we wish.

* Possibly drop daily rawhide composes in favor of just rebuilding
  things when something in them changes. ie, new kernel build? then
  rebuild images/trees, new cowsay, just rebuild the repo. 

Feedback welcome. 

kevin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/rel-eng/attachments/20150819/07559470/attachment.sig>