Summary of the Fedora Secondary Arch Team for Power meeting at FUDCon Lawrence and Statement of Direction

Thu Jan 31 13:41:24 UTC 2013

Hi everyone.

As the Fedora Secondary Arch Team for Power met again this year at the North
American FUDCon, this year in Lawrence we've put together a pretty long list
of topics we wanted to cover and discuss during those days. Details were
available here
https://fedoraproject.org/wiki/Architectures/PowerPC/Meetings/FUDCon_Lawrence_2013
which was linked from the main secondary arch wiki for Power on fp.org 
the
last few months. 

The team meeting at Lawrence were:

Phil Knirsch
Karsten Hopp
David Aquilina
Brent Baude
Mark Hamzy
Gustavo Luiz Duarte

+ special guests appearances from the Fedora release engineering,
infrastructure and the ARM teams

Lets dive into the various topics we dicussed and what came out of it.

F18 retrospective
-----------------

The main point here was to list the several good and bad things we noticed
during the F18 development so we could repeate the good ones and fix the bad
ones. Here the final list:

Good:
  - Managed to hit almost all target dates
  - Kept schedule in sync with primary
  - Almost passed final RC checklist for our RC4 (just 2 bugs)
  - Overall only 2 power specific bugs

Bad:
  - Signing was/is still an issue:
    o Reliability
    o Process
  - Detected Custom paritioning bug specific for Power very late during beta
    testing, resulting in a delay there
  - Multipath needed lots of cycles and attention to get working properly

Regarding the bad things we've had several discussions:

For the signing issues we had we've already started to address some of the
know issues during the development for F18. The main things that still 
bugged
us were the reliability and the overall process. For reliability we'll 
be
working with Miloslav Trmac to provide network dumps and error messages. 

Several from the team will also be at the DevConf in Brno in February to 
work
with him on this. 

Regarding the process issues we've discussed several options and came up 
with
the following flow: 

1) Sign packages for tag X. Returns list of all signed packages for tag X
2) Using this list run mesh to compose tree
3) Depending on tree (updates or release) create images and push bits

This will ensure that all packages that get pushed will always be signed on
the first go. It will run the chance that a push on a certain day might 
miss a
few packages that were just built, but we think the benefit from always 
only
pushing signed bits more than outweights this. 

Next up the late discovery of a critical bug for Beta. I really helped to
begin with that we stuck to the primary schedule as closely as possible. 
That
allowed us to get composes and images really early on. But in this 
specific
instance this didn't really help as it took us several days until we 
realized
that this was actually a Power specific issue (with Prep partitions), 
which
meant the fix for it would be late for Beta. On the other hand it 
allowed us
to test the updates.img functionality even on the install media. To 
ensure we
cover the most important scenarios we'll be moving towards using the 
testing
templates as primary already does, of course with slight modifications 
and
additions. Additionally providing a one-stop pointer on the arch wiki to 
learn
about the current state of builds, signing, composes and install tests 
using a
dashboard style will help keep everyone informed about all important 
stages.

Last up was multipath. We've hit that during the late Alpha testing and
noticed that multipath wasn't on any of the criterias during Beta testing.
We'll propose to add those to the criteria also for primary for upcoming
Fedora releases.

Supported archs/hardware for F19
--------------------------------

We discussed this for a bit and at the end came to the conculsion not to
do to much changing there compared to Fedora 18:
  - Compiler flags just as before for ppc64, generic 64bit optimization
  - For ppc we only provide Everything and Fedora trees, no more install 
media
    or images 

  - Following primary make ppc64 pure, meaning installs won't contain 
ppc
    packages anymore 

grub2 as bootloader on install media (DVD)
------------------------------------------

The driving factor here was that with switching to grub2 for Fedora 18 
we now
have a modern bootloader for Power that allows lots of cool things, but 
for
the media we've still used yaboot (similar to primary using syslinux as 
the
media bootloader). As it turns out we already are doing that on primary 

for UEFI images, so this has already gotten some testing and the 
necessary
code is already available. The plan is now to create experimental boot 
ISOs
with grub2 to see if they work properly and write a proposal post Fedora 
19 to
switch to grub2 and touch base with primary releng & anaconda in the 
time in
between to coordinate the efforts. 

Discuss use of test templates as used on primary:
https://fedoraproject.org/wiki/QA:Fedora_18_Install_Results_Template
--------------------------------------------------------------------

This went pretty quickly as we all agreed that this was a really good idea.
The list will certainly need some review and weeding/additions based on what
we've seen (e.g. multipath install test case). Additionally the final 
goal is
to link the release criteria with specific test cases where applicable. 
This
will make testing and checking if we meet the release criteria much 
easier.

Ideas on how to improve the visibility of the whole process (koji-shadow 
logs,
signing progress, updates, etc.) aka Dashboard 

---------------------------------------------- 

This topic was related to some of the issues we faced during Alpha and Beta
testing. The main goal for the discussion about this was to improve 
information
availability and visibility to everyone. So the idea is to create a 
dashboard
online that contains all following current information: 

  - Compose Status (last ran, logs) 

  - koji-shadow status (last ran, logs) 

  - Signing Status (last run, signed vs. unsigned #s) 

  - buildbranched status 

  - Failed builds 

  - Testing Status (IBM QA? AutoQA? ) 

Work on that has been started and David Aquilina has some of it already
available in text form.

Infrstructure review
--------------------

As we've had quite a few infrastructure changes over the last year we wanted
to take the time and review the current state and if there are any areas 
where
we need changes or improvements. Kevin Fenzi from the infrastructure 
team came
over to talk with us about it in an overall fashion. Main points coming 
out if
it were: 

  - EPEL needs action (new machine), especially to support RHEL-7 in the 
future
  - Overall no issues otherwise, capacity and setup works really well 

  - Storage might become an issue again, but NAS upgrade planed anyway 

  - Backups for important boxes (koji & composer) 

Continued optimization opportunities
------------------------------------

Just a brief look at what packages we currently compile as ppc64p7 and
brainstormed about a possible switch to ppc64p8 at some point.

SDK tutorial/demo - discussion on applicability for Fedora
----------------------------------------------------------

Unfortunately had to skip that due to lack of time.

Collaberative articles (brainstorm session)
-------------------------------------------

Talked about more articles for the next releases, e.g. followup articles for
mock: What are repositories? among others things.

Project page rewrite/re-do/redesign
-----------------------------------

As the main wiki for Power in fp.org got a bit cluttered over the past 3
releases we thought it was time to clean that up. Basic idea is to just 
have a
bit of immediate information on the main page and have separate sub 
pages for
specific details. 

EPEL discussion
---------------

Related to Infrstructure review there seems to be the wish to drop Power
support for EPEL at some point. Discussed again with Kevin what we'd need to
do to keep that working. Mainly need a current HW for the buildsystem 
for it.

Releng discussion w/ primary (process & schedule, checklist items)
------------------------------------------------------------------

We discussed various topics around release engineering with Dennis Gilmore
from rel-eng and Brendan Conoboy from the ARM teams.

Here the summary of the subtopics we had:

- Checklist/release criteria: Which does primary care about, which not?
         - As long as it's sane, primary doesn't care about our checklist
- Scheduling - what are the milestones we need to sync on? how long does 
it take for content to hit the mirrors?
         - Content has to be staged for mirrors on Friday for a Tuesday 
release
         - Must release on a Tuesday (preferrably announcing at 10a Eastern)
- Signing - messy & unreliable. What can we do to help debug?
         - Packet trace between client & bridge needed
         - bodhi messier than we thought (requires multiple sign & mash
           attempts even on primary)
         - just deal with it.
- koji garbage collection?
         - script still needs to be written to sync stages of gc b/w 
primary &
           secondary
- koji-shadow strategy?
         - shadowing stable and -updates-testing simultaneously is still 
sound
- -override tag sync between primary and secondary?
         - no complaints if we run our own version of the sync tag 
script for
           the override tags, but listening on the fedmsg message bus 
might be
           better (but would need new tooling)
- (infra?) how does primary monitor koji, storage, buildbranched, etc?
         - infra might be able to hook up basic system monitoring for us
         - cronjob output goes to dgilmore, should probably set up an
           arch-releng list.

Bug review
----------

Started doing that early Sunday but only did a brief overview and some 
initial
pruning of old bugs or bugs that we knew were already fixed or weren't
relevant anymore. Will do more individual pruning in the upcoming weeks via
irc meetings.

Statememt of Direction
----------------------

The Fedora Secondary Arch for Power team intends to focus and deliver the
following items for Fedora 19:
  - Deliver a release for Power systems installable on typical currently
    available 64bit hardware
  - All packages will be built with general 64bit Power support
  - Higher optimized builds for a select list of components available as
    additional ppc64p7 arch packages
  - Deliver 32bit compatibility packages for all components
  - No 32bit install trees and DVD ISOs
  - Sync development with the primary schedule
  - Ensure release criteria are met for all milestones

Thanks & regards, Phil

PS: I've also added this info the the wiki at 
https://fedoraproject.org/wiki/Architectures/PowerPC/Meetings/FUDCon_Lawrence_2013

-- 
Philipp Knirsch              | Tel.:  +49-711-96437-470
Manager Core Services        | Fax.:  +49-711-96437-111
Red Hat GmbH                 | Email: Phil Knirsch <pknirsch at redhat.com>
Wankelstrasse 5              | Web:   http://www.redhat.com/
D-70563 Stuttgart, Germany