[fedora-arm] ExcludeArch tracker doesn't appear to be effective

[Matthew Garrett]

On Tue, Jun 10, 2014 at 10:52:19PM +0100, Peter Robinson wrote:
> On Tue, Jun 10, 2014 at 10:20 PM, Matthew Garrett <mjg59 at srcf.ucam.org> wrote:
> > In the past 6 months, 6 bugs added, 2 bugs closed -
> > https://bugzilla.redhat.com/show_activity.cgi?id=485251 .
> 
> If you're going on just the bug tracker possibly but there's a lot of
> stuff we fix and enhance that doesn't even make the that tracker, the
> Ada stuff I mentioned earlier is but one example. Rightly or wrongly
> it's not the canonical source of information. For example if I
> discover an exclude arch I'll generally just go and fix it rather than
> spending the time to open a bug, fix it myself and then close it
> myself. Just go and read the rawhide reports. If that's what you want
> us to do to give a warm fuzzy feeling of progress.... I don't see it
> as a time effective agile way of necessarily dealing with it. Feel
> free to suggest alternatives :-)

Ok, I was entirely unaware of that, and it does change things. Thanks 
for letting me know. I'll look into whether it's practical to generate a 
list of all the existing ExcludeArch packages and automatically check 
whether they have tracker bugs filed - does that seem helpful? It 
*would* be good to have meaningful metrics on this, but I don't want 
there to be excessive process overhead.

> > Where there's reliance on specific hardware functionality that's absent
> > then yes, absolutely, there's no benefit in building the packages.
> > Figuring out how to communicate that to users isn't an entirely solved
> > problem, but with luck nobody's actually buying ARM hardware with
> > unrealistic expectations of its functionality.
> >
> > But I can't find any examples of those in the tracker. They all seem to
> > be cases where packages are either missing dependencies, take too long
> > to build or are just missing support. They're code. We can fix them.
> 
> Are you offering to do so? For example some time ago I approached one
> of the root maintainers (comes out of CERN I believe) and they said
> they didn't see the point nor have the time to maintain anything other
> than x86 at this time. Should we fork it and support it just to say
> we're "feature complete with x86" just because it's code? Well maybe,
> but I'm not sure in that case it would add value to the distro for the
> expenditure of time and we've had exactly zero enquires about it (and
> it's 3 dependencies in the tracker). In the case of Hadoop there might
> possibly be value (and in later releases the issues might even have
> been fixed) but again we've had no queries and so we've focused on
> things people want. In the case of Ada we had some requests for
> support since F-20 was released so we've added that for F-21.

Yeah, I think where it's practical for us to maintain feature 
compatibility we should do so even if upstream disagree. We make 
modifications to upstream code all the time in order to meet Fedora 
policy.

As for who should be doing that work - well, yeah. That's kind of my 
point. Responsibility for this kind of thing is really something we 
should have figured out prior to promotion, and I'm sorry that I didn't 
think about it in a more timely manner. I'm looking for answers here, 
not insisting that anyone take on more work.

> > I'm genuinely sorry if I gave the impression of bullying here. I want to
> > feel comfortable pointing at the ARM port as an equal to the x86_64 one.
> > I don't feel entirely comfortable doing so at the moment, and the
> > current process doesn't seem to be getting us to the point where I would
> > be.
> 
> Well you need to engage with us better. Every time you make an
> appearance it appears to us all it's just to bully us. People
> genuinely shudder when your nick appears on on the channel and I've
> had 3 people thank me for replying so they don't have to.

That's unfortunate. I'm sorry. I'll try to ensure that I'm interacting 
in a more productive way.

> So moving on from that.... why don't you feel comfortable pointing to
> the ARM port? I'm aware we're still not perfect but it was always
> going to be a road to improvement.

Basically that you're still not perfect, to the extent that anything in 
Fedora is. I'd have been significantly happier if more time had been 
spent on, for instance, Anaconda before ARM was promoted. But 
realistically there's obviously a tension between perfectionism and 
maintaining enthusiasm - especially with the longer F21 cycle, I suspect 
the right decisions were made at the time.

> We're supporting massively more hardware now than we were at F-20
> release time and the types of HW are getting better with the likes of
> the Tegra K1 supporting full desktop equivalent graphics and we might
> even have the support for that upstream in time for F-21 release, the
> open graphics drivers are improving that they might be usable in that
> timeframe too. Those were problems that were well known when we went
> to primary and the path is basically what was agreed and mapped out in
> that regards.

The work that's been done on the open graphics stack is immensely 
exciting.

> In terms of the general userspace I'm not aware of anywhere we
> massively differ in feature set of packages or features within those
> packages. Any time someone finds an area where we differ we fix up.
> ExcludeArch packages are the easy ones to find, it's almost impossible
> to find internal package ifarch issues for features without a manual
> grep or similar of all .spec files and we fix when we find, if you've
> got suggestions to improve how we deal with those I'm open to
> suggestions rather than pure criticism but again this isn't the ARM
> team actively trying to sweep problems under the carpet.

Again, I don't want to imply that you've been behaving inappropriately 
here. I have no suggestions for how any individual involved in the 
process could have done a better job. The information I had (the tracker 
bug that's supposed to be, well, a tracker...) turns out to have been 
incomplete, and the situation may well not be anywhere near as bad as 
I'd thought. I'll try to do a better job of obtaining that information.

If it turns out that there's strong progress being made towards getting 
rid of all the ExcludeArchs that are practical to drop, that's awesome. 
I've been worried about nothing. If not, it's still a problem we need to 
solve. I'm not trying to find people to blame for the current situation. 

> > The quantity of the archive that's built and working on ARM so far is a
> > testament to the amount of effort that the ARM community have put into
> > this port. The question is how to finish that. All I'm saying here is
> > that the current approach of filing bugs doesn't appear to be resulting
> > in people actually fixing their packages. It's unreasonable to expect
> > you to do all of it. So what do we do?
> 
> I'm not sure but then I don't see you've actually contributed anything
> to improving it either as yet, I'd love to hear of some positive
> suggestions as I suspect it would also help the other non x86 non
> mainline arches too.

Well right. Figuring this out now is going to make life significantly 
easier for ppc. It's bad luck on your side that you ended up going 
through this process first - I'm sure it'll be smoother for whoever 
follows (and hey, just think how much easier arm64 is going to be in 
comparison).

I don't have concrete suggestions right now. If there is a real problem 
then it's not going to be an easy one to solve, but neither is it one we 
can ignore.
 
> I'd also love to hear of your suggestions of how we might encourage
> more contributors to ARM. We know we have a lot of consumers of Fedora
> ARM (I know of at least one installation of 100K devices) but they
> don't necessarily contribute (apparently they had no issues for their
> use case when I spoke with them about it) and while we have HW vendors
> that actively test their HW and occasionally pop up here and there
> they only care about their bits. We've tried HW giveaways but in most
> cases people grab and run. So it varies. Thoughts and suggestions
> welcome :-)

Rough ideas:

1) Once we have a chance, attempt a rebuild of everything that's 
currently ExcludeArch: arm. See if any succeed. If so, do a pass and see 
which of them are obviously hardware-related. File bugs on the rest of 
the ones that built asking the maintainer whether the ExcludeArch is 
still legtimate.

2) As part of the magic AutoQA future, ensure that any added ExcludeArch 
references a bug and that said bug is attached to the appropriate 
tracker.

3) Tag ExcludeArch bugs to indicate whether the reason is missing 
hardware support, missing dependencies, missing functionality in the 
codebase (eg, a package that contains x86 assembly but no ARM port) or 
whether there's just a bug.

These obviously don't actually fix most of the bugs. But they do make it 
easier for someone interested to just jump in, and reducing the barrier 
to entry makes contributions more likely.

-- 
Matthew Garrett | mjg59 at srcf.ucam.org