[fedora-arm] ExcludeArch tracker doesn't appear to be effective

[Peter Robinson]

>> If you're going on just the bug tracker possibly but there's a lot of
>> stuff we fix and enhance that doesn't even make the that tracker, the
>> Ada stuff I mentioned earlier is but one example. Rightly or wrongly
>> it's not the canonical source of information. For example if I
>> discover an exclude arch I'll generally just go and fix it rather than
>> spending the time to open a bug, fix it myself and then close it
>> myself. Just go and read the rawhide reports. If that's what you want
>> us to do to give a warm fuzzy feeling of progress.... I don't see it
>> as a time effective agile way of necessarily dealing with it. Feel
>> free to suggest alternatives :-)
>
> Ok, I was entirely unaware of that, and it does change things. Thanks
> for letting me know. I'll look into whether it's practical to generate a
> list of all the existing ExcludeArch packages and automatically check
> whether they have tracker bugs filed - does that seem helpful? It
> *would* be good to have meaningful metrics on this, but I don't want
> there to be excessive process overhead.

I agree metrics would be useful for all involved.

>> > Where there's reliance on specific hardware functionality that's absent
>> > then yes, absolutely, there's no benefit in building the packages.
>> > Figuring out how to communicate that to users isn't an entirely solved
>> > problem, but with luck nobody's actually buying ARM hardware with
>> > unrealistic expectations of its functionality.
>> >
>> > But I can't find any examples of those in the tracker. They all seem to
>> > be cases where packages are either missing dependencies, take too long
>> > to build or are just missing support. They're code. We can fix them.
>>
>> Are you offering to do so? For example some time ago I approached one
>> of the root maintainers (comes out of CERN I believe) and they said
>> they didn't see the point nor have the time to maintain anything other
>> than x86 at this time. Should we fork it and support it just to say
>> we're "feature complete with x86" just because it's code? Well maybe,
>> but I'm not sure in that case it would add value to the distro for the
>> expenditure of time and we've had exactly zero enquires about it (and
>> it's 3 dependencies in the tracker). In the case of Hadoop there might
>> possibly be value (and in later releases the issues might even have
>> been fixed) but again we've had no queries and so we've focused on
>> things people want. In the case of Ada we had some requests for
>> support since F-20 was released so we've added that for F-21.
>
> Yeah, I think where it's practical for us to maintain feature
> compatibility we should do so even if upstream disagree. We make
> modifications to upstream code all the time in order to meet Fedora
> policy.

I agree it's worth the effort where there's the demand for it, we
already do this for some packages on ARM where upstream don't support
it because people want to use it. I don't really see the point in
expending limited resources where there's no demand.

> As for who should be doing that work - well, yeah. That's kind of my
> point. Responsibility for this kind of thing is really something we
> should have figured out prior to promotion, and I'm sorry that I didn't
> think about it in a more timely manner. I'm looking for answers here,
> not insisting that anyone take on more work.
>
>> > I'm genuinely sorry if I gave the impression of bullying here. I want to
>> > feel comfortable pointing at the ARM port as an equal to the x86_64 one.
>> > I don't feel entirely comfortable doing so at the moment, and the
>> > current process doesn't seem to be getting us to the point where I would
>> > be.
>>
>> Well you need to engage with us better. Every time you make an
>> appearance it appears to us all it's just to bully us. People
>> genuinely shudder when your nick appears on on the channel and I've
>> had 3 people thank me for replying so they don't have to.
>
> That's unfortunate. I'm sorry. I'll try to ensure that I'm interacting
> in a more productive way.
>
>> So moving on from that.... why don't you feel comfortable pointing to
>> the ARM port? I'm aware we're still not perfect but it was always
>> going to be a road to improvement.
>
> Basically that you're still not perfect, to the extent that anything in
> Fedora is. I'd have been significantly happier if more time had been
> spent on, for instance, Anaconda before ARM was promoted. But
> realistically there's obviously a tension between perfectionism and
> maintaining enthusiasm - especially with the longer F21 cycle, I suspect
> the right decisions were made at the time.
>
>> We're supporting massively more hardware now than we were at F-20
>> release time and the types of HW are getting better with the likes of
>> the Tegra K1 supporting full desktop equivalent graphics and we might
>> even have the support for that upstream in time for F-21 release, the
>> open graphics drivers are improving that they might be usable in that
>> timeframe too. Those were problems that were well known when we went
>> to primary and the path is basically what was agreed and mapped out in
>> that regards.
>
> The work that's been done on the open graphics stack is immensely
> exciting.
>
>> In terms of the general userspace I'm not aware of anywhere we
>> massively differ in feature set of packages or features within those
>> packages. Any time someone finds an area where we differ we fix up.
>> ExcludeArch packages are the easy ones to find, it's almost impossible
>> to find internal package ifarch issues for features without a manual
>> grep or similar of all .spec files and we fix when we find, if you've
>> got suggestions to improve how we deal with those I'm open to
>> suggestions rather than pure criticism but again this isn't the ARM
>> team actively trying to sweep problems under the carpet.
>
> Again, I don't want to imply that you've been behaving inappropriately
> here. I have no suggestions for how any individual involved in the
> process could have done a better job. The information I had (the tracker
> bug that's supposed to be, well, a tracker...) turns out to have been
> incomplete, and the situation may well not be anywhere near as bad as
> I'd thought. I'll try to do a better job of obtaining that information.
>
> If it turns out that there's strong progress being made towards getting
> rid of all the ExcludeArchs that are practical to drop, that's awesome.
> I've been worried about nothing. If not, it's still a problem we need to
> solve. I'm not trying to find people to blame for the current situation.
>
>> > The quantity of the archive that's built and working on ARM so far is a
>> > testament to the amount of effort that the ARM community have put into
>> > this port. The question is how to finish that. All I'm saying here is
>> > that the current approach of filing bugs doesn't appear to be resulting
>> > in people actually fixing their packages. It's unreasonable to expect
>> > you to do all of it. So what do we do?
>>
>> I'm not sure but then I don't see you've actually contributed anything
>> to improving it either as yet, I'd love to hear of some positive
>> suggestions as I suspect it would also help the other non x86 non
>> mainline arches too.
>
> Well right. Figuring this out now is going to make life significantly
> easier for ppc. It's bad luck on your side that you ended up going
> through this process first - I'm sure it'll be smoother for whoever
> follows (and hey, just think how much easier arm64 is going to be in
> comparison).

We're already well aware of that and in the package fixing I've been
doing right from the word go I've always taken all secondary arches
into account in the fixes I apply, there's not enough people as it is
so why have multiple people do the same thing when one person can fix
it once.

> I don't have concrete suggestions right now. If there is a real problem
> then it's not going to be an easy one to solve, but neither is it one we
> can ignore.

>From my point of view the vast majority of packages that we have
problems with are the ones that aren't actively maintained in mainline
Fedora and a lot of the issues I have would be fixed with the latest
version of the package in the distro and I spend a lot of time dealing
with those issues. For example and entire stack that I've avoided is
mono which is basically unmaintained AFAICT. mono 3.x improves ARM
support massively, I've thought of updating it for the ARM support but
then do I have to deal with all the 100s of packages that have
bindings and if it breaks stuff on x86 do I then have to fix those due
to an AWOL maintainer? Ultimately I don't believe that is fair but in
it's current state it's a sub par experience even on x86 and it could
conceivably be argued that it would be best to drop it entirely from
the distro. If I go back through my commit logs there's literally 100s
of other examples, when I touch a package I also check the BZ queue
for it to see if there's other easy fixes. Ultimately improving the
mainline maintainership issues would make it easier for secondary as
often the newer releases "just work"

>> I'd also love to hear of your suggestions of how we might encourage
>> more contributors to ARM. We know we have a lot of consumers of Fedora
>> ARM (I know of at least one installation of 100K devices) but they
>> don't necessarily contribute (apparently they had no issues for their
>> use case when I spoke with them about it) and while we have HW vendors
>> that actively test their HW and occasionally pop up here and there
>> they only care about their bits. We've tried HW giveaways but in most
>> cases people grab and run. So it varies. Thoughts and suggestions
>> welcome :-)
>
> Rough ideas:
>
> 1) Once we have a chance, attempt a rebuild of everything that's
> currently ExcludeArch: arm. See if any succeed. If so, do a pass and see
> which of them are obviously hardware-related. File bugs on the rest of
> the ones that built asking the maintainer whether the ExcludeArch is
> still legtimate.

We basically do something very similar to that now, see 0ad (a game I
believe) as a recent example where issues were fixed upstream that
allowed us to enable it on ARM

> 2) As part of the magic AutoQA future, ensure that any added ExcludeArch
> references a bug and that said bug is attached to the appropriate
> tracker.

It would be nice to have some automated means of determining this, in
some cases I've seen maintainers put ExclusiveArch just because they
couldn't be bothered to deal with it.

> 3) Tag ExcludeArch bugs to indicate whether the reason is missing
> hardware support, missing dependencies, missing functionality in the
> codebase (eg, a package that contains x86 assembly but no ARM port) or
> whether there's just a bug.

It would be useful for all arches to have an easy way of dealing with
this. Whether it be via BZ or some of the other Fedora infra web apps.

> These obviously don't actually fix most of the bugs. But they do make it
> easier for someone interested to just jump in, and reducing the barrier
> to entry makes contributions more likely.

Yep, don't disagree.