Re: Packaging guidelines - validation of AppStream metadata files
On Mon, Sep 25, 2023 at 1:45 PM Alexander Ploumistos
<alex.ploumistos(a)gmail.com> wrote:
On Mon, Sep 25, 2023 at 7:33 PM Artur Frenszek-Iwicki
<suve(a)fedoraproject.org> wrote:
>
> Without docs, the whole process is a black box.
+1
That was actually one of the two reasons I started this thread.
For instance, is this the package that must be rebuilt in order to see
an AppStream metadata file appear/refreshed in GNOME Software?
https://src.fedoraproject.org/rpms/appstream-data
Could we get an explainer (for dummies) of how the entire pipeline works?
The other reason was someone from Debian telling one of my upstreams
"Don't use appstream-util to validate the file, it's deprecated, use
appstreamcli". Frankly, that struck a chord, considering x years ago I
had spent a considerable amount of time convincing dozens of projects
to adopt "AppData" (at the time), filing tickets and submitting PRs.
So here's some backstory...
Way back in the days of yore (about 10 years ago), people were working
on a cross-distro Software Center:
https://fedoraproject.org/wiki/Features/SoftwareCenter
For that "software center", the AppData (renamed to AppStream) format
was created initially to port over desktop files to XML for more
reliable parsing at scale. The goal here was to create a cheaper and
more reliable method of providing rich metadata in repositories that
could be indexed and used to present useful information about
applications.
My understanding is that the original AppStream implementation at the
time was moribund due to its maintainer being in school, so a new one
was made (unhelpfully named appstream-glib... why "unhelpfully" you
ask? well, the original appstream was *also* glib based!). This got
plugged into what became GNOME Software. And because Fedora needed a
tool to generate the indexes by scanning all the RPMs,
appstream-builder was created to do that. (As an aside, appstream-glib
was never used by KDE because there was no Qt/C++ binding for it, but
the original AppStream did have one, so that's what Plasma Discover
has always used.)
But there was a problem: unlike RPM repodata generation, AppStream
repodata generation is slow. Really slow. This is because AppStream
repodata requires opening up the RPM, plucking data out of it, and
applying the necessary transformations per repodata generation policy
(downloading remote images, recompression, etc.). There are arguments
about whether it's appstream-builder specific (given that Debian's
archive is bigger, their package format is more complex to "pluck"
from, etc and appstream-generator does fine there), but the point is
that for Fedora with appstream-builder, it's too slow. Note that the
context of this is back in the era where Fedora composes took half a
day or longer. Adding even more hours to that process was not
appealing (it would have to be done for every repo being composed for
every architecture...).
So a stopgap solution was implemented: appstream-data. Richard Hughes
maintains a local mirror of the full Fedora repositories and generates
the appstream data from that using scripts[1] and extra data[2] to
produce the appstream-data package[3]. The goal is to someday get to a
point where we can have this done properly as repodata through
createrepo_c[4], but we're not able to do that yet.
Now, appstream-util is the appstream-glib equivalent of appstreamcli.
We test using appstream-util because we need the AppStream data to be
valid for appstream-builder to consume it. If it is not, the
application will be skipped in the next round of appstream repodata
updates. If we ever move to a process that uses libappstream-compose
as the backend library for producing AppStream repodata, then we can
stop caring about appstream-util. But for now, here we are.
[1]: https://github.com/hughsie/appstream-scripts
[2]: https://github.com/hughsie/fedora-appstream
[3]: https://src.fedoraproject.org/rpms/appstream-data
[4]: https://github.com/rpm-software-management/createrepo_c/issues/75
--
真実はいつも一つ！/ Always, there's only one truth!