New Fedora Packages Ready For Testing

[Brendan Early]

Hi all,

Fedora Packages Static is intended to replace the old Fedora Packages 
app and is now available for testing at 
https://packages.stg.fedoraproject.org. Any feedback would be 
appreciated here or on Pagure at https://pagure.io/fedora-packages-static.

The app is designed to function with the minimum amount of dynamic 
server components needed. Pages are statically generated using a Python 
script; the only dynamically generated page is the search result page. 
Data is updated hourly. Search is handled externally by an instance of 
Apache Solr. Clients do not need JS to use the app.

Thanks to Timothée Floure for the original version of Fedora Packages 
Static and Kevin Fenzi for helping deploy the app on OpenShift.


Brendan Early

[Zbigniew Jędrzejewski-Szmek]

On Sat, May 29, 2021 at 09:38:46PM -0500, Brendan Early wrote:
>  Hi all,
>  
>  Fedora Packages Static is intended to replace the old Fedora
>  Packages app and is now available for testing at
>  https://packages.stg.fedoraproject.org. Any feedback would be
>  appreciated here or on Pagure at
>  https://pagure.io/fedora-packages-static. 
Which pages is it supposed to replace? What is the relation to
https://src.fedoraproject.org/rpms/?

>  The app is designed to function with the minimum amount of dynamic
>  server components needed. Pages are statically generated using a
>  Python script; the only dynamically generated page is the search
>  result page. Data is updated hourly. Search is handled externally by
>  an instance of Apache Solr. Clients do not need JS to use the app. 
That sounds
good. I can confirm that the staging site seems snappy.

>  Thanks to Timothée Floure for the original version of Fedora
>  Packages Static and Kevin Fenzi for helping deploy the app on
>  OpenShift. 
Some initial comments:
- the search results page seems … messy. Would it be possible to
  group results by parent package? E.g.
  https://packages.stg.fedoraproject.org/search?query=systemd returns
  four pages of results, and a large number of entries is for
  subpackages. Grouping them would make it easier to find other
  interesting packages which are not part of the main package.

  Maybe also sort the results somehow? I think the method that dnf
  uses is pretty good: result are grouped by how they are matched:
  "Name Exactly Matched", "Name & Summary Matched", "Name
Matched",
  "Summary Matched". And within each group, results are sorted
  alphabetically. This gives a "clean" look (also because subpackages
  tend to be grouped together), gives the most relevant result first,
  and still makes it easy to scan the list.

- A search result for a subpackage links to a page for that subpackage.
  That's more confusing than useful:
  https://packages.stg.fedoraproject.org/pkgs/systemd-networkd/ shows
  the description for systemd-networkd, but all the information is actually
  for systemd source package.

  I think linking to a single page named after the source package
  would be more useful to users. Subpackages should be listed on this
  page. This is how Debian/Ubuntu to it. (It would also seem quite
  natural if the results were grouped by source package, as suggested
  above.) [EDIT: Oh, I see that this is done to an extent, the main
  package has "Subpackages" section. Unfortunately this doesn't take
  into account that different Fedora releases can have different sets
  of subpackages. This information is very important to users too.]

- Related to the above: source package names are unique. Binary package
  names are also unique for any given release of Fedora/EPEL/ELN/etc,
  but not unique globally. There's an epel-only systemd-extras source package
  which also builds systemd-networkd binary subpackage. But right 
  now the name is "occupied" by one of the subpackages, and no information
  is shown for the other one. It would be even worse if there was an
  epel-only systemd-networkd source package, because the main package
  couldn't be shown at all.

  Putting both binary and source packages under the same prefix
  https://packages.stg.fedoraproject.org/pkgs/ cannot work.

- Are the descriptions takes from some old release? For systemd I see
  "built from the 245.4-stable branch of systemd" while even f32 has
  245.9 now.

- Whitespace in descriptions is squished. E.g.
  https://packages.stg.fedoraproject.org/pkgs/systemd-swap/ looks very
  strange. Generally package descriptions are supposed to be preformatted
  text that fits in 80 columns. I don't think it should be wrapped at all.
  'dnf info systemd-swap' looks good;
  https://src.fedoraproject.org/rpms/systemd-swap does a reasonable job,
  except that it centers everything;
  https://packages.stg.fedoraproject.org/pkgs/systemd-swap/ is hard to
  read.

- There's "Recent Activity" at the bottom. For some packages this works,
  but for others it only shows "Loading…". I see "TypeError: NetworkError
  when attempting to fetch resource" shown for a fraction of a second.

- "SCM" links to a page that calls itself "fedora PACKAGE SOURCES"
and
  that everybody seems to refer to as "dist-git" or "pagure" in
  conversations. Maybe it would be better to call the link "sources"
  or "dist-git source" or "package source"?

  (And "FAF" links to "retrace.fedoraproject.org" page that calls
itself
  "ABRT Analytics". I don't even know what to suggest here ;| .)

- https://packages.stg.fedoraproject.org/pkgs/systemd-swap/ has a number
  of broken images in the "Recent Activity" part. Since that's generated
  content, it's not so easy to see what's going on…

I hope that's useful feedback. Thanks for working on this!

Zbyszek

[Fabio Valentini]

On Sun, May 30, 2021 at 11:59 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  - There's "Recent Activity" at the bottom. For some packages this works,
>    but for others it only shows "Loading…". I see "TypeError: NetworkError
>    when attempting to fetch resource" shown for a fraction of a second. 
It looks like this is a side effect of confusing source and binary
package names as well.
As far as I can tell, it only works "correctly" when binary and source
package name match. If they don't, it might display an error, or show
activity for an entirely different package (for example, "zincati"
shows activity for the retired "zincati" module, but not for the
actually still existing "rust-zincati" source package).

Fabio

[Brendan Early]

>  On Sun, May 30, 2021 at 11:59 AM Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  
>  It looks like this is a side effect of confusing source and binary
>  package names as well.
>  As far as I can tell, it only works "correctly" when binary and source
>  package name match. If they don't, it might display an error, or show
>  activity for an entirely different package (for example, "zincati"
>  shows activity for the retired "zincati" module, but not for the
>  actually still existing "rust-zincati" source package).
>   
That is a bug, it should be using the source package name.

Brendan

[Otto Urpelainen]

Zbigniew Jędrzejewski-Szmek kirjoitti 30.5.2021 klo 12.59:
>  On Sat, May 29, 2021 at 09:38:46PM -0500, Brendan Early wrote:
> > Hi all,
> >
> > Fedora Packages Static is intended to replace the old Fedora
> > Packages app and is now available for testing at
> > https://packages.stg.fedoraproject.org. Any feedback would be
> > appreciated here or on Pagure at
> > https://pagure.io/fedora-packages-static.
>  
>  Which pages is it supposed to replace? What is the relation to
>  https://src.fedoraproject.org/rpms/?
>   
(snip)
>  
>  - "SCM" links to a page that calls itself "fedora PACKAGE SOURCES"
> and
>     that everybody seems to refer to as "dist-git" or "pagure" in
>     conversations. Maybe it would be better to call the link "sources"
>     or "dist-git source" or "package source"?
>  
>     (And "FAF" links to "retrace.fedoraproject.org" page that calls
> itself
>     "ABRT Analytics". I don't even know what to suggest here ;| .) 
I think it would also be good to compare these links to basically 
identical links at the package sources [1]. The corresponding names 
there are:

Bodhi -> Updates Status
Bugzilla -> Bug Reports
FAF -> not included, why the difference?
Koji -> Builds Status
SCM -> not included, because it would link to the same page

and package source have two items not found in this app:

Packages -> should link to this app when this is deployed? currently 
returns 503
Koschei Status -> would make sense to have in this app also?

[1]: https://src.fedoraproject.org/rpms/firefox

Otto

[Brendan Early]

>  and package source have two items not found in this app:
>  
>  Packages -> should link to this app when this is deployed? currently 
>  returns 503 
Yes, when deployed I would like to setup a 301 redirect at the old url pointing to the new
app.

>  Koschei Status -> would make sense to have in this app also?
>   
It would; I will add a link.

Brendan

[Ankur Sinha]

Thanks for this! It's great! I've been really missing the packages app :)

I see that packages have a note of this form: "You can contact the
maintainers of this package at vit dash owner at fedoraproject dot org."

Do we want this information there? My worry is that it may result in
maintainers receiving e-mails with issues etc. instead of bug reports.

(Also, do we now prefer <package&gt;-maintainers(a)fp.o to <package>-owner?)

-- 
Thanks,
Regards,
Ankur Sinha "FranciscoD" (He / Him / His) |
https://fedoraproject.org/wiki/User:Ankursinha
Time zone: Europe/London

[Brendan Early]

>  Which pages is it supposed to replace? What is the relation to
>  https://src.fedoraproject.org/rpms/?
>  
It is unrelated to src.fedoraproject.org. This is replacing the former 
https://apps.fedoraproject.org/packages/. It was taken down permanently 
during the datacenter move.


>  Some initial comments:
>  - the search results page seems … messy. Would it be possible to
>    group results by parent package? E.g.
>  https://packages.stg.fedoraproject.org/search?query=systemd returns
>    four pages of results, and a large number of entries is for
>    subpackages. Grouping them would make it easier to find other
>    interesting packages which are not part of the main package.
>  
There seems to be a method to group things in Solr. The results are 
likely bloated because the query parser defaults to OR-ing multiple 
terms rather than dnf's default AND. I think that increasing the amount 
of results and using the shorter description should also make results 
more compact.


>    Maybe also sort the results somehow? I think the method that dnf
>    uses is pretty good: result are grouped by how they are matched:
>    "Name Exactly Matched", "Name & Summary Matched", "Name
>  Matched",
>    "Summary Matched". And within each group, results are sorted
>    alphabetically. This gives a "clean" look (also because subpackages
>    tend to be grouped together), gives the most relevant result first,
>    and still makes it easy to scan the list.
>  
I think that this specifically is not possible without hacking together 
multiple queries. This would be better accomplished through changing the 
search rankings via reranking or using the DisMax query parser on the 
package descriptions and names with appropriate weighting.


>  - A search result for a subpackage links to a page for that
> subpackage.
>    That's more confusing than useful:
>  https://packages.stg.fedoraproject.org/pkgs/systemd-networkd/ shows
>    the description for systemd-networkd, but all the information is  
actually
>    for systemd source package.
>  
>    I think linking to a single page named after the source package
>    would be more useful to users. Subpackages should be listed on this
>    page. This is how Debian/Ubuntu to it. (It would also seem quite
>    natural if the results were grouped by source package, as suggested
>    above.) [EDIT: Oh, I see that this is done to an extent, the main
>    package has "Subpackages" section. Unfortunately this doesn't take
>    into account that different Fedora releases can have different sets
>    of subpackages. This information is very important to users too.]
>  
I disagree here. systemd-networkd's metadata is significantly different 
from other subpackages for that source package. For example, the 
descriptions, provides, requires, and files (some of which is only 
visible after clicking a version number) for each subpackage are 
completely different. In some cases, such as texlive, even the versions 
for each subpackage differ. Clobbering all of this information into one 
page would be confusing for large source packages such as glibc. Every 
single subpackage would have to be enumerated along with a version table 
and description, completely hiding the recent activity section.

Additionally, it doesn't make sense to return something completely 
different that what the user asked for. E.g. if I lookup 
systemd-networkd, I should retrieve information only for that specific 
subpackage rather than show a list of all related subpackages. From the 
point of view of someone who does not understand how packages are built, 
this behavior would be extremely confusing. This is also the case with 
Debian from my understanding? You can search for a package in a specific 
release and the source package is only linked at the top left as an index.

I think the takeaway here is that everything needs to be grouped by 
source packages as you said. My actions on this will probably be:

- Change urls to /pkgs/<source pkg>/<subpackage>
- /pkgs/<source pkg> should become an index listing subpackages similar 
to suggested
- /pkgs/<source pkg>/<subpackage> will contain the current subpackage pages
- Subpackages pages will contain a "Related Packages" section that links 
to other subpackages for a source package
- If a subpackage matches the name of the source package, it is treated 
as the "primary" subpackage

Different releases are taken into account to my understanding. E.g. 
systemd-oomd-defaults only lists versions for F34 and rawhide whereas 
the systemd package lists more releases. I plan to introduce filtering 
in search based on what version of Fedora a package is available for.


>  - Related to the above: source package names are unique. Binary
> package
>    names are also unique for any given release of Fedora/EPEL/ELN/etc,
>    but not unique globally. There's an epel-only systemd-extras source  
package
>    which also builds systemd-networkd binary subpackage. But right
>    now the name is "occupied" by one of the subpackages, and no 
information
>    is shown for the other one. It would be even worse if there was an
>    epel-only systemd-networkd source package, because the main package
>    couldn't be shown at all.
>
>    Putting both binary and source packages under the same prefix
>  https://packages.stg.fedoraproject.org/pkgs/ cannot work.
>  
Above plans should resolve this.


>  - Are the descriptions takes from some old release? For systemd I
> see
>    "built from the 245.4-stable branch of systemd" while even f32 has
>    245.9 now.
>  
Descriptions should be taken from rawhide metadata. I will look into 
what is happening there.


>  - Whitespace in descriptions is squished. E.g.
>  https://packages.stg.fedoraproject.org/pkgs/systemd-swap/ looks very
>    strange. Generally package descriptions are supposed to be preformatted
>    text that fits in 80 columns. I don't think it should be wrapped at  
all.
>    'dnf info systemd-swap' looks good;
>  https://src.fedoraproject.org/rpms/systemd-swap does a reasonable job,
>    except that it centers everything;
>  https://packages.stg.fedoraproject.org/pkgs/systemd-swap/ is hard to
>    read.
>  
I will have it respect line breaks to resolve this, I would rather not 
use preformatted text for smaller devices like phones.


>  - There's "Recent Activity" at the bottom. For some
> packages this works,
>    but for others it only shows "Loading…". I see "TypeError: NetworkError
>    when attempting to fetch resource" shown for a fraction of a second. 
That's a bug, I need to have it listen for when the browser starts to 
navigate away and gracefully cancel that http request.


>  - "SCM" links to a page that calls itself "fedora
> PACKAGE SOURCES"
>  and
>    that everybody seems to refer to as "dist-git" or "pagure" in
>    conversations. Maybe it would be better to call the link "sources"
>    or "dist-git source" or "package source"?
>
>    (And "FAF" links to "retrace.fedoraproject.org" page that calls
>  itself
>    "ABRT Analytics". I don't even know what to suggest here ;| .) 
"FAF" (Fedora Analysis Framework) is the old name for ABRT Analytics to 
my understanding. I have been leaning towards just renaming all of those 
links to generic names such as "Problems" or "Crash Reports" as you
have 
suggested. Anyone who isn't a packager wouldn't understand what "Koji" 
and "Bodhi" are.


>  - https://packages.stg.fedoraproject.org/pkgs/systemd-swap/ has a
> number
>    of broken images in the "Recent Activity" part. Since that's generated
>    content, it's not so easy to see what's going on… 
The image URLs are pulled from datagrepper, I need to go and fix those 
upstream.


>  I hope that's useful feedback. Thanks for working on this!
I appreciate this, it is very detailed and exactly what I was looking for!


Brendan Early

[Miro Hrončok]

On 30. 05. 21 11:59, Zbigniew Jędrzejewski-Szmek wrote:
>  Some initial comments:
>  - the search results page seems … messy. Would it be possible to
>     group results by parent package? E.g.
>     https://packages.stg.fedoraproject.org/search?query=systemd  returns
>     four pages of results, and a large number of entries is for
>     subpackages. Grouping them would make it easier to find other
>     interesting packages which are not part of the main package.
>  
>     Maybe also sort the results somehow? I think the method that dnf
>     uses is pretty good: result are grouped by how they are matched:
>     "Name Exactly Matched", "Name & Summary Matched", "Name
> Matched",
>     "Summary Matched". And within each group, results are sorted
>     alphabetically. This gives a "clean" look (also because subpackages
>     tend to be grouped together), gives the most relevant result first,
>     and still makes it easy to scan the list. 
Some anecdata. I've tried to search for python3.9 which is an exact name of a 
package I co-maintain.

https://packages.stg.fedoraproject.org/search?query=python3.9

It found 6997 results but the python3.9 package is not on the first 10 pages 
(and I suspect it's a problem with python3.9 being a component?).

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Brendan Early]

On 5/31/21 3:52 AM, Miro Hrončok wrote:
>  Some anecdata. I've tried to search for python3.9 which is an
> exact 
>  name of a package I co-maintain.
>
>  https://packages.stg.fedoraproject.org/search?query=python3.9
>
>  It found 6997 results but the python3.9 package is not on the first 10 
>  pages (and I suspect it's a problem with python3.9 being a component?).
>  
Yes, it currently searches on built package names. If you view 
https://packages.stg.fedoraproject.org/pkgs/python3/, it will state that 
it is a "Subpackage of python3.9". This should hopefully change after I 
implement the search changes.