Shell Completions

[Maxwell G]

Hi everyone,

Currently, packaging shell completions is a bit of a hassle. You have to
memorize a bunch of long directory names (`%{_datadir}/bash-completion/
completions`, `%{_datadir}/fish/vendor_completions.d`, and `%{_datadir}/zsh/
site-functions`), and repeat them multiple times in the specfile. For some
reason, the bash completion directories are owned by filesystem, but zsh and
fish's shell completions directories are not. Therefore, every package needs
to copy

```
%files
[...]
%{_datadir}/bash-completion/completions/%{name}
%dir %{_datadir}/fish
%dir %{_datadir}/fish/vendor_completions.d
%{_datadir}/fish/vendor_completions.d/%{name}.fish
%dir %{_datadir}/zsh
%dir %{_datadir}/zsh/site-functions
%{_datadir}/zsh/site-functions/_%{name}
```

or own each shell's respective root directory and not bother with `%dir`
and explicit paths. I don't like doing the latter, as I prefer to be explicit
and follow the spirit of https://docs.fedoraproject.org/en-US/packaging-guidelines/#_explicit_lists.

I was thinking of creating an sh-completions-packaging package (name
suggestions welcome) that provides a subpackage containing macros for each of
those directories that would be part of the default buildroot and a filesystem
subpackage that owns these directories so packages can Require it and not have
to copy this long boiler plate.

I would appreciate any feedback you have, particularly regarding naming of the
packages and macros. For the macros, would `%{_bashcompdir}`,
`%{_zshcompdir}`, and `%{_fishcompdir}` be appropriate macro names? I'm a bit
unclear about macro namespacing conventions.

Even if you don't like this idea, I think the guidelines could at least use
some clarification in this area. I have seen or dealt with multiple packages
that install shell completions to the wrong directories, don't properly own the
directories, or have other problems.

[Miro Hrončok]

On 24. 06. 22 21:26, Maxwell G wrote:
> Hi everyone,
>
> Currently, packaging shell completions is a bit of a hassle. You have to
> memorize a bunch of long directory names (`%{_datadir}/bash-completion/
> completions`, `%{_datadir}/fish/vendor_completions.d`, and `%{_datadir}/zsh/
> site-functions`), and repeat them multiple times in the specfile. For some
> reason, the bash completion directories are owned by filesystem, but zsh and
> fish's shell completions directories are not. Therefore, every package needs
> to copy
> %files
> [...]
> %{_datadir}/bash-completion/completions/%{name}
> %dir %{_datadir}/fish
> %dir %{_datadir}/fish/vendor_completions.d
> %{_datadir}/fish/vendor_completions.d/%{name}.fish
> %dir %{_datadir}/zsh
> %dir %{_datadir}/zsh/site-functions
> %{_datadir}/zsh/site-functions/_%{name}
>
> or own each shell's respective root directory and not bother with `%dir`
> and explicit paths. I don't like doing the latter, as I prefer to be explicit
> and follow the spirit of https://docs.fedoraproject.org/en-US/packaging-guidelines/#_explicit_lists.
>
> I was thinking of creating an sh-completions-packaging package (name
> suggestions welcome) that provides a subpackage containing macros for each of
> those directories that would be part of the default buildroot and a filesystem
> subpackage that owns these directories so packages can Require it and not have
> to copy this long boiler plate.
>
> I would appreciate any feedback you have, particularly regarding naming of the
> packages and macros. For the macros, would `%{_bashcompdir}`,
> `%{_zshcompdir}`, and `%{_fishcompdir}` be appropriate macro names? I'm a bit
> unclear about macro namespacing conventions.
>
> Even if you don't like this idea, I think the guidelines could at least use
> some clarification in this area. I have seen or dealt with multiple packages
> that install shell completions to the wrong directories, don't properly own the
> directories, or have other problems.

I welcome the idea. Some suggestions:

1) Consider defining the macros in redhat-rpm-config -- I do not except they 
would change often and would need a separate component -- new component might 
get dropped from RHEL 10 because RHEL maintainers want to get rid of 
components, it also means more metadata and generally is a waste of resources.

2) If you insist on new component, I'd name it shell-completions-rpm-macros 
and the built package shell-completions-srpm-macros if it is required by 
redhat-rpm-config.

3) If you want the filesystem stuff to be subpackages, consider adding a 
dependency generator (package installs files to Bash completion dir -> depends 
on shell-completions-filesystem-bash) -- I can help write it, but see e.g. 
https://src.fedoraproject.org/rpms/python-rpm-generators/blob/rawhide/f/pyth... 
for a similar example

4) If you go with your own component and subpackages, consider making the 
filesystem subpackages part of this component, so it's easier to do changes 
atomicaly at one place.

4) However, I'd consider adding the directories to filesystem directly to 
make it simpler and avoid any new dependencies. It's less cool, but also less code.

5) %{bash_completions_dir}, %{fih_completions_dir}, %{zsh_completions_dir}
  or %{bash_completions}, %{fih_completions}, %{zsh_completions}

[Maxwell G]

Hi Miro,

Thanks for the feedback!

On Saturday, June 25, 2022 5:16:32 AM CDT Miro Hrončok wrote:
> Consider defining the macros in redhat-rpm-config

Yeah, I did think about this, but I figured I'd prefer to have them in a 
package that I actually control. This is probably less of a problem for you as 
a maintainer of redhat-rpm-config ;-). Also, if I want to backport them to 
EPEL, I would have to copy the same thing to epel-rpm-macros and maintain them 
in two places.
> If you insist on new component, I'd name it
>
> shell-completions-rpm-macros
> and the built package shell-completions-srpm-macros if it is required by
> redhat-rpm-config.

I'm fine with replacing sh with shell, but what's the point of the `srpm-
macros` part? At least based on what we've discussed so far, there's not 
anything in here that's required to build a SRPM.
> If you want the filesystem stuff to be subpackages, consider adding a
>
> dependency generator (package installs files to Bash completion dir ->
> depends on shell-completions-filesystem-bash) -- I can help write it, but
> see e.g.
> https://src.fedoraproject.org/rpms/python-rpm-generators/blob/rawhide/f/pyt
> hon.attr for a similar example

Thanks for the offer. I did consider writing one, and I do have some 
experience with modifying the one that @ignatenkobrain wrote for ansible, but 
I thought it might be overcomplicating the problem (which I have a tendency to 
do :)). I think your other idea is better, though.

I do think I'd like to write an analogue to `%pyproject_save_files` that would 
accept 1 or more completions name globs and find the appropriately named files 
in the appropriate directories (name for bash, _name for zsh, name.fish for 
fish). %files can accept multiple `-f` arguments (I tested it), so it 
shouldn't be a problem for packages to use both.
> If you go with your own component and subpackages, consider making the
>
> filesystem subpackages part of this component, so it's easier to do changes
> atomicaly at one place.

Yeah, my idea was to create a single component named sh-completions-packaging 
(but I can `s/sh/shell/` as you suggested).
> However, I'd consider adding the directories to filesystem directly to
>
> make it simpler and avoid any new dependencies. It's less cool, but also
> less code.

I agree. The bash completions dir is already owned by filesystem, so we might 
as well add the other ones. This is also simpler for me/packagers (I wouldn't 
have to write a dependency generator/they wouldn't have to Require the new 
package).

The only wrinkle I see is handling this for EPEL. RHEL might take a PR to own 
the zsh completions directory in filesystem, but I doubt they'd own fish 
directories as fish is only part of EPEL.

[Miro Hrončok]

On 25. 06. 22 19:32, Maxwell G wrote:
> Hi Miro,
>
> Thanks for the feedback!
>
> On Saturday, June 25, 2022 5:16:32 AM CDT Miro Hrončok wrote:
> > Consider defining the macros in redhat-rpm-config
>
> Yeah, I did think about this, but I figured I'd prefer to have them in a
> package that I actually control. This is probably less of a problem for you as
> a maintainer of redhat-rpm-config ;-). Also, if I want to backport them to
> EPEL, I would have to copy the same thing to epel-rpm-macros and maintain them
> in two places.

OK, makes sense. BTW I am not a  maintainer of redhat-rpm-config.
> > If you insist on new component, I'd name it
> >
> > shell-completions-rpm-macros
> > and the built package shell-completions-srpm-macros if it is required by
> > redhat-rpm-config.
>
> I'm fine with replacing sh with shell, but what's the point of the `srpm-
> macros` part? At least based on what we've discussed so far, there's not
> anything in here that's required to build a SRPM.

My understanding was that the new macros would exist in the default buildroot, 
hence live in either redhat-rpm-config or in a package required by 
redhat-rpm-config. Such packages should generally be called -srpm-macros. Yes, 
technically such macros would not be needed to build a SRPM.

OTOH we *could* make it non-default, but that requires:

BuildRequires: shell-completions-rpm-macros / shell-completions-packaging

...to use them.
> > If you want the filesystem stuff to be subpackages, consider adding a
> >
> > dependency generator (package installs files to Bash completion dir ->
> > depends on shell-completions-filesystem-bash) -- I can help write it, but
> > see e.g.
> > https://src.fedoraproject.org/rpms/python-rpm-generators/blob/rawhide/f/pyt
> > hon.attr for a similar example
>
> Thanks for the offer. I did consider writing one, and I do have some
> experience with modifying the one that @ignatenkobrain wrote for ansible, but
> I thought it might be overcomplicating the problem (which I have a tendency to
> do :)). I think your other idea is better, though.

Ack.
> I do think I'd like to write an analogue to `%pyproject_save_files` that would
> accept 1 or more completions name globs and find the appropriately named files
> in the appropriate directories (name for bash, _name for zsh, name.fish for
> fish). %files can accept multiple `-f` arguments (I tested it), so it
> shouldn't be a problem for packages to use both.

Note that %pyproject_save_files reads upstream metadata, not files on disk. 
Maybe doing this is an overkill?

Consider this:

%files
...
%{bash_completions}/foo

Vs:

%install
...
%save_bash_completions foo

%files -f %{bash_completions}
...
> > If you go with your own component and subpackages, consider making the
> >
> > filesystem subpackages part of this component, so it's easier to do changes
> > atomicaly at one place.
>
> Yeah, my idea was to create a single component named sh-completions-packaging
> (but I can `s/sh/shell/` as you suggested).
> > However, I'd consider adding the directories to filesystem directly to
> >
> > make it simpler and avoid any new dependencies. It's less cool, but also
> > less code.
>
> I agree. The bash completions dir is already owned by filesystem, so we might
> as well add the other ones. This is also simpler for me/packagers (I wouldn't
> have to write a dependency generator/they wouldn't have to Require the new
> package).

+1
> The only wrinkle I see is handling this for EPEL. RHEL might take a PR to own
> the zsh completions directory in filesystem, but I doubt they'd own fish
> directories as fish is only part of EPEL.

Introduce a filesystem-epel package?

[Maxwell G]

Hi,

On Saturday, June 25, 2022 3:18:55 PM CDT Miro Hrončok wrote:
> My understanding was that the new macros would exist in the default
> buildroot, hence live in either redhat-rpm-config or in a package required
> by
> redhat-rpm-config. Such packages should generally be called -srpm-macros.

Yeah, I suppose you're right. All of the macros packages required by redhat-
rpm-config are named X-srpm-macros except rpmautospec-rpm-macros.
> OTOH we *could* make it non-default, but that requires:
>
> BuildRequires: shell-completions-rpm-macros / shell-completions-packaging
> ...to use them.

I'd prefer them to be in the buildroot.
> Note that %pyproject_save_files reads upstream metadata, not files on disk.
> Maybe doing this is an overkill?
>
> Consider this:
>
> %files
> ...
> %{bash_completions}/foo
>
> Vs:
>
> %install
> ...
> %save_bash_completions foo
>
> %files -f %

Well,

```
%install
[...]
%sh_completions_save_files X

%files -f %{sh_completions_files}
```

is shorter than

```
%files
%{bash_completions_dir}/X
%{zsh_completions_dir}/_X
%{fish_completions_dir}/X.fish
```

but even with that slightly larger difference, it still probably isn't worth 
the trouble. It would be more useful if we could automate installing them 
also, but that wouldn't work well due to the varying ways these files are 
generated and organized in upstream repos.
> Introduce a filesystem-epel package?

+1. I guess we could add that as a subpackage to epel-release.

Thanks again for the advice. If it's really only three static macros that I'm 
adding, I'm thinking it probably isn't worth creating a separate component, 
which is what you said in the first place :). redhat-rpm-config's repo 
structure and the divergent git branches and changelogs are just really 
annoying to work with :(.

[Maxwell G]

On Saturday, June 25, 2022 12:32:02 PM CDT Maxwell G wrote:
> > However, I'd consider adding the directories to filesystem directly to
> >
> > make it simpler and avoid any new dependencies. It's less cool, but also
> > less code.
>
> I agree. The bash completions dir is already owned by filesystem, so we
> might as well add the other ones.

I have submitted https://src.fedoraproject.org/rpms/filesystem/pull-request/7.

On Sunday, June 26, 2022 6:32:26 PM CDT Maxwell G wrote:
> Thanks again for the advice. If it's really only three static macros that
> I'm adding, I'm thinking it probably isn't worth creating a separate
> component, which is what you said in the first place :).
> redhat-rpm-config's repo structure and the divergent git branches and
> changelogs are just really annoying to work with :(.

I have also submitted https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/206.

[Vít Ondruch]

BTW just noticet this ticket:

https://bugzilla.redhat.com/show_bug.cgi?id=1312594

Vít

Dne 27. 06. 22 v 22:46 Maxwell G napsal(a):
> On Saturday, June 25, 2022 12:32:02 PM CDT Maxwell G wrote:
> > > However, I'd consider adding the directories to filesystem directly to
> > >
> > > make it simpler and avoid any new dependencies. It's less cool, but also
> > > less code.
> >
> > I agree. The bash completions dir is already owned by filesystem, so we
> > might as well add the other ones.
>
> I have submitted https://src.fedoraproject.org/rpms/filesystem/pull-request/7.
>
> On Sunday, June 26, 2022 6:32:26 PM CDT Maxwell G wrote:
> > Thanks again for the advice. If it's really only three static macros that
> > I'm adding, I'm thinking it probably isn't worth creating a separate
> > component, which is what you said in the first place :).
> > redhat-rpm-config's repo structure and the divergent git branches and
> > changelogs are just really annoying to work with :(.
>
> I have also submitted https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/206.
>
> packaging mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Vít Ondruch]

Dne 24. 06. 22 v 21:26 Maxwell G napsal(a):
> Hi everyone,
>
> Currently, packaging shell completions is a bit of a hassle. You have to
> memorize a bunch of long directory names (`%{_datadir}/bash-completion/
> completions`, `%{_datadir}/fish/vendor_completions.d`, and `%{_datadir}/zsh/
> site-functions`), and repeat them multiple times in the specfile. For some
> reason, the bash completion directories are owned by filesystem, but zsh and
> fish's shell completions directories are not. Therefore, every package needs
> to copy
> %files
> [...]
> %{_datadir}/bash-completion/completions/%{name}
> %dir %{_datadir}/fish
> %dir %{_datadir}/fish/vendor_completions.d
> %{_datadir}/fish/vendor_completions.d/%{name}.fish

I think that in practice, owning just `%{_datadir}/fish` would be 
enough, because the package one prepares probably won't have any 
additional content in this directory apart from the 
`%{_datadir}/fish/vendor_completions.d/%{name}.fish` file. But of 
course, the more verbose version you used provides more control.

Vít

[Vít Ondruch]

Dne 27. 06. 22 v 17:55 Vít Ondruch napsal(a):
> Dne 24. 06. 22 v 21:26 Maxwell G napsal(a):
> > Hi everyone,
> >
> > Currently, packaging shell completions is a bit of a hassle. You have to
> > memorize a bunch of long directory names (`%{_datadir}/bash-completion/
> > completions`, `%{_datadir}/fish/vendor_completions.d`, and 
> > `%{_datadir}/zsh/
> > site-functions`), and repeat them multiple times in the specfile. For 
> > some
> > reason, the bash completion directories are owned by filesystem, but 
> > zsh and
> > fish's shell completions directories are not. Therefore, every 
> > package needs
> > to copy
> > %files
> > [...]
> > %{_datadir}/bash-completion/completions/%{name}
> > %dir %{_datadir}/fish
> > %dir %{_datadir}/fish/vendor_completions.d
> > %{_datadir}/fish/vendor_completions.d/%{name}.fish
>
> I think that in practice, owning just `%{_datadir}/fish` would be 
> enough, because the package one prepares probably won't have any 
> additional content in this directory apart from the 
> `%{_datadir}/fish/vendor_completions.d/%{name}.fish` file. But of 
> course, the more verbose version you used provides more control.
>
> Vít

BTW why not have package such as bash-completion-filesystem, etc? I wish 
the filesystem package was smaller.

Vít

[Maxwell G]

On Monday, June 27, 2022 10:59:05 AM CDT Vít Ondruch wrote:
> Dne 27. 06. 22 v 17:55 Vít Ondruch napsal(a):
> > I think that in practice, owning just `%{_datadir}/fish` would be 
> > enough, because the package one prepares probably won't have any 
> > additional content in this directory apart from the 
> > `%{_datadir}/fish/vendor_completions.d/%{name}.fish` file. But of 
> > course, the more verbose version you used provides more control.

Yeah, that's what I said right below the part you quoted :).
> BTW why not have package such as bash-completion-filesystem, etc? I wish 
> the filesystem package was smaller.

I do agree that the filesystem package is rather large. When I created my PR to 
add the directories to filesystem, there were so many arguments being passed to 
mkdir that I had to split them up to avoid an error message[1].

The bash completion directories are already owned by filesystem, so I agreed 
that it made sense to add the other ones. If there is consensus to remove 
those directories from filesystem and not add the others, I can create a shell-
completions-filesystem package (or separate filesystem packages per shell). 
FWIW,

```
$ parallel rpm -qf -- /usr/share/bash-completion/completions/* | pkgname | 
sort | uniq | wc -l
119
```

on my system, so if that's at all representative, I'm not sure that it's 
worthwhile to create a separate package that is almost certain to be pulled in 
by something.

[Vít Ondruch]

Dne 29. 06. 22 v 1:17 Maxwell G napsal(a):
> On Monday, June 27, 2022 10:59:05 AM CDT Vít Ondruch wrote:
> > Dne 27. 06. 22 v 17:55 Vít Ondruch napsal(a):
> > > I think that in practice, owning just `%{_datadir}/fish` would be
> > > enough, because the package one prepares probably won't have any
> > > additional content in this directory apart from the
> > > `%{_datadir}/fish/vendor_completions.d/%{name}.fish` file. But of
> > > course, the more verbose version you used provides more control.
>
> Yeah, that's what I said right below the part you quoted :).

Ah, you are right, sorry. The example probably caught all my attention :)
> > BTW why not have package such as bash-completion-filesystem, etc? I wish
> > the filesystem package was smaller.
>
> I do agree that the filesystem package is rather large. When I created my PR to
> add the directories to filesystem, there were so many arguments being passed to
> mkdir that I had to split them up to avoid an error message[1].
>
> The bash completion directories are already owned by filesystem, so I agreed
> that it made sense to add the other ones. If there is consensus to remove
> those directories from filesystem and not add the others, I can create a shell-
> completions-filesystem package (or separate filesystem packages per shell).
> FWIW,
> $ parallel rpm -qf -- /usr/share/bash-completion/completions/* | pkgname |
> sort | uniq | wc -l
> 119
>
> on my system, so if that's at all representative, I'm not sure that it's
> worthwhile to create a separate package that is almost certain to be pulled in
> by something.

Just FTR, I am coming to this request from quite different angle and 
that are SCLs (not that they are very much alive, but ...). The issue is 
that every SCL emulates the filesystem structure in /opt, while probably 
just minority of the directories are used. Not mentioning that these 
days, nobody really knows how is the filesystem emulated in SCLs. From 
this POV, it would be much better if filesystem contained just the 
essential directories.

Also, installing just the filesystem consumes surprisingly huge amount 
of disk space.

IOW, filesystem (or RPM directory handling) would IMO deserve big 
overhaul. But don't get me wrong, I am not against your proposal, I just 
see more opportunities ;)

Vít