More declarative RPMs

[Matthew Kenigsberg]

Hi,
I was discussing containerization and making RPMs more declarative with Aleksandra
Fedorova, and she suggested I ask here and mention Zbigniew Jędrzejewski-Szmek.
It sounds like there's already some effort to make tasks in RPMs like adding users
more declarative:
https://docs.fedoraproject.org/en-US/packaging-guidelines/UsersAndGroups/
I'm working with rpm-ostree (https://coreos.github.io/rpm-ostree/) and Butane
(https://coreos.github.io/butane/) to manage system state more declaratively. Currently
rpm-ostree manages RPMs and then Butane handles things like putting config files in /etc
or creating users. It sounds like there's some overlap between the changes in RPMs
around managing users and Butane, so I'm wondering if there could be more
cooperation.
If RPMs relied less on scriptlets and more on declarative config that could be merged, it
would make optimizations for containerization a lot easier. Each package in a container
image could be in a separate layer, which would allow much better sharing between
container images. Adding a single package to a container image could just require pulling
the package, rather than running a scriptlet that could possibly change the base image. 

Do you think it would be possible to move some of the configuration tasks performed in
scriptlets into something declarative that could be merged with other configuration?
Are there any ways rpm-ostree/Butane could align itself better with the direction RPMs are
taking? Is there a solution or format that would satisfy the requirements of both tools?

Thanks,
Matthew

[Miroslav Suchý]

Dne 15. 02. 22 v 22:08 Matthew Kenigsberg napsal(a):
>  It sounds like there's already some effort to make tasks in RPMs
> like adding users more
> declarative:https://docs.fedoraproject.org/en-US/packaging-guidelines/Use...
Which is still not finished and you still need to call the old scriptlet when you
need the user to exist during %post. :(
>  Do you think it would be possible to move some of the configuration
> tasks performed in scriptlets into something declarative that could be merged with other
> configuration? 
Yes. Take the

https://src.fedoraproject.org/lookaside/rpm-specs-latest.tar.xz

grep for common scriptlets and resolve them one-by-one. Starting with common one. But it
is huge task!

You can choose where you will start:

* mkfontdir - which can be replaced by filetriggers.

* update-desktop-database - which - I believe - has been already replaced by filetriggers,
but it is still used.

*update-alternatives - which no one yet touched

.... and many many others

Miroslav

[Adam Williamson]

On Wed, 2022-02-16 at 09:17 +0100, Miroslav Suchý wrote:
>  Dne 15. 02. 22 v 22:08 Matthew Kenigsberg napsal(a):
>  > It sounds like there's already some effort to make tasks in RPMs like adding
> users more
> declarative:https://docs.fedoraproject.org/en-US/packaging-guidelines/Use...
>  Which is still not finished and you still need to call the old scriptlet when you need
> the user to exist during %post. :(
>  > Do you think it would be possible to move some of the configuration tasks performed
> in scriptlets into something declarative that could be merged with other configuration?
>  
>  Yes. Take the
>  
>  https://src.fedoraproject.org/lookaside/rpm-specs-latest.tar.xz
>  
>  grep for common scriptlets and resolve them one-by-one. Starting with common one. But it
> is huge task!
>  
>  You can choose where you will start:
>  
>  * mkfontdir - which can be replaced by filetriggers.
>  
>  * update-desktop-database - which - I believe - has been already replaced by
> filetriggers, but it is still used.
>  
>  *update-alternatives - which no one yet touched
>  
>  .... and many many others 
As well as Zbigniew, I believe Will Woods was quite interested in this
stuff for a while. Not sure if he still is.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Vitaly Zaitsev]

On 16/02/2022 09:17, Miroslav Suchý wrote:
>  * update-desktop-database - which - I believe - has been already 
>  replaced by filetriggers, but it is still used.
>   
update-desktop-database, update-mime-database, ldconfig, update-manpages 
and gtk-update-icon-cache should be removed from all Fedora SPECs. 
Proven packages can do this.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Vitaly Zaitsev]

On 16/02/2022 09:17, Miroslav Suchý wrote:
>  * update-desktop-database - which - I believe - has been already 
>  replaced by filetriggers, but it is still used.
>   
update-desktop-database, update-mime-database, ldconfig, update-manpages 
and gtk-update-icon-cache should be removed from all Fedora SPECs. 
Proven packagers can do this.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Aleksandra Fedorova]

Hi,

On Wed, Feb 16, 2022 at 9:18 AM Miroslav Suchý <msuchy(a)redhat.com&gt; wrote:
>  Dne 15. 02. 22 v 22:08 Matthew Kenigsberg napsal(a):
>
>  It sounds like there's already some effort to make tasks in RPMs like adding users
> more declarative:
> https://docs.fedoraproject.org/en-US/packaging-guidelines/UsersAndGroups/
>
>  Which is still not finished and you still need to call the old scriptlet when you need
> the user to exist during %post. :(
>
>  Do you think it would be possible to move some of the configuration tasks performed in
> scriptlets into something declarative that could be merged with other configuration?
>
>  Yes. Take the
>
>    https://src.fedoraproject.org/lookaside/rpm-specs-latest.tar.xz
>
>  grep for common scriptlets and resolve them one-by-one. Starting with common one. But it
> is huge task!
>
>  You can choose where you will start:
>
>  * mkfontdir - which can be replaced by filetriggers.
>
>  * update-desktop-database - which - I believe - has been already replaced by
> filetriggers, but it is still used.
>
>  *update-alternatives - which no one yet touched
>
>  .... and many many others 
Do I understand correctly that there is a general agreement that it is
the right direction to take, and the main reason why we are not there
yet is the size of the effort?

I tried to find any good doc but the page in Packaging Guidelines
doesn't say anything about scriptlets being discouraged:
https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/

Maybe we should make some long-term landing page for this initiative,
similarly to what Miro and the Python team did for Python 2 to Python
3 migration? So that it is easier to get people started on it.

The list you posted can already be a good start.

And I don't think proven packagers rights will be really necessary as
the work can be done via merge requests and should be accessible for
all.

-- 
Aleksandra Fedorova
bookwar

[Miro Hrončok]

On 16. 02. 22 9:43, Aleksandra Fedorova wrote:
>  Do I understand correctly that there is a general agreement that it
> is
>  the right direction to take, and the main reason why we are not there
>  yet is the size of the effort? 
Correct.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Richard W.M. Jones]

On Wed, Feb 16, 2022 at 09:43:05AM +0100, Aleksandra Fedorova wrote:
>  Hi,
>  
>  On Wed, Feb 16, 2022 at 9:18 AM Miroslav Suchý <msuchy(a)redhat.com&gt; wrote:
>  >
>  > Dne 15. 02. 22 v 22:08 Matthew Kenigsberg napsal(a):
>  >
>  > It sounds like there's already some effort to make tasks in RPMs like adding
> users more declarative:
> https://docs.fedoraproject.org/en-US/packaging-guidelines/UsersAndGroups/
>  >
>  > Which is still not finished and you still need to call the old scriptlet when you
> need the user to exist during %post. :(
>  >
>  > Do you think it would be possible to move some of the configuration tasks performed
> in scriptlets into something declarative that could be merged with other configuration?
>  >
>  > Yes. Take the
>  >
>  >   https://src.fedoraproject.org/lookaside/rpm-specs-latest.tar.xz
>  >
>  > grep for common scriptlets and resolve them one-by-one. Starting with common one.
> But it is huge task!
>  >
>  > You can choose where you will start:
>  >
>  > * mkfontdir - which can be replaced by filetriggers.
>  >
>  > * update-desktop-database - which - I believe - has been already replaced by
> filetriggers, but it is still used.
>  >
>  > *update-alternatives - which no one yet touched
>  >
>  > .... and many many others
>  
>  Do I understand correctly that there is a general agreement that it is
>  the right direction to take, and the main reason why we are not there
>  yet is the size of the effort? 
Definitely.  I have long said that any time there's a scriptlet in an
RPM package, we should work towards removing it.

BTW I found this F31 change proposal.  Did it happen?  The bug was
CLOSED/CURRENTRELEASE.

https://fedoraproject.org/wiki/Features/LimitScriptletUsage
https://bugzilla.redhat.com/show_bug.cgi?id=1754666

>  I tried to find any good doc but the page in Packaging Guidelines
>  doesn't say anything about scriptlets being discouraged:
>  https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/
>  
>  Maybe we should make some long-term landing page for this initiative,
>  similarly to what Miro and the Python team did for Python 2 to Python
>  3 migration? So that it is easier to get people started on it.
>  
>  The list you posted can already be a good start.
>  
>  And I don't think proven packagers rights will be really necessary as
>  the work can be done via merge requests and should be accessible for
>  all. 
Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

[James Antill]

On Wed, 2022-02-16 at 10:35 +0000, Richard W.M. Jones wrote:
>  Definitely.  I have long said that any time there's a scriptlet
> in an
>  RPM package, we should work towards removing it.
>  
>  BTW I found this F31 change proposal.  Did it happen?  The bug was
>  CLOSED/CURRENTRELEASE.
>  
>  https://fedoraproject.org/wiki/Features/LimitScriptletUsage
>  https://bugzilla.redhat.com/show_bug.cgi?id=1754666
>   
 Didn't have enough resources, so got dropped. Some upstream code/data
is here:

https://github.com/james-antill/scriptlets-monitor
https://fedoraproject.org/wiki/JamesAntill/ScriptletUsage
https://github.com/james-antill/combined

...and a couple of things went into the distro.
 IIRC the rough thoughts/conclusions, at the time, was that the big win
was when you could get a complete install done without any scriptlets
and it should be possible to get a bunch of the small cases to that win
state (@minimal, base image). 
 Maybe get @server a release or two later.

[Miroslav Suchý]

Dne 16. 02. 22 v 9:43 Aleksandra Fedorova napsal(a):
>  doesn't say anything about scriptlets being discouraged: 
because we are far from using a word "discouraged". But yes, filetriggers and
other similar methods should be prefered.

Miroslav

[Neal Gompa]

On Wed, Feb 16, 2022 at 7:54 AM Miroslav Suchý <msuchy(a)redhat.com&gt; wrote:
>  Dne 16. 02. 22 v 9:43 Aleksandra Fedorova napsal(a):
>  > doesn't say anything about scriptlets being discouraged:
>
>  because we are far from using a word "discouraged". But yes, filetriggers and
> other similar methods should be prefered.
>  
I'd even been looking into designing some triggers for eliminating the
systemd unit macros we use in spec files too...

It's tricky, but I'd like to get rid of them. :)



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Matthew Kenigsberg]

Glad to hear there's a lot of interest in this!

I think file triggers would definitely help with optimizing containerization, but I think
it still has some issues. Two properties that would be desirable for optimizing
containerization are:
1. Mergeability. If multiple packages perform the same task, it can just be run once.
2. Equivalence of incremental changes and a clean build (or anti-hysteresis; see
https://blog.verbum.org/2020/08/22/immutable-%E2%86%92-reprovisionable-an...).
If different systems install (or uninstall) the same packages in a different order, is the
final state guaranteed to be identical?

I think transactional file triggers help with 1, but I don't think they currently
guarantee 2. I'm wondering if there'd be any way to allow RPMs to guarantee 2.

[Vitaly Zaitsev]

On 15/02/2022 22:08, Matthew Kenigsberg wrote:
>  It sounds like there's already some effort to make tasks in RPMs
> like adding users more
> declarative:https://docs.fedoraproject.org/en-US/packaging-guidelines/Use...
https://pagure.io/packaging-committee/issue/1158

>  Do you think it would be possible to move some of the configuration
> tasks performed in scriptlets into something declarative that could be merged with other
> configuration? 
+1 for removing scriptlets.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)