[Fedora-packaging] SCL Notes and Questions

[Marcela Mašláňová]

On 09/16/2013 08:25 PM, Toshio Kuratomi wrote:
> Note: This started as notes and questions from FPC members at the
> meeting, progressed to more notes and questions that FPC members
> Remi|Fedora and abadger1999 (me) put together after the meeting, and
> ended with a thorough rewrite by langdon (one of the SCL Guideline
> drafters) after listening to abadger1999 and Remi|Fedora's
> explanations of their notes and questions.  The final rewrite of these
> comments mostly takes the form of things that Should be changed in the
> draft but it must be noted that the final form wasn't created by the
> FPC so it is quite possible that the FPC will disagree or want to seek
> other options about some of the suggestions made here.  Sorry for the
> confusing parentage.
>
>
> Comments on the SCL Draft:
> https://fedoraproject.org/wiki/User:Bkabrda/SCLGuidelinesDraft
>
> * Can we assume that the implementation can be changed to address
> problems, add features, or follow different conventions?  Can be done
> if backwards compat is possible?  Or is implementation set in stone so
> we'd have to kludge around any shortcomings?
>
Yes, we can assume changes. If you have ideas for improvement I'd like 
to hear them. Current guidelines are based on usage by wide variety of 
developers and users, so there shouldn't be needed many changes.

> * One of the goals of SCLs is isolation from the rest of the system,
> correct?  It seems, though, that once you run the enable script, that
> shell would have a different PATH with the possibility of a different
> language interpreter.  As long as scripts can use /usr/bin/env as a
> shebang or scripts are using PATH-based commands these will end up
> using programs from the SCL.  Should FPC revisit banning these?  Is it
> okay to run with SCLs even without banning?  Just something to
> document?
>
There are mainly two possibilities how to run apps in collections. It 
depends what you wish to do with your application. Please do not ban 
anything, people have to decide for themselves what is their use-case.

1/ #!/usr/bin/env something is good for developers who want to switch 
between versions. I guess on Fedora will be some developers.
2/ #!/opt/vendor/something/root/usr/bin/something it's used as wrapper, 
good for deployed apps, because those who deploy app can define in 
wrapper, which collections have to be enabled by default

We do not provide wrappers inside collections, projects usually write 
them for themselves.
I would be fix it only in documentation.

> == Criteria: When is a Collection the Right Choice? ==
>
> * What is the logical limit of the "size" of that collection?
>    * (toshio) Criteria should state whether collections are to be
> "thin" or "thick".  Thin collections would consist of only a specific
> target (for instance, ruby1.9.3) and the packages needed to implement
> it.  Thick collections would include the collection, the packages to
> implement it, and packages implemented on top of it.
>    * langdon has the concept of "Ease of use" packages that are often
> needed by people developing in a certain environment (for instance,
> adding rails to the ruby1.9.3 collection).  Including ease of use
> packages would be a restricted variety of thick collections.
>    * langdon and I also discussed thick collections allowing as many
> packages being added to it as maintainers wanted.  To take a python
> example, if people wanted to support django and flask and pyramid and
> TurboGears1 and TurboGears2 on python2.6 SCL, they could have packages
> for all of those in the python2.6 Collection.  langdon thought this
> would be preferable to having framework-level SCLs but I have concerns
> about how this plays out with a backwards compatibility policy.
>    * thin collections would depend on inter-scl dependencies to provide
> people with things they want for "ease of use".  For instance, there
> could be a rails3 scl that depends on scl-ruby1.9.3 that provides
> rails3 for ruby1.9.3.
>      * Naming of scls with inter-scl dependencies needs to be worked
> out.  You might have people who want to package: rails3 + ruby1.9.3,
> rails4 + ruby1.9.3, rails3 + ruby2.0, rails4 + ruby2.0 and all of
> those need unique names.
>
We are afraid that giving so much space for combination might lead to 
chaos. I do not want to see many combinations of ruby and rails of 
different versions with different versions of bundler etc.
I proposed having SCL SIG which would say something like: "We already 
have Ruby193 and Rails2.x. If you want to just update rubygem-XY it's 
fine with us." or "We have it, updating rubygem-XY will broke it for 
sure for those users, create your own collection."

I agree having smaller collections than we have now would be 
improvement, but I'd like to give boundaries to all those possible 
combinations. In Fedora must be shipped only those collections, which 
are known as stable, well tested, with packages working together.

I'm not sure about this whole idea. The freedom might be a good thing, 
but we would have to set boundaries for what will go into Fedora. I do 
not have any idea how to do it, then give power to those who work on SCL 
or testers in bodhi, but than we would need really huge number of +1 or 
someone with veto.

> * What may the purpose of the Collection be?
>     * Criteria should indicate that a "language" may be included but
> not a framework (Rails) or a complete platform (LAMP). However,
> optional, "ease of use" packages may be included but not installed by
> the main meta-package. Otherwise, the scls will need to change/fork
> too fast
>       * (toshio): langdon is convinced this is the right path but I'm
> not sure.  There's a lot of problems with how this interacts with
> backwards compat requirements.  I think a whole package has to be
> worked out with how those shape up.
Collection in Fedora should give developers, who are not ready to move 
to latest version, time to develop on older version.

Typically, Ruby and Rails, Python and mod_wsgi, but I do not suggest to 
add fast moving frameworks. We have cpan/pip for such packages.

>     * Databases and other "servers" may be included but should never
> "replace" the native versions
>
I agree with that. We need some system database (or daemon with some 
specific functionality).

> * Collections and backwards compat:
>     * A requirement that scls should fit the same size as reasonable
> backwards  compat defintions from upstream.  For instance, if
> foo-X.Y.Z and every  foo-X release is backwards compatible with other
> foo-X releases, the scl  should be for fooX and not fooXY.
Fine.
>     * The "API" of an scl should never "shrink" only "grow." For
> example, if Rails3 is included in the Ruby1.9 SCL, and Rails4 wants to
> be included it does not replace Rails3. If, hypothetically, Rails4
> cannot be installed "next to" Rails3, a new SCL would need to be
> created to support Rails4, which requires a "real" change.
Agreed.
>       * (toshio) What would the new SCL be named?
>       * (toshio) This is problematic without also specifying a lifetime
> for SCLs and replacement of existing SCLs.  Some of the knock-on
> questions around this:
>         * What are we going to do about security issues in SCLs?  Does
> our policy say that backwards compat is more important than security
> issues?
No. We do not want to support not supportable (terribly broken) versions.
>         * Maybe Fedora simply isn't about backwards compat to this
> extent and so we have to allow API breaks within a release.
No, it would loose the purpose of scl for projects building above it.
>         * Maybe the policy for SCLs in The Outer Rings should have a
> strict backwards compat section but SCLs in Fedora Commons are more
> relaxed (that would make the model: Fedora Commons SCLs are primarily
> for enabling different versions of dependencies for software that we
> want to ship in Fedora Commons; Outer Ring SCLs are primarily for
> enabling developers to target an SCL rather than Fedora as their
> platform.)
Rings are still unclear.
>         * Maybe our policy should only allow "thin" SCLs.  ie: a
> ruby1.9 SCL would just have ruby1.9 and anything it needed in order to
> run.  If someone wanted to ship rails3 and rails4 targeted for
> ruby1.9, those would be two separate SCLs that depended on the ruby1.9
> SCL.  This allows individual SCL pieces to be deprecated without
> interfering with their parents or neighbors.
>         * Only partially addressing the situation, could figure out
> some way to mark some pieces of an SCL as having guaranteed backwards
> compat while others are only implementation detail.  If you had an SCL
> that shipped ruby1.9 but it needed a newer version of OpenSSL you
> might ship that inside the SCL at first.  But you wouldn't want people
> to rely on that particular version being available in the SCL in the
> future (once the base OS was updated to include that version.)
I might misunderstand. Do you suggest collection should "bundle" such 
package as openssl? That's security hole, not good for Fedora in circle 
1, 2 or 3.

>         * No matter where it is, need to have some criteria aroud how
> to obsolete an SCL, how to remove an SCL altogether, and whether two
> separate SCLs can ship largely the same thing.
Collection can be obsoleted same way as other packages, but let's say 
that maintainer should say, he do not want to support it for F-22 
anymore (before its release). Those still interested could carry on.

>     * (toshio) Can we make a recommendation on how or when to use
> scl-dependencies. For example, in the Rails4 example above, there may
> be scenarios that we want to allow this in Fedora. Although, the
> recommendation could be that frameworks and the like should be
> provided outside of Fedora Commons.
Do you mean dependencies among collections?

>   * Should a collection be used only for things that cannot be parallel
> packaged (e.g. python 3.3 can be packaged natively so it's not a
> candidate for an SCL but ruby1.9 cannot and thus it is)
>
Depends... Some projects would like to work with collections only, but 
currently I'm not convinced if it should be a rule or not.

> == Naming ==
> * problem with the examples (ruby193 and examples from:
> http://fedorahosted.org/SoftwareCollections ) For instance, python3.3
> would conflict with a python3.3 parallel stack that did not use scls.
Not true. We used python33 as a metapackage without dot.

> * Heard about namespace prefixes and that there were thoughts to name
> them after vendors (rht-ruby1.9).  If we use /opt/$vendor that might
> make sense.
>    * Could still have clashes with things like redhat-release,
> redhat-rpm-config, etc... make sure the vendors we use doesn't have
> any pre-existing conflicts of this sort.
> * scl makes sense as well.  mmaslano had said that end-users wouldn't
> know what an scl was but remi says you'd need to use a tool called
> "scl" to manage which scls are active anyway.  So that may not be a
> problem?
That's true. We were also thinking about users who are interested in app 
and know nothing about scl, so they just install app, run it and are not 
aware of scl at all.

>    * If we have multi-vendor for scl (fedora, rpmfusion, local admin
> case again), then vendor may make more sense again.
> * Change example naming: ruby193 => ruby1.9.3  Dots are clearer and
> thus preferred
Dots and such stuff are evil. I guess dots were omitted on purpose.

> * Do we really want minor versions in there?  or should this be
> ruby1.9 packages?  Need to make clear that the version should be about
> backwards compatibility.  So if ruby doesn't ensure that minor
> versions have compatibility, scl-ruby1.9.3 would be appropriate.
> OTOH, scl-python2.7 would be appropriate for an upstream that does
> have minor version backwards compat.
>
It depends on software. For example many people believes that Ruby do 
bigger changes in minor releases, on the other hand Python is not known 
for adding features into minor releases, so it's just  python33 and not 
python335.
>
> == File Location ==
>
> * If we're going to use /opt/vendor:
>    * scls should then obey the rest of the standard: /var/opt and
> /etc/opt are for host specific state and config.  Is that the case?
> can it be fixed if not?
Sometimes were used locations in /var/log or /etc because collections 
wouldn't work in case of mariadb and other daemons. We had to put init 
file somewhere and also logfile. It's documented and we didn't find 
better solution yet.

>    * We'd need to get a fedora vendor string from LANANA
>      * Might not want it to be "fedora" that might be more likely to be
> taken than something else. "fedoraproject"?  "fdr"?
> * Other locations:
>    * /usr/scl
>      * (toshio): "Large software packages must not use a direct
> subdirectory under the /usr hierarchy." which could be interpreted as
> forbiding this location.  (Although others may be more strict about
> defining "large sofware packages").
>    * %{_libdir}/scl
>      * (toshio): I don't see an FHS problem with this although the same
> issue arises about putting the host-specific config in /etc and the
> variable state data in /var.  I'm also coming around to
> /opt/$registered_vendor being the least-wrong place for this.
I support installation into /opt because it will be confusing to have in 
some downstreams /opt and in other /usr. Also it's possible to do 
symlinks to workaround some problems and these might broke.

> * What does the filesystem layout look like inside of this prefix?
> /opt/$vendor/scl/rht-ruby1.9.3/{normal filesystem}?
yes, normal filesystem.
> * Does this allow local sites and addon repos to make their own
> vendor?  How?  (Thinking of rhel/centos, epel, and local admin here.
> Or fedora/rpmfusion, and local admin)
Modify vendor in macros of scl.
> * Are scls allowed to install files outside of the scl tree?  If so,
> what types of files (criteria) and the naming of these? (systemd unit
> files, rpm macro files, launchers for applications/language
> interpreters in the scl?  Should we go with the FHS recommendation
> about /opt/vendor containing everything that is necessary for the
> package and then making copies(symlinks maybe?) to other needed
> locations?
I'll add those exceptions.
>    * (langdon) personally, i dont think it should touch anything
> besides opt and, potentially, symlinks
>      * (toshio) Well, how do things like the postgres package that
> start a daemon work then?
>      * (toshio) I don't think we can have symlinks into /opt work as
> expected but symlinks out of /opt should work.
> /etc/opt/scl/[...]/config and /var/opt/scl/[...]/state are both
> host-specific while /opt may be a shared network filesystem.
>
> == Spec Files ==
>   * Why don't we follow the mingw model and have separate spec files
> (mingw is both separate spec and separate package) for scl?
> Advantages:
>     * Many fewer conditionals in spec files (greater readability/maintainabilty)
>     * Primary maintainer doesn't need to know or care about SCLs if
> other people do
>     * SCL could be the only spec that is necessary or desired for a
> particular component
>   * Can a SCL spec be "only" an SCL spec (will not build a non-scl
> rpm)? If so, we need some more content describing this scenario.
>   * Propose moving the "style guide" to before the example (e.g.
> Section 3 before Section 2), so that some of the following questions
> don't come up for the reader
>   * Why are there so many macros that mean the same thing? -- %scl,
> %pkg_name, %scl_name, %name
>
I believe we should have branch for scl and other branches might 
voluntarily add macros, but it shouldn't be mandatory. For example perl 
or httpd don't have nice specfile and adding more condition wouldn't 
improve readability.
Having scl macros might improve speed of patching some packages, but not 
all.

> === Specific comments on the draft ===
>   * Move template to the bottom of the draft.  Had many questions after
> reading the template that were eventually answered in other parts of
> the draft.
>   * Style guide is not to have a period at the end of Summary
>   * Description in template should probably be more clear that
> packagers should fill in a proper description
>   * Template skips %prep and %build.  Are these empty?  If so, just
> include the empty %prep and %build headers.
>   * "The -build subpackage should include Requires: scl-utils-build."
> Why a should and not a must?
>   * Content about the enable scriptlet implies that the usual path
> macros (_bindir etc) are changed in the scl package.  Need to make it
> clear what standard macros are being overridden to what. Perhaps
> consider "tables" like
> http://fedoraproject.org/wiki/Packaging:MinGW#Filesystem_location_macros
> but with slightly different columns, e,g, macro | override? | Normally
> Modified? | example | description
>   * "The macros.%{scl}-config in %{scl}-build package contains the %scl
> macro definition (name of the SCL). If you need to use some more
> macros for this SCL, add them into this file." <= but there's nothing
> in the spec file that obviously creates this.  Need instructions on
> how to create this file for manual use and then how to include the
> file in the spec file and not have whatever automated macro normally
> creates it not overwrite it.
>   * "essential for its runtime". "Essential" needs to be explained and
> hopefully it is just a link to the earlier part of the doc.
>   * -runtime package must be arch specific => it contains
> %{scl_prefix}%{_libdir} so it's arch dependent. (and this means the
> main package must be arch specific too)
>
>   * Creating the enable file in %install -- would rather see macros
> like:  - %scl_add_path %scl_add_ld_library_path %scl_add_manpath than
> the HERE-doc style thing that will cut and paste to most specs.
>   * "set of macros that contain the values of the original RPM macros"
> should be expanded on.  naming of these could be better => establish a
> common prefix for anything that is being overridden instead of
> %pkg_name vs %_root_.  also pkg_name probably conflicts with some
> existing packages.  Maybe something like %_prescl_name,
> %_prescl_libdir, etc.  [can change the recommendation from: packagers
> should define %pkg_name to packagers should define a macro that is the
> value of pkg_name => %{?scl: %global srcname %pkg_name and %{?!scl:
> %global srcname %name}
>   * Little more guidance needed for Requires. Maybe making the examples
> more real-life would clarify things.  ("Since ifconfig is a binary
> we're going to use the system version of, do not use scl_prefix with
> it. rubygem-foo, otoh, has to be in the path of the ruby interpreter
> which this scl is to be a part of. Therefore it needs to use
> scl_prefix"). Also, an example of a "Versioned Requires" would be
> helpful.
>   * Need more wording on Requires: of non-scl packages as they can act
> as conflicts.  Something like: Requires: foo = 1.2.3 … where the scl
> package is the only thing keeping foo back at that version.  So you
> can't update "the OS" without updating the scl.  wording might be
> something like "yes, you can require non-scl stuff … but you have to
> be very general in your requirements, and if you need something that
> might be updated you have to either scl bundle it or be willing to
> rebuild the scl when it updates".
>   * Should we list somewhere the scl-modified deps scripts?  Otherwise
> how will someone creating a new ocaml-scl know whether an ocaml-scl
> dep script has already been written?
>   * Please elaborate on the function of %scl_package_override()
> * In the dealing with macro files section: I think that the "Now
> consider a situation[...]" starts a whole new scenario?  Correct?  If
> so, needs to be better phrased right now it's associating with the
> previous examples but it doesn't seem to make sense in that context.
> * So If I've understood everything correctly, building in a standard
> buildroot will get you a non-scl'd package.  Building with the
> $scl-build package installed will build the scl version?
>    * Should say that in the first line of Building Packages
> * In Building Packages: Don't need to include scl-utils-build
> explicitly in the config_opts line (brought in via dependencies),
> correct?
>
>
> == Design ==
>   * What directories is an SCL allowed to modify?
>   * Naming of packages is a problem as currently designed. Liklihood of
> conflict with native packages is very high. Propose to include "scl-"
> at the begining of names.
Why don't we already prefixed scl before all packages? Because it would 
be scl-ruby193-rubygems-blabla-1.1.f21. The prefix must not conflict 
with existing packages. It doesn't have to be ruby193, it might be name 
which specify what is inside.
>   * Include "dots" in names, e.g. ruby1.9 rather than ruby19
I prefer without dots, punctuation does everything harder.
>
> == Conversion to SCLs ==
>   * Parts of this section are more "tips & tricks"? Should it be on a
> non-guideline page?
>     * Probably best to vette the advice anyway and then move what makes
> sense to a separate page afterwards.
>   * Add information about why /usr/bin/env is not a generally good
> thing in SCLs.  If this is more general than SCLs, does FPC need to
> revisit a ban on /usr/bin/env in shebangs?
>   * sed line should limit to the first line of the file as well as the
> shebang pattern.
>   * how  does scl deal with compiled-in paths inside of binaries?  Does
> it  depend on %configure to do the right thing with %{_libdir} etc?
> If so  we should point out to people that old-school builds which
> aren't  configure driven may need patches (actually, this applies to
> install  location as well as compiled-in paths and to scripts that
> have file  paths embedded as well as binaries)
>   * How do auto provides and requires for elf libraries work?
>
> == Inter-SCL Dependencies ==
> * Example of using this with versioned Requires?
> * The %scl_require example just doesn't seem right somehow.... The
> ruby193 scl brings in rails-3.2.3?  That seems like poor naming (if
> the metapackage brings in rails, then it should be named rails.  OTOH,
> if the package purpose is to provide ruby193 then rails should be
> optional not essential.) or poor dependency management (Which would be
> part of the higher level questions of what can be an SCL and what can
> be inside an SCL).
>
> * Are non-scl packages allowed to dep on scl packages?  If so, how do
> they do so?
>
> * The filters don't work in EL6 thing.... people seem more excited
> about SCLs in EPEL than in fedora so we probably need to document how
> to filter the provides and requires for EL6.  However, we can point to
> the EPEL6 Differences page from here and document the behaviour there.
>
>
>
> == Build & Distribution ==
>   * What are the impacts on the build system?
Until now we used internal instance of koji. I can sum up what we did there.
>     * How are we preventing build problems from the mainline package
> from interfering with the scl package and vice versa?
There are no build interference problems. Package has specific name.
>     * How are we allowing for a package to be built in more than one scl?
Didn't solve yet. We related build targets (collection) with branch. How 
to build for more collections is a good question for our releng. But do 
we need it?
>       * How are we allowing for differences between scls? (ie:
> ruby1.9.3 scl with rubygem-foo1.0; ruby2.0 scl with rubygem-foo2.0)
>     * How are we preventing build problems from one scl from
> interfering with a different scl?
>     * Remi mentions that he thinks it would be done in a separate
> branch.  Would we need a branch per scl that the package is built in?
Can't happen. Every branch is set to build in some tag. Packages tagged 
in one tag are shipped as one collection. You are able to inherit from 
one tag to another.
As I said I prefer branch per collection. How else would be tags defined?
>   * How do SCLs get distributed?
>     * Is this a "separate repo" or is it just in the normal repo(s)?
Currently separate repo.
>       * (toshio) Both.  Some SCLs can go into Fedora Commons which
> would continue to have one repo. SCLs in the outer rings would have
> some separate structure.
Agreed.
>   * Build section would benefit from minor sections, perhaps:
> "Overview," "Testing" and "Example"? Sections need a bit more color to
> be understandable.
>
I'll add it.
> == General Comments ==
>   * mingw is a similar set of guidelines (but in separate spec files
> rather than the same one.)  So issues like what their naming solves
> and tracking of fedora packages could be looked at.
> http://fedoraproject.org/wiki/Packaging:MinGW. The mingw packaging
> guidelines should also be reviewed for "missing content" vs this
> proposal.
>   * No definition of a SCL appears in the document
>     * Example text: Software Collections (SCLs) give rpm packagers a
> means of packaging multiple versions of software for a single
> distribution.  The version from the software collection does not
> interact with the system version. They can provide backwards and
> forwards compat versions of a package compared to what's on the
> system.
>   * Include "definitions" at the beginning of the document, some
> example things that need definition (and suggestions)
>     * SCL -> whole software collection? <- originally the term was
> "dynamic software collection" for the "whole thing" but it has fallen
> in to disuse... we could bring it back
>     * SCL defining package -> the srpm-level package that defines the scl.
>     * SCL metapackage -> built from the SCL defining package.  defines
> the essential packages you get when you install the scl.
>     * SCL runtime package -> built from the SCL defining package.  It's
> the setup of the system for this SCL including registering the SCL
> with the system and setting up the filesystem.
>     * SCL build package -> built from the SCL defining package.  It
> contains macro definitions needed to build an SCL.
>     * SCL package -> any package which can be built for an SCL.
>   * Elaborate on what "belongs" in the "-runtime" package
>     * Extracting one of these from the sample repository seems like
> it's a filesystem package directory tree inside of the scl?  Anything
> else?
> * From the SCLinFedora page, why the differentiation between regular
> package reviews and scl package reviews?
>
> -Toshio
> --
> packaging mailing list
> packaging at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/packaging
>
Thanks for your input. I was waiting for something like that so I could 
add what is missing or unclear to people who never build a collection.
I'm looking forward to reaction on updated drafts. We will do it with 
Slavek tomorrow.

Marcela