[Fedora-packaging] SCL Notes and Questions

[Toshio Kuratomi]

Note: This started as notes and questions from FPC members at the
meeting, progressed to more notes and questions that FPC members
Remi|Fedora and abadger1999 (me) put together after the meeting, and
ended with a thorough rewrite by langdon (one of the SCL Guideline
drafters) after listening to abadger1999 and Remi|Fedora's
explanations of their notes and questions.  The final rewrite of these
comments mostly takes the form of things that Should be changed in the
draft but it must be noted that the final form wasn't created by the
FPC so it is quite possible that the FPC will disagree or want to seek
other options about some of the suggestions made here.  Sorry for the
confusing parentage.


Comments on the SCL Draft:
https://fedoraproject.org/wiki/User:Bkabrda/SCLGuidelinesDraft

* Can we assume that the implementation can be changed to address
problems, add features, or follow different conventions?  Can be done
if backwards compat is possible?  Or is implementation set in stone so
we'd have to kludge around any shortcomings?

* One of the goals of SCLs is isolation from the rest of the system,
correct?  It seems, though, that once you run the enable script, that
shell would have a different PATH with the possibility of a different
language interpreter.  As long as scripts can use /usr/bin/env as a
shebang or scripts are using PATH-based commands these will end up
using programs from the SCL.  Should FPC revisit banning these?  Is it
okay to run with SCLs even without banning?  Just something to
document?

== Criteria: When is a Collection the Right Choice? ==

* What is the logical limit of the "size" of that collection?
  * (toshio) Criteria should state whether collections are to be
"thin" or "thick".  Thin collections would consist of only a specific
target (for instance, ruby1.9.3) and the packages needed to implement
it.  Thick collections would include the collection, the packages to
implement it, and packages implemented on top of it.
  * langdon has the concept of "Ease of use" packages that are often
needed by people developing in a certain environment (for instance,
adding rails to the ruby1.9.3 collection).  Including ease of use
packages would be a restricted variety of thick collections.
  * langdon and I also discussed thick collections allowing as many
packages being added to it as maintainers wanted.  To take a python
example, if people wanted to support django and flask and pyramid and
TurboGears1 and TurboGears2 on python2.6 SCL, they could have packages
for all of those in the python2.6 Collection.  langdon thought this
would be preferable to having framework-level SCLs but I have concerns
about how this plays out with a backwards compatibility policy.
  * thin collections would depend on inter-scl dependencies to provide
people with things they want for "ease of use".  For instance, there
could be a rails3 scl that depends on scl-ruby1.9.3 that provides
rails3 for ruby1.9.3.
    * Naming of scls with inter-scl dependencies needs to be worked
out.  You might have people who want to package: rails3 + ruby1.9.3,
rails4 + ruby1.9.3, rails3 + ruby2.0, rails4 + ruby2.0 and all of
those need unique names.

* What may the purpose of the Collection be?
   * Criteria should indicate that a "language" may be included but
not a framework (Rails) or a complete platform (LAMP). However,
optional, "ease of use" packages may be included but not installed by
the main meta-package. Otherwise, the scls will need to change/fork
too fast
     * (toshio): langdon is convinced this is the right path but I'm
not sure.  There's a lot of problems with how this interacts with
backwards compat requirements.  I think a whole package has to be
worked out with how those shape up.
   * Databases and other "servers" may be included but should never
"replace" the native versions

* Collections and backwards compat:
   * A requirement that scls should fit the same size as reasonable
backwards  compat defintions from upstream.  For instance, if
foo-X.Y.Z and every  foo-X release is backwards compatible with other
foo-X releases, the scl  should be for fooX and not fooXY.
   * The "API" of an scl should never "shrink" only "grow." For
example, if Rails3 is included in the Ruby1.9 SCL, and Rails4 wants to
be included it does not replace Rails3. If, hypothetically, Rails4
cannot be installed "next to" Rails3, a new SCL would need to be
created to support Rails4, which requires a "real" change.
     * (toshio) What would the new SCL be named?
     * (toshio) This is problematic without also specifying a lifetime
for SCLs and replacement of existing SCLs.  Some of the knock-on
questions around this:
       * What are we going to do about security issues in SCLs?  Does
our policy say that backwards compat is more important than security
issues?
       * Maybe Fedora simply isn't about backwards compat to this
extent and so we have to allow API breaks within a release.
       * Maybe the policy for SCLs in The Outer Rings should have a
strict backwards compat section but SCLs in Fedora Commons are more
relaxed (that would make the model: Fedora Commons SCLs are primarily
for enabling different versions of dependencies for software that we
want to ship in Fedora Commons; Outer Ring SCLs are primarily for
enabling developers to target an SCL rather than Fedora as their
platform.)
       * Maybe our policy should only allow "thin" SCLs.  ie: a
ruby1.9 SCL would just have ruby1.9 and anything it needed in order to
run.  If someone wanted to ship rails3 and rails4 targeted for
ruby1.9, those would be two separate SCLs that depended on the ruby1.9
SCL.  This allows individual SCL pieces to be deprecated without
interfering with their parents or neighbors.
       * Only partially addressing the situation, could figure out
some way to mark some pieces of an SCL as having guaranteed backwards
compat while others are only implementation detail.  If you had an SCL
that shipped ruby1.9 but it needed a newer version of OpenSSL you
might ship that inside the SCL at first.  But you wouldn't want people
to rely on that particular version being available in the SCL in the
future (once the base OS was updated to include that version.)
       * No matter where it is, need to have some criteria aroud how
to obsolete an SCL, how to remove an SCL altogether, and whether two
separate SCLs can ship largely the same thing.
   * (toshio) Can we make a recommendation on how or when to use
scl-dependencies. For example, in the Rails4 example above, there may
be scenarios that we want to allow this in Fedora. Although, the
recommendation could be that frameworks and the like should be
provided outside of Fedora Commons.
 * Should a collection be used only for things that cannot be parallel
packaged (e.g. python 3.3 can be packaged natively so it's not a
candidate for an SCL but ruby1.9 cannot and thus it is)

== Naming ==
* problem with the examples (ruby193 and examples from:
http://fedorahosted.org/SoftwareCollections ) For instance, python3.3
would conflict with a python3.3 parallel stack that did not use scls.
* Heard about namespace prefixes and that there were thoughts to name
them after vendors (rht-ruby1.9).  If we use /opt/$vendor that might
make sense.
  * Could still have clashes with things like redhat-release,
redhat-rpm-config, etc... make sure the vendors we use doesn't have
any pre-existing conflicts of this sort.
* scl makes sense as well.  mmaslano had said that end-users wouldn't
know what an scl was but remi says you'd need to use a tool called
"scl" to manage which scls are active anyway.  So that may not be a
problem?
  * If we have multi-vendor for scl (fedora, rpmfusion, local admin
case again), then vendor may make more sense again.
* Change example naming: ruby193 => ruby1.9.3  Dots are clearer and
thus preferred
* Do we really want minor versions in there?  or should this be
ruby1.9 packages?  Need to make clear that the version should be about
backwards compatibility.  So if ruby doesn't ensure that minor
versions have compatibility, scl-ruby1.9.3 would be appropriate.
OTOH, scl-python2.7 would be appropriate for an upstream that does
have minor version backwards compat.


== File Location ==

* If we're going to use /opt/vendor:
  * scls should then obey the rest of the standard: /var/opt and
/etc/opt are for host specific state and config.  Is that the case?
can it be fixed if not?
  * We'd need to get a fedora vendor string from LANANA
    * Might not want it to be "fedora" that might be more likely to be
taken than something else. "fedoraproject"?  "fdr"?
* Other locations:
  * /usr/scl
    * (toshio): "Large software packages must not use a direct
subdirectory under the /usr hierarchy." which could be interpreted as
forbiding this location.  (Although others may be more strict about
defining "large sofware packages").
  * %{_libdir}/scl
    * (toshio): I don't see an FHS problem with this although the same
issue arises about putting the host-specific config in /etc and the
variable state data in /var.  I'm also coming around to
/opt/$registered_vendor being the least-wrong place for this.
* What does the filesystem layout look like inside of this prefix?
/opt/$vendor/scl/rht-ruby1.9.3/{normal filesystem}?
* Does this allow local sites and addon repos to make their own
vendor?  How?  (Thinking of rhel/centos, epel, and local admin here.
Or fedora/rpmfusion, and local admin)
* Are scls allowed to install files outside of the scl tree?  If so,
what types of files (criteria) and the naming of these? (systemd unit
files, rpm macro files, launchers for applications/language
interpreters in the scl?  Should we go with the FHS recommendation
about /opt/vendor containing everything that is necessary for the
package and then making copies(symlinks maybe?) to other needed
locations?
  * (langdon) personally, i dont think it should touch anything
besides opt and, potentially, symlinks
    * (toshio) Well, how do things like the postgres package that
start a daemon work then?
    * (toshio) I don't think we can have symlinks into /opt work as
expected but symlinks out of /opt should work.
/etc/opt/scl/[...]/config and /var/opt/scl/[...]/state are both
host-specific while /opt may be a shared network filesystem.

== Spec Files ==
 * Why don't we follow the mingw model and have separate spec files
(mingw is both separate spec and separate package) for scl?
Advantages:
   * Many fewer conditionals in spec files (greater readability/maintainabilty)
   * Primary maintainer doesn't need to know or care about SCLs if
other people do
   * SCL could be the only spec that is necessary or desired for a
particular component
 * Can a SCL spec be "only" an SCL spec (will not build a non-scl
rpm)? If so, we need some more content describing this scenario.
 * Propose moving the "style guide" to before the example (e.g.
Section 3 before Section 2), so that some of the following questions
don't come up for the reader
 * Why are there so many macros that mean the same thing? -- %scl,
%pkg_name, %scl_name, %name

=== Specific comments on the draft ===
 * Move template to the bottom of the draft.  Had many questions after
reading the template that were eventually answered in other parts of
the draft.
 * Style guide is not to have a period at the end of Summary
 * Description in template should probably be more clear that
packagers should fill in a proper description
 * Template skips %prep and %build.  Are these empty?  If so, just
include the empty %prep and %build headers.
 * "The -build subpackage should include Requires: scl-utils-build."
Why a should and not a must?
 * Content about the enable scriptlet implies that the usual path
macros (_bindir etc) are changed in the scl package.  Need to make it
clear what standard macros are being overridden to what. Perhaps
consider "tables" like
http://fedoraproject.org/wiki/Packaging:MinGW#Filesystem_location_macros
but with slightly different columns, e,g, macro | override? | Normally
Modified? | example | description
 * "The macros.%{scl}-config in %{scl}-build package contains the %scl
macro definition (name of the SCL). If you need to use some more
macros for this SCL, add them into this file." <= but there's nothing
in the spec file that obviously creates this.  Need instructions on
how to create this file for manual use and then how to include the
file in the spec file and not have whatever automated macro normally
creates it not overwrite it.
 * "essential for its runtime". "Essential" needs to be explained and
hopefully it is just a link to the earlier part of the doc.
 * -runtime package must be arch specific => it contains
%{scl_prefix}%{_libdir} so it's arch dependent. (and this means the
main package must be arch specific too)

 * Creating the enable file in %install -- would rather see macros
like:  - %scl_add_path %scl_add_ld_library_path %scl_add_manpath than
the HERE-doc style thing that will cut and paste to most specs.
 * "set of macros that contain the values of the original RPM macros"
should be expanded on.  naming of these could be better => establish a
common prefix for anything that is being overridden instead of
%pkg_name vs %_root_.  also pkg_name probably conflicts with some
existing packages.  Maybe something like %_prescl_name,
%_prescl_libdir, etc.  [can change the recommendation from: packagers
should define %pkg_name to packagers should define a macro that is the
value of pkg_name => %{?scl: %global srcname %pkg_name and %{?!scl:
%global srcname %name}
 * Little more guidance needed for Requires. Maybe making the examples
more real-life would clarify things.  ("Since ifconfig is a binary
we're going to use the system version of, do not use scl_prefix with
it. rubygem-foo, otoh, has to be in the path of the ruby interpreter
which this scl is to be a part of. Therefore it needs to use
scl_prefix"). Also, an example of a "Versioned Requires" would be
helpful.
 * Need more wording on Requires: of non-scl packages as they can act
as conflicts.  Something like: Requires: foo = 1.2.3 … where the scl
package is the only thing keeping foo back at that version.  So you
can't update "the OS" without updating the scl.  wording might be
something like "yes, you can require non-scl stuff … but you have to
be very general in your requirements, and if you need something that
might be updated you have to either scl bundle it or be willing to
rebuild the scl when it updates".
 * Should we list somewhere the scl-modified deps scripts?  Otherwise
how will someone creating a new ocaml-scl know whether an ocaml-scl
dep script has already been written?
 * Please elaborate on the function of %scl_package_override()
* In the dealing with macro files section: I think that the "Now
consider a situation[...]" starts a whole new scenario?  Correct?  If
so, needs to be better phrased right now it's associating with the
previous examples but it doesn't seem to make sense in that context.
* So If I've understood everything correctly, building in a standard
buildroot will get you a non-scl'd package.  Building with the
$scl-build package installed will build the scl version?
  * Should say that in the first line of Building Packages
* In Building Packages: Don't need to include scl-utils-build
explicitly in the config_opts line (brought in via dependencies),
correct?


== Design ==
 * What directories is an SCL allowed to modify?
 * Naming of packages is a problem as currently designed. Liklihood of
conflict with native packages is very high. Propose to include "scl-"
at the begining of names.
 * Include "dots" in names, e.g. ruby1.9 rather than ruby19

== Conversion to SCLs ==
 * Parts of this section are more "tips & tricks"? Should it be on a
non-guideline page?
   * Probably best to vette the advice anyway and then move what makes
sense to a separate page afterwards.
 * Add information about why /usr/bin/env is not a generally good
thing in SCLs.  If this is more general than SCLs, does FPC need to
revisit a ban on /usr/bin/env in shebangs?
 * sed line should limit to the first line of the file as well as the
shebang pattern.
 * how  does scl deal with compiled-in paths inside of binaries?  Does
it  depend on %configure to do the right thing with %{_libdir} etc?
If so  we should point out to people that old-school builds which
aren't  configure driven may need patches (actually, this applies to
install  location as well as compiled-in paths and to scripts that
have file  paths embedded as well as binaries)
 * How do auto provides and requires for elf libraries work?

== Inter-SCL Dependencies ==
* Example of using this with versioned Requires?
* The %scl_require example just doesn't seem right somehow.... The
ruby193 scl brings in rails-3.2.3?  That seems like poor naming (if
the metapackage brings in rails, then it should be named rails.  OTOH,
if the package purpose is to provide ruby193 then rails should be
optional not essential.) or poor dependency management (Which would be
part of the higher level questions of what can be an SCL and what can
be inside an SCL).

* Are non-scl packages allowed to dep on scl packages?  If so, how do
they do so?

* The filters don't work in EL6 thing.... people seem more excited
about SCLs in EPEL than in fedora so we probably need to document how
to filter the provides and requires for EL6.  However, we can point to
the EPEL6 Differences page from here and document the behaviour there.



== Build & Distribution ==
 * What are the impacts on the build system?
   * How are we preventing build problems from the mainline package
from interfering with the scl package and vice versa?
   * How are we allowing for a package to be built in more than one scl?
     * How are we allowing for differences between scls? (ie:
ruby1.9.3 scl with rubygem-foo1.0; ruby2.0 scl with rubygem-foo2.0)
   * How are we preventing build problems from one scl from
interfering with a different scl?
   * Remi mentions that he thinks it would be done in a separate
branch.  Would we need a branch per scl that the package is built in?
 * How do SCLs get distributed?
   * Is this a "separate repo" or is it just in the normal repo(s)?
     * (toshio) Both.  Some SCLs can go into Fedora Commons which
would continue to have one repo. SCLs in the outer rings would have
some separate structure.
 * Build section would benefit from minor sections, perhaps:
"Overview," "Testing" and "Example"? Sections need a bit more color to
be understandable.

== General Comments ==
 * mingw is a similar set of guidelines (but in separate spec files
rather than the same one.)  So issues like what their naming solves
and tracking of fedora packages could be looked at.
http://fedoraproject.org/wiki/Packaging:MinGW. The mingw packaging
guidelines should also be reviewed for "missing content" vs this
proposal.
 * No definition of a SCL appears in the document
   * Example text: Software Collections (SCLs) give rpm packagers a
means of packaging multiple versions of software for a single
distribution.  The version from the software collection does not
interact with the system version. They can provide backwards and
forwards compat versions of a package compared to what's on the
system.
 * Include "definitions" at the beginning of the document, some
example things that need definition (and suggestions)
   * SCL -> whole software collection? <- originally the term was
"dynamic software collection" for the "whole thing" but it has fallen
in to disuse... we could bring it back
   * SCL defining package -> the srpm-level package that defines the scl.
   * SCL metapackage -> built from the SCL defining package.  defines
the essential packages you get when you install the scl.
   * SCL runtime package -> built from the SCL defining package.  It's
the setup of the system for this SCL including registering the SCL
with the system and setting up the filesystem.
   * SCL build package -> built from the SCL defining package.  It
contains macro definitions needed to build an SCL.
   * SCL package -> any package which can be built for an SCL.
 * Elaborate on what "belongs" in the "-runtime" package
   * Extracting one of these from the sample repository seems like
it's a filesystem package directory tree inside of the scl?  Anything
else?
* From the SCLinFedora page, why the differentiation between regular
package reviews and scl package reviews?

-Toshio