[Fedora-packaging] SCL Notes and Questions
Toshio Kuratomi
a.badger at gmail.com
Thu Sep 26 15:53:11 UTC 2013
Note, I'm posting the complete list of questions and answers onto a wiki
page. But please, do discussion here and I'll update the wiki page as
discussion progresses (wiki will allow to easily see what questions are
answered. Mailing list will allow for easier discussion.)
On Tue, Sep 17, 2013 at 05:33:03PM +0200, Marcela Mašláňová wrote:
>
> >* One of the goals of SCLs is isolation from the rest of the system,
> >correct? It seems, though, that once you run the enable script, that
> >shell would have a different PATH with the possibility of a different
> >language interpreter. As long as scripts can use /usr/bin/env as a
> >shebang or scripts are using PATH-based commands these will end up
> >using programs from the SCL. Should FPC revisit banning these? Is it
> >okay to run with SCLs even without banning? Just something to
> >document?
> >
> There are mainly two possibilities how to run apps in collections. It
> depends what you wish to do with your application. Please do not ban
> anything, people have to decide for themselves what is their
> use-case.
>
> 1/ #!/usr/bin/env something is good for developers who want to switch
> between versions. I guess on Fedora will be some developers.
> 2/ #!/opt/vendor/something/root/usr/bin/something it's used as
> wrapper, good for deployed apps, because those who deploy app can
> define in wrapper, which collections have to be enabled by default
>
> We do not provide wrappers inside collections, projects usually write
> them for themselves.
> I would be fix it only in documentation.
>
I think you're discussing the use of /usr/bin/env inside of an scl which was
a question asked under iConversion to SCLs. This section was actually asking
about the following type of scenario:
The Fedora Core provided yum package has /usr/bin/yum. The shebang for
that is: #!/usr/bin/env python A sys admin has both that package installed
and a python2.4 scl. When the sysadmin enables the python2.4 scl, yum will
suddenly start using the scl provided python executable (I assume, only in
the shell that the sysadmin enabled the scl in? Correct me if the scl can
somehow be enabled for a user's whole session instead of within a single
shell session). This is very likely to break yum.
FPC discussed this issue
https://fedoraproject.org/wiki/Script_Interpreters_(draft) before:
http://meetbot.fedoraproject.org/fedora-meeting/2009-08-19/fedora-meeting.2009-08-19-16.01.log.html#l-38
and at the time decided not to implement a ban on /usr/bin/env in shebangs.
I'm wondering if SCLs change the need for that as before the scenario was
that the end user had to install something from source in order to break
scripts which use /usr/bin/env. Now we'll provide packages that break
those scripts. However, the end user still has to manually enable the scl
so perhaps it's still something that we should just stick in documentation.
> >== Criteria: When is a Collection the Right Choice? ==
> >
> > * Naming of scls with inter-scl dependencies needs to be worked
> >out. You might have people who want to package: rails3 + ruby1.9.3,
> >rails4 + ruby1.9.3, rails3 + ruby2.0, rails4 + ruby2.0 and all of
> >those need unique names.
> >
> We are afraid that giving so much space for combination might lead to
> chaos. I do not want to see many combinations of ruby and rails of
> different versions with different versions of bundler etc.
> I proposed having SCL SIG which would say something like: "We already
> have Ruby193 and Rails2.x. If you want to just update rubygem-XY it's
> fine with us." or "We have it, updating rubygem-XY will broke it for
> sure for those users, create your own collection."
(Quick note -- if we allow these at all then we need to figure out naming
that allows for it. This bullet was about naming but that doesn't seem to
be addressed at all in the anwer :-).
>
> I agree having smaller collections than we have now would be
> improvement, but I'd like to give boundaries to all those possible
> combinations. In Fedora must be shipped only those collections, which
> are known as stable, well tested, with packages working together.
>
> I'm not sure about this whole idea. The freedom might be a good
> thing, but we would have to set boundaries for what will go into
> Fedora. I do not have any idea how to do it, then give power to those
> who work on SCL or testers in bodhi, but than we would need really
> huge number of +1 or someone with veto.
I expressed some of this in the ticket but let me go a little more in depth
here:
I agree that it would be good for maintenance to limit the combinations.
However, the method of limiting that I'd propose is to have written
criteria that prevents too many combinations. If having a group decide on
allowing exceptions is in lieu of well defined criteria then I think that's
a bad idea as it leaves everyone wondering where the line should be drawn
and leaves people feeling slighted when their SCL is rejected but another
SCL that they consider similar is accepted. We have experience with
something like that in the bundled library exception process. Many people
find it easier to ask for a bundling exceptions than to analyze the
situation and try to correct the code first and its a constant struggle to
be sure that we're fairly applying the guidelines.
Now as for what group should apply the written criteria to the proposed
SCLs.... I could see this as something that packagers take on similar to
package reviews. IMHO, this should be the case if we can come up with
fairly complete criteria. If we think that our criteria leave some gaping
holes we may need to do something more like the bundled library exception
process where new SCLs are approved by a group that follows the criteria.
From past experience, the group should not be subject to high turnover,
should not be elected, and should have a focus on documenting new criteria
when an SCL is accepted that doesn't seem to fit under the existing
criteria. These are because the group needs to be consistent in how it
applies the criteria, should be very willing to say no, and needs to record
their decisions in a manner that allows people to have their SCLs revisited
for inclusion as the criteria changes. SIGs are probably much too ad hoc to
do this. Perhaps the Stacks and Environments Working Group would be okay
but they might have other things that they'll need to work on (and we don't
know how it will compose itself moving into the future either... it might be
higher turnover than is good for a group making these decisions).
>
> >* What may the purpose of the Collection be?
> > * Criteria should indicate that a "language" may be included but
> >not a framework (Rails) or a complete platform (LAMP). However,
> >optional, "ease of use" packages may be included but not installed by
> >the main meta-package. Otherwise, the scls will need to change/fork
> >too fast
> > * (toshio): langdon is convinced this is the right path but I'm
> >not sure. There's a lot of problems with how this interacts with
> >backwards compat requirements. I think a whole package has to be
> >worked out with how those shape up.
> Collection in Fedora should give developers, who are not ready to
> move to latest version, time to develop on older version.
>
> Typically, Ruby and Rails, Python and mod_wsgi, but I do not suggest
> to add fast moving frameworks. We have cpan/pip for such packages.
>
If we want to enable ruby on rails then we really are looking at enabling an
scl up to the framework level. How we implement that is separate from that
-- the framework could be a part of the language scl or it could be its
own scl with a language dependency or the language could be a part of the
framework scl.
> > * The "API" of an scl should never "shrink" only "grow." For
> >example, if Rails3 is included in the Ruby1.9 SCL, and Rails4 wants to
> >be included it does not replace Rails3. If, hypothetically, Rails4
> >cannot be installed "next to" Rails3, a new SCL would need to be
> >created to support Rails4, which requires a "real" change.
> Agreed.
> > * (toshio) What would the new SCL be named?
> > * (toshio) This is problematic without also specifying a lifetime
> >for SCLs and replacement of existing SCLs. Some of the knock-on
> >questions around this:
> > * What are we going to do about security issues in SCLs? Does
> >our policy say that backwards compat is more important than security
> >issues?
> No. We do not want to support not supportable (terribly broken) versions.
So... what do we do with them? Do we break backwards compatibility in those
cases? Do we remove the SCL? what happens if the security issue is just in
a piece of the SCL? For instance, if we have a thick SCL of ruby1.9
+ rails3 and rails3 has an unresovled security issue, do we have an option
to only drop the rails portion?
> > * Maybe Fedora simply isn't about backwards compat to this
> >extent and so we have to allow API breaks within a release.
> No, it would loose the purpose of scl for projects building above it.
It would lose one part of the purpose but there are other reasons for scls
to exist. ** Note: We passed the question of backwards compat to fesco
at today's FPC meeting. So we should have this question answered externally
at the next meeting.
https://fedorahosted.org/fesco/ticket/1175
> > * Maybe the policy for SCLs in The Outer Rings should have a
> >strict backwards compat section but SCLs in Fedora Commons are more
> >relaxed (that would make the model: Fedora Commons SCLs are primarily
> >for enabling different versions of dependencies for software that we
> >want to ship in Fedora Commons; Outer Ring SCLs are primarily for
> >enabling developers to target an SCL rather than Fedora as their
> >platform.)
> Rings are still unclear.
>
So are the guidelines for SCLs :-) But we have to anticipate that they're
coming and figure out how the pieces all fit together.
> > * Only partially addressing the situation, could figure out
> >some way to mark some pieces of an SCL as having guaranteed backwards
> >compat while others are only implementation detail. If you had an SCL
> >that shipped ruby1.9 but it needed a newer version of OpenSSL you
> >might ship that inside the SCL at first. But you wouldn't want people
> >to rely on that particular version being available in the SCL in the
> >future (once the base OS was updated to include that version.)
> I might misunderstand. Do you suggest collection should "bundle" such
> package as openssl? That's security hole, not good for Fedora in
> circle 1, 2 or 3.
>
I don't suggest it "should". I suggest it is a requirement of enabling scls
(at least as it was explained to me). An SCL provides a layer that can be
used as compatibility between OS releases and even between OS's. The
further the bases you are targetting get from each other, the more packages
that you may need to include in order for your SCL to function on all of
those platforms. So at some point we aren't going to be able to ship the
same backwards compatible SCL on a newer version of Fedora without also
shipping something that it depends on. My example above was openssl but the
dependent package could be either better or worse.
If we don't want SCLs to include thngs they depend on we need to put in
criteria to that effect. Possibilities:
* SCLs must be built on other packages from the system. they cannot include
packages that they depend on.
* SCLs must not include any of the following packages:
* a libc implementation
* crypto implementations (bindings to system crypto libraries are okay but
be careful -- if the API of the library changes in the future and those
details leak through to the bindings you are giving to users of your SCL
the SCL may need to be retired because it would be a backwards
incompatible change.
> > * No matter where it is, need to have some criteria aroud how
> >to obsolete an SCL, how to remove an SCL altogether, and whether two
> >separate SCLs can ship largely the same thing.
> Collection can be obsoleted same way as other packages, but let's say
> that maintainer should say, he do not want to support it for F-22
> anymore (before its release). Those still interested could carry on.
>
It's more difficult than this though... Let's say we have a ruby1.9 + rails3
SCL. No one wants to maintain this any longer. Someone else would like to
maintain a ruby1.9 without rails3 SCL. Are they allowed to do so? Can it
take the old SCL's name? Does it have a new name? If so, what does that
new name need to be? If a new maintainer could do this, is there anything
stopping the old maintainer from doing the same thing in order to make their
backwards incompatible change? removing an SCL: Does an SCL get removed
from all releases of Fedora or only from the rawhide branch? Is the name
available for reuse? Immediately? After the last Fedora Release it was
present in has gone EOL?
> > * (toshio) Can we make a recommendation on how or when to use
> >scl-dependencies. For example, in the Rails4 example above, there may
> >be scenarios that we want to allow this in Fedora. Although, the
> >recommendation could be that frameworks and the like should be
> >provided outside of Fedora Commons.
> Do you mean dependencies among collections?
>
Yep.
>
> >== Naming ==
> >* problem with the examples (ruby193 and examples from:
> >http://fedorahosted.org/SoftwareCollections ) For instance, python3.3
> >would conflict with a python3.3 parallel stack that did not use scls.
> Not true. We used python33 as a metapackage without dot.
>
Uh huh. There's two reasons I'm going to say this argument doesn't hold
water:
* The naming guidelines allow versions in package names to either use the
dot or drop it (mostly historical... dot used to be prohibited but whenwe
were presented with the fact that versions were being put into names of
compat packages, we voted to allow dots if those were part of versions.
But the current guidelines don't prohibit not using a dot in versions
either: "Note that we do not use delimiters in the name in this situation.
We attach the version number to the name, optionally removing the dot from
the version.").
* Officially promote package names that differ only in whether they have
a dot or not is a road to madness: You'll have people installing the
wrong package, misfiling bugs, blogging instructions with the wrong name,
looking up ownership information of the wrong component, etc etc etc.
It's bad enough when we have upstreams that choose similar names (ddrescue
vs dd_rescue) that we're not going to make Fedora Policy that
canonicalizes doing the same thing.
> >* Heard about namespace prefixes and that there were thoughts to name
> >them after vendors (rht-ruby1.9). If we use /opt/$vendor that might
> >make sense.
> > * Could still have clashes with things like redhat-release,
> >redhat-rpm-config, etc... make sure the vendors we use doesn't have
> >any pre-existing conflicts of this sort.
> >* scl makes sense as well. mmaslano had said that end-users wouldn't
> >know what an scl was but remi says you'd need to use a tool called
> >"scl" to manage which scls are active anyway. So that may not be a
> >problem?
> That's true. We were also thinking about users who are interested in
> app and know nothing about scl, so they just install app, run it and
> are not aware of scl at all.
>
Ah, yeah -- so that would tie into whether we want to allow packages which
are not scls to depend on scls. If we don't allow that, then an application
that depended on the version of a stack in an scl (puppet depending on
ruby1.9.3, for instance).would have to be in its own scl. And then the
package name would be scl-puppet.... OTOH, there's also the desire by
langdon that the SCLs store as few resources as possible outside of
/opt/vendor/scl. If we follow that, then the user would still have to scl
enable the puppet scl to get it working. (OTOH, we could decide that is
a non-goal and ship a wrapper).
In my vision of the non-scl-depending-on-scl case a wrapper would be used so
invoking puppet will scl enable the ruby1.9 scl before running puppet.
> > * If we have multi-vendor for scl (fedora, rpmfusion, local admin
> >case again), then vendor may make more sense again.
> >* Change example naming: ruby193 => ruby1.9.3 Dots are clearer and
> >thus preferred
> Dots and such stuff are evil. I guess dots were omitted on purpose.
>
Dots in version are less confusing. Is ruby111 Ruby-11.1, Ruby-1.1.1 or
Ruby-1.11? The guidelines allow non-dots in versions of mainstream packages
because dots were on the prohibited list before and it proved confusing. We
shouldn't make the same mistake with a brand new area.
> >
> >== File Location ==
> >
> >* If we're going to use /opt/vendor:
> > * scls should then obey the rest of the standard: /var/opt and
> >/etc/opt are for host specific state and config. Is that the case?
> >can it be fixed if not?
> Sometimes were used locations in /var/log or /etc because collections
> wouldn't work in case of mariadb and other daemons. We had to put
> init file somewhere and also logfile. It's documented and we didn't
> find better solution yet.
>
<nod> I think we can't avoid storing some files in /var and /etc. We have to
assume that /opt follows the same pattern (shared between hosts; read-only)
as /usr which makes it unsuitable for some files. As part of this question
I was wondering if we're following the FHS specification to use /var/opt/
and /etc/opt/ instead of things in either the "normal" /etc/ and /var/ or
in the /opt/vendor/scl hierarchy and if we aren't whether it would be
possible to do so.
>
> >* What does the filesystem layout look like inside of this prefix?
> >/opt/$vendor/scl/rht-ruby1.9.3/{normal filesystem}?
> yes, normal filesystem.
and it's /opt/$vendor/scl ?
I'm checking that scls are less likely to conflict with other vendor packages.
> >* Does this allow local sites and addon repos to make their own
> >vendor? How? (Thinking of rhel/centos, epel, and local admin here.
> >Or fedora/rpmfusion, and local admin)
> Modify vendor in macros of scl.
Is this the macros inside the SCL spec file? Inside the
scl-utils package? Or elsewhere? Is it one macro to change and it affects
everything else?
> >
> >== Spec Files ==
> > * Why don't we follow the mingw model and have separate spec files
> >(mingw is both separate spec and separate package) for scl?
> >Advantages:
> > * Many fewer conditionals in spec files (greater readability/maintainabilty)
> > * Primary maintainer doesn't need to know or care about SCLs if
> >other people do
> > * SCL could be the only spec that is necessary or desired for a
> >particular component
> > * Can a SCL spec be "only" an SCL spec (will not build a non-scl
> >rpm)? If so, we need some more content describing this scenario.
> > * Propose moving the "style guide" to before the example (e.g.
> >Section 3 before Section 2), so that some of the following questions
> >don't come up for the reader
> > * Why are there so many macros that mean the same thing? -- %scl,
> >%pkg_name, %scl_name, %name
> >
> I believe we should have branch for scl and other branches might
> voluntarily add macros, but it shouldn't be mandatory. For example
> perl or httpd don't have nice specfile and adding more condition
> wouldn't improve readability.
> Having scl macros might improve speed of patching some packages, but
> not all.
>
With this and the other information about how you're currently building I'm
starting to think that following the mingw model of separate packages is
a better choice than doing these in the same spec file in the same git tree.
But I will admit that I don't understand all of what you are currently
doing so I'm not 100% settled on this opinion yet.
> >== Design ==
> > * What directories is an SCL allowed to modify?
> > * Naming of packages is a problem as currently designed. Liklihood of
> >conflict with native packages is very high. Propose to include "scl-"
> >at the begining of names.
> Why don't we already prefixed scl before all packages? Because it
> would be scl-ruby193-rubygems-blabla-1.1.f21. The prefix must not
> conflict with existing packages. It doesn't have to be ruby193, it
> might be name which specify what is inside.
I'm not sure what you're getting at here? Could you give some full examples
to explain?
> > * Include "dots" in names, e.g. ruby1.9 rather than ruby19
> I prefer without dots, punctuation does everything harder.
I explained the rationale for dots earlier. Could you write some
justification for how leaving the separator in the version is "evil" or
"harder".
> >
> >== Build & Distribution ==
> > * What are the impacts on the build system?
> Until now we used internal instance of koji. I can sum up what we did there.
>
Please do.
> > * How are we preventing build problems from the mainline package
> >from interfering with the scl package and vice versa?
> There are no build interference problems. Package has specific name.
>
Not just name but also preventing changes to the spec file. From earlier
answers, my undertanding is that you have a separate branch for each scl
needing to be built for now.
> > * How are we allowing for a package to be built in more than one scl?
> Didn't solve yet. We related build targets (collection) with branch.
> How to build for more collections is a good question for our releng.
> But do we need it?
Yeah, almost definitely. The idea I've heard is that dependent packages
that are no longer provided by the system would need to be packaged inside
of an scl. If they aren't in a separate scl then they would need to be
packaged inside of each scl individually.
You could also have two different scls that require different versions of
a third package.
Oh -- one note about terminology; we should avoid "collections" as
a synonymous term with scls because collections and branches are already
synonymous terms.
> > * How are we allowing for differences between scls? (ie:
> >ruby1.9.3 scl with rubygem-foo1.0; ruby2.0 scl with rubygem-foo2.0)
> > * How are we preventing build problems from one scl from
> >interfering with a different scl?
> > * Remi mentions that he thinks it would be done in a separate
> >branch. Would we need a branch per scl that the package is built in?
> Can't happen. Every branch is set to build in some tag. Packages
> tagged in one tag are shipped as one collection. You are able to
> inherit from one tag to another.
> As I said I prefer branch per collection. How else would be tags defined?
I'm afraid I'm not understanding the big picture here. Or perhaps the
details. Anyhow -- I'm not understanding what you're doing and what you
think is impossible and whether that's what Remi had in mind or not.
> > * How do SCLs get distributed?
> > * Is this a "separate repo" or is it just in the normal repo(s)?
> Currently separate repo.
> > * (toshio) Both. Some SCLs can go into Fedora Commons which
> >would continue to have one repo. SCLs in the outer rings would have
> >some separate structure.
> Agreed.
> > * Build section would benefit from minor sections, perhaps:
> >"Overview," "Testing" and "Example"? Sections need a bit more color to
> >be understandable.
> >
> I'll add it.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/packaging/attachments/20130926/16ef6ebe/attachment.sig>
More information about the packaging
mailing list