RFC: Proposal for a more agile "Fedora.next" (draft of my Flock talk)

Tue Jul 23 16:25:27 UTC 2013

----- Original Message -----
> From: "Robert Rati" <rrati at redhat.com>
> To: devel at lists.fedoraproject.org
> Sent: Tuesday, July 23, 2013 7:01:33 PM
> Subject: Re: RFC: Proposal for a more agile "Fedora.next" (draft of my Flock	talk)
> 
> I think there is another data point to include in the discussion.  As
> Pete mentioned, a lot of the effort to get Hadoop into Fedora has
> revolved around updating the dependencies.  Most of these changes are
> being sent upstream in the hopes of moving them forward on their
> dependencies.
> 
> However, there are cases where we have had to patch the source and the
> result was not thing we could push upstream.  The tomcat and jsp
> compiler changes was one such patch, but the bigger one in my mind is jetty.
> 
> We patched upward to jetty8 when we were testing against F18, and that
> patch has a chance of being included upstream.  F19 moved to jetty9
> which drastically changed some of the interfaces and dropped support for
> java6 (the version upstream Hadoop uses).  I was able to patch Hadoop to
> make it work with jetty9, but I would not claim it to be an optimal
> solution nor is there much hope upstream would accept a patch to jetty9
> anytime soon because of the java version incompatibility.
> 
> While packagers can update dependencies, they may not be the best people
> to decide on the correct technical direction.  Using jetty9 as an
> example, the interface changed substantially and in such a way that it
> might make more sense to re-architect some areas of Hadoop.  The team
> packaging Hadoop is certainly not the right people to take on that task.
>   With the java incompatibility issues with jetty9, upstream is unlikely
> to take that on any time soon.
> 
> Having packagers continually patch to newer versions of dependencies not
> only potentially moves them further away from upstream, but also could
> decrease the likelihood of patches being accepted upstream because the
> solution is not the direction the project wants to head in.

I think there is some kind of misunderstanding here. You are free to package and maintain 
jetty6, jetty7, jetty8 yourself as much as you want with all the CVEs and etc. By moving 
things to some ring/layer/carpet/whatever you would not get the version of jetty hadoop requires.
It's either you do maintain the whole stack you need or you do adapt to what others have packaged.
You can trust me that I have experienced that many times with the Eclipse stack. 
And maintaining the old version is really not helping that much. Let me give you a simple example
jakarta-commons-httpclient is shipped in Fedora and it has security bugs every month or two which upstream 
would not fix as this was obsoleted by apache httpcomponents years ago. Patching it to fix these CVEs is not fun job
at all for the sake of some lazy developers (fwiw jetty is not that different too). If Fedora project decides to 
not care about such things it would be plain spitting on Fedora brand. There is no easy way to overcome this - it costs 
time, a whole lot of time - no matter whether you would patch upstream or maintain some legacy. The first approach at 
least can gain something in the future while the second is a plain time sucker and with every day passed it becomes harder and 
harder to move upstream as they had more and more things workarounding old way of working.

Alexander Kurtakov
Red Hat Eclipse team

> 
> Rob
> 
> On 07/22/2013 05:54 PM, Peter MacKinnon wrote:
> > Shout out to my fellow Flocker, Matt...
> >
> > -------- Original Message --------
> >> Subject: RFC: Proposal for a more agile "Fedora.next" (draft of my
> >> Flock talk)
> >> Date: Mon, 22 Jul 2013 09:38:54 -0400
> >> From: Matthew Miller <mattdm at fedoraproject.org>
> >> Reply-To: Development discussions related to Fedora
> >> <devel at lists.fedoraproject.org>
> >> To: Fedora Development List <devel at lists.fedoraproject.org>
> >>
> >> <snip>
> >>
> >> Obviously, no-bundled-libs is a crucial part of the packaging guidelines
> >> today. As a sysadmin, I know why it's important. This is not just a noble
> >> goal, but also something that pragmatically makes systems better. But,
> >> it's
> >> also keeping us from having software that people really use in Fedora.
> >> Chef
> >> and Hadoop are two big examples. This hurts us more than it helps the
> >> world.
> >> So, in some areas, we need a different approach.
> >>
> >
> > The Big Data SIG is trying to adapt Hadoop 2.x into Fedora for F20
> > <https://fedoraproject.org/wiki/Changes/Hadoop>, and I'll be sharing our
> > insights on this at Flock <http://sched.co/185PNI9> in a couple of
> > weeks. In Matt's conceptual architecture I suppose Hadoop Common would
> > live in the Ring 2-to-3 orbit somewhere. It is a core in it's own right
> > (it provides a distributed, replicated file system) in that there is an
> > every growing software ecosystem that has emerged around it, and the SIG
> > would like Fedora to be the OS of choice for that ecosystem. Stable
> > enough for deployment but a feature-rich, current and productive
> > environment for the developers in that evolving ecosystem. The Hadoop
> > runtime is an orchestration of JVM-based daemons which can be viewed as
> > system-level services, thus an obvious candidate for well-defined
> > integration with Fedora via packaging: correct permissions, systemd
> > scripts, logs, etc.
> >
> > However, the root of that core is a set of older and deprecated Java
> > dependencies (e.g., Jetty 6, Tomcat 5.5) which are expressed via the
> > Apache Maven build tool. The "quick and dirty" label used by another
> > poster of a VERY popular build tool like Maven does it a disservice. The
> > fact is that it is exceedingly popular in the Java development community
> > and has been for some time. Anyway, the challenge for this project is
> > the reconciliation of it's stable dependencies versus the ever-changing
> > bleeding edge that is typically found in the latest Fedora release. A
> > lot of our efforts so far have been the various API and build
> > specification changes necessary to try to make Hadoop fit into Fedora.
> >
> > So far, so good...sort of. We can make the basic use case and tests work
> > with the modified dependencies but in doing so we risk giving up parity
> > with the Apache baseline (including the JRE) and potentially lose out to
> > other so-called "dirty RPMs". Ideally, we wouldn't be forced into some
> > of these adaptations and compromises if there were Fedora packaging
> > alternatives that would give us (a SIG ring?) more control over the
> > bundles needed by Hadoop as opposed to the ones mandated by the latest
> > Fedora release. Make no mistake: patches are fed from the SIG to the
> > Hadoop community to try to bump the versions there. But the upstream
> > project can't and won't chase an ever-vanishing point in the distance.
> > They view their lower dependencies much like a stable OS such as RHEL
> > and change should be deliberated there.
> >
> > I feel like Matt has at least kick-started the discussion around how
> > Fedora could evolve to support orthogonal dependency models that more
> > readily adapt to external projects like Hadoop. Not that our SIG has any
> > profound answers. :-)
> >
> > Thus, we are very interested in _any_ packaging architecture proposals
> > that could help relieve our initiative's pain points, and look forward
> > to further constructive discussion of the same.
> >
> > My $0.02,
> > \Pete
> >
> >
> > --
> > Peter MacKinnon
> > MRG Grid/Big Data
> > Red Hat Inc.
> > Raleigh, NC
> >
> >
> >
> --
> devel mailing list
> devel at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/devel