Is it time to remove the existing Big Data packages from Fedora?

[Gerald Henriksen]

So the discussion about Spark has revealed:

1) the Big Data community doesn't trust the Fedora packages

2) it seems a consensus that the realities of the Big Data ecosystem
don't work very well with making packages for Fedora

3) result of 2, move towards newer (Docker) style of making Big Data
available on Fedora, perhaps with a side benefit of it then also being
possible to have it available on RHEL.

Should we then also have a discussion, or just do it, of removing the
existing packages from Fedora 23?

The packages that are there are now in some cases at least several
versions out of date, which isn't doing either Fedora or the Big Data
SIG any favours.

Additionally, if we do create Docker images they offer the possibility
of creating bug/error opportunities if a user somehow brings an rpm
package into the Docker image.

To use the now favourite term, do we "reboot" the Big Data SIG and
start fresh?

Thoughts?

[Peter Robinson]

On Fri, Jul 31, 2015 at 3:27 PM, Gerald Henriksen <ghenriks(a)gmail.com&gt; wrote:
>  So the discussion about Spark has revealed:
>
>  1) the Big Data community doesn't trust the Fedora packages
>
>  2) it seems a consensus that the realities of the Big Data ecosystem
>  don't work very well with making packages for Fedora
>
>  3) result of 2, move towards newer (Docker) style of making Big Data
>  available on Fedora, perhaps with a side benefit of it then also being
>  possible to have it available on RHEL. 
Where does the contents of the docker images come from? There are
plans for layered images in Fedora but ultimately the content of said
images still comes from rpm content in Fedora.

>  Should we then also have a discussion, or just do it, of removing
> the
>  existing packages from Fedora 23?
>
>  The packages that are there are now in some cases at least several
>  versions out of date, which isn't doing either Fedora or the Big Data
>  SIG any favours.
>
>  Additionally, if we do create Docker images they offer the possibility
>  of creating bug/error opportunities if a user somehow brings an rpm
>  package into the Docker image.
>
>  To use the now favourite term, do we "reboot" the Big Data SIG and
>  start fresh?
>
>  Thoughts?
>  _______________________________________________
>  bigdata mailing list
>  bigdata(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/bigdata

[Gerald Henriksen]

On Fri, 31 Jul 2015 16:09:56 +0100, you wrote:

> On Fri, Jul 31, 2015 at 3:27 PM, Gerald Henriksen
> <ghenriks(a)gmail.com&gt; wrote:
> > So the discussion about Spark has revealed:
> >
> > 1) the Big Data community doesn't trust the Fedora packages
> >
> > 2) it seems a consensus that the realities of the Big Data ecosystem
> > don't work very well with making packages for Fedora
> >
> > 3) result of 2, move towards newer (Docker) style of making Big Data
> > available on Fedora, perhaps with a side benefit of it then also being
> > possible to have it available on RHEL.
>
> Where does the contents of the docker images come from? There are
> plans for layered images in Fedora but ultimately the content of said
> images still comes from rpm content in Fedora. 
But the whole point of going to docker images was the reality that
Hadoop and company are basically unpackageable in Fedora.

If the docker images need Hadoop and its ecosytem to be packaged up
into official rpm packages, then there won't be any Fedora Big Data
docker images.

[Javi Roman]

The more "simple" option is again a Fedora big data software
collection (with multiple versions of libraries - for
Java/Scala/Clojure) for populate the docker image.
--
Javi Roman


On Fri, Jul 31, 2015 at 6:46 PM, Gerald Henriksen <ghenriks(a)gmail.com&gt; wrote:
>  On Fri, 31 Jul 2015 16:09:56 +0100, you wrote:
>
> >On Fri, Jul 31, 2015 at 3:27 PM, Gerald Henriksen <ghenriks(a)gmail.com&gt; wrote:
> >> So the discussion about Spark has revealed:
> >>
> >> 1) the Big Data community doesn't trust the Fedora packages
> >>
> >> 2) it seems a consensus that the realities of the Big Data ecosystem
> >> don't work very well with making packages for Fedora
> >>
> >> 3) result of 2, move towards newer (Docker) style of making Big Data
> >> available on Fedora, perhaps with a side benefit of it then also being
> >> possible to have it available on RHEL.
> >
> >Where does the contents of the docker images come from? There are
> >plans for layered images in Fedora but ultimately the content of said
> >images still comes from rpm content in Fedora.
>
>  But the whole point of going to docker images was the reality that
>  Hadoop and company are basically unpackageable in Fedora.
>
>  If the docker images need Hadoop and its ecosytem to be packaged up
>  into official rpm packages, then there won't be any Fedora Big Data
>  docker images.
>  _______________________________________________
>  bigdata mailing list
>  bigdata(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/bigdata

[Christopher]

On Fri, Jul 31, 2015 at 10:27 AM, Gerald Henriksen <ghenriks(a)gmail.com&gt; wrote:
>  So the discussion about Spark has revealed:
>
>  1) the Big Data community doesn't trust the Fedora packages 
> From my perspective, I see a lot of things happening in the "Big
> Data 
community" which indicates not so much distrust for Fedora packages
(or RHEL, for that matter), but more a complete lack of understanding
about the benefits of packaging for these projects.

>  2) it seems a consensus that the realities of the Big Data ecosystem
>  don't work very well with making packages for Fedora 
I don't think that's necessarily true. I think a lot of the community
is following the "Hadoop Way", where developers are just trying to
mimic the way Hadoop does things, but Hadoop seems to be pretty poorly
packaged and has had a lot of problems over time, from complete lack
of dependency convergence, lack of packaging standards, improper use
of the Linux filesystem hierarchy, confusing use of shell scripts and
environment variables set all over the place, etc.

A lot of this is where downstream packagers can bring good things back
to the upstream communities. So, I don't think it's that the ecosystem
doesn't work well with Fedora packaging... I think it's more that they
haven't had sufficient feedback from those communities to push them to
make their software more easily integrated. (Part of this is a Java
thing more than a big data thing... let's face it... Java packaging
hasn't really had a lot of good support until quite recently, with
xmvn and related tools.)

>  3) result of 2, move towards newer (Docker) style of making Big Data
>  available on Fedora, perhaps with a side benefit of it then also being
>  possible to have it available on RHEL. 
Docker still has some limitations, and is not necessarily suitable to
everybody's situation. I'd still prefer Fedora RPMs over Docker-based
packaging. I think the community could benefit from both.

>  Should we then also have a discussion, or just do it, of removing the
>  existing packages from Fedora 23? 
I'd be against that. It's one thing if there's no maintainer, but it's
another to decide to oust a whole class of projects.

>  The packages that are there are now in some cases at least several
>  versions out of date, which isn't doing either Fedora or the Big Data
>  SIG any favours. 
Big Data packages aren't the only packages to suffer from this.
There's many projects that are out of date. I'd rather Fedora have a
relatively stable version than no version at all. Maybe instead we
should try working with the upstream communities more to ease
integration, so it's easier to update?

>  Additionally, if we do create Docker images they offer the possibility
>  of creating bug/error opportunities if a user somehow brings an rpm
>  package into the Docker image.
>
>  To use the now favourite term, do we "reboot" the Big Data SIG and
>  start fresh?
>
>  Thoughts? 


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

[Javi Roman]

--
Javi Roman


On Sat, Aug 1, 2015 at 1:22 AM, Christopher <ctubbsii-fedora(a)apache.org&gt; wrote:
>  On Fri, Jul 31, 2015 at 10:27 AM, Gerald Henriksen
> <ghenriks(a)gmail.com&gt; wrote:
> > So the discussion about Spark has revealed:
> >
> > 1) the Big Data community doesn't trust the Fedora packages
>
>  From my perspective, I see a lot of things happening in the "Big Data
>  community" which indicates not so much distrust for Fedora packages
>  (or RHEL, for that matter), but more a complete lack of understanding
>  about the benefits of packaging for these projects.
>
> > 2) it seems a consensus that the realities of the Big Data ecosystem
> > don't work very well with making packages for Fedora
>
>  I don't think that's necessarily true. I think a lot of the community
>  is following the "Hadoop Way", where developers are just trying to
>  mimic the way Hadoop does things, but Hadoop seems to be pretty poorly
>  packaged and has had a lot of problems over time, from complete lack
>  of dependency convergence, lack of packaging standards, improper use
>  of the Linux filesystem hierarchy, confusing use of shell scripts and
>  environment variables set all over the place, etc.
>
>  A lot of this is where downstream packagers can bring good things back
>  to the upstream communities. So, I don't think it's that the ecosystem
>  doesn't work well with Fedora packaging... I think it's more that they
>  haven't had sufficient feedback from those communities to push them to
>  make their software more easily integrated. (Part of this is a Java
>  thing more than a big data thing... let's face it... Java packaging
>  hasn't really had a lot of good support until quite recently, with
>  xmvn and related tools.)
>
> >
> > 3) result of 2, move towards newer (Docker) style of making Big Data
> > available on Fedora, perhaps with a side benefit of it then also being
> > possible to have it available on RHEL.
>
>  Docker still has some limitations, and is not necessarily suitable to
>  everybody's situation. I'd still prefer Fedora RPMs over Docker-based
>  packaging. I think the community could benefit from both.
>
> >
> > Should we then also have a discussion, or just do it, of removing the
> > existing packages from Fedora 23?
>
>  I'd be against that. It's one thing if there's no maintainer, but it's
>  another to decide to oust a whole class of projects.
>
> >
> > The packages that are there are now in some cases at least several
> > versions out of date, which isn't doing either Fedora or the Big Data
> > SIG any favours.
>
>  Big Data packages aren't the only packages to suffer from this.
>  There's many projects that are out of date. I'd rather Fedora have a
>  relatively stable version than no version at all. Maybe instead we
>  should try working with the upstream communities more to ease
>  integration, so it's easier to update?
>  
The problem here is the big data tools, for example Apache Flume, or
Apache Spark are evolving quickly with new important features, however
some of the libraries used by those tools (for example Java artifacts in Flume)
have frozen versions, because it's works for the developer. In Fedora
some of those
libraries are increasing their versions (the most updated versions
from the upstream project), and the big data tool affected breaks in
compilation time.

So we have two options:

1. The package maintainer (of the tools) must to patch the tool over
and over, in order to adapt the code to the new interfaces, sometimes
very complex task. And very difficult to follow the Fedora release policy cycle.

2. Try to convince the upstream developer for update the version of
the library of his tool. This is very improbable to achieve, because
the developer is thinking in the new features and he is not thinking
about change the way the tools are writing in the log (log4j), for
example.

I think we can not solve easily this situation without multi-version
packages. On the other hand we can help with things such as FHS,
environment variables, weird shell scripts and so forth.

> >
> > Additionally, if we do create Docker images they offer the possibility
> > of creating bug/error opportunities if a user somehow brings an rpm
> > package into the Docker image.
> >
> > To use the now favourite term, do we "reboot" the Big Data SIG and
> > start fresh?
> >
> > Thoughts?
>
>
>
>  --
>  Christopher L Tubbs II
>  http://gravatar.com/ctubbsii
>  _______________________________________________
>  bigdata mailing list
>  bigdata(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/bigdata

[Matthew Miller]

On Sat, Aug 01, 2015 at 09:32:22AM +0200, Javi Roman wrote:
>  The problem here is the big data tools, for example Apache Flume, or
>  Apache Spark are evolving quickly with new important features, however
>  some of the libraries used by those tools (for example Java artifacts in Flume)
>  have frozen versions, because it's works for the developer. In Fedora
>  some of those
>  libraries are increasing their versions (the most updated versions
>  from the upstream project), and the big data tool affected breaks in
>  compilation time. 
This isn't a problem just with big data tools. Many developers want to
do this. The problem is that having all of those multiple versions
becomes a maintenance nightmare. When there's a security problem, how
do you identify which packages have a library with the problem? Which
versions are affected? Does the same patch apply to them all? Who fixes
it if it doesn't?


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Javi Roman]

On Sat, Aug 1, 2015 at 5:48 PM, Matthew Miller <mattdm(a)fedoraproject.org&gt; wrote:
>  On Sat, Aug 01, 2015 at 09:32:22AM +0200, Javi Roman wrote:
> > The problem here is the big data tools, for example Apache Flume, or
> > Apache Spark are evolving quickly with new important features, however
> > some of the libraries used by those tools (for example Java artifacts in Flume)
> > have frozen versions, because it's works for the developer. In Fedora
> > some of those
> > libraries are increasing their versions (the most updated versions
> > from the upstream project), and the big data tool affected breaks in
> > compilation time.
>
>  This isn't a problem just with big data tools. Many developers want to
>  do this. The problem is that having all of those multiple versions
>  becomes a maintenance nightmare. When there's a security problem, how
>  do you identify which packages have a library with the problem? Which
>  versions are affected? Does the same patch apply to them all? Who fixes
>  it if it doesn't?
>  
You are right Matthew.

Probably the way we are working right now it's the correct way:

1. Use the great utility xmvn.
2. Patch the Java/Scala/Clojure ... ecosystem tool in order to use the
most updated upstream libraries.
3. Try to commit this changes to the upstream tool.

So we only need more developers interested in the big data Fedora
ecosystem for this huge task, or some full time position like in the
Fedora Cloud SIG [1], however this last is another matter even more
complex to deal.

[1] http://paul.frields.org/2015/07/21/fedora-engineering-team-opening/

[Gerald Henriksen]

On Sat, 1 Aug 2015 11:48:59 -0400, you wrote:

> On Sat, Aug 01, 2015 at 09:32:22AM +0200, Javi Roman wrote:
> > The problem here is the big data tools, for example Apache Flume, or
> > Apache Spark are evolving quickly with new important features, however
> > some of the libraries used by those tools (for example Java artifacts in Flume)
> > have frozen versions, because it's works for the developer. In Fedora
> > some of those
> > libraries are increasing their versions (the most updated versions
> > from the upstream project), and the big data tool affected breaks in
> > compilation time.
>
> This isn't a problem just with big data tools. Many developers want to
> do this. The problem is that having all of those multiple versions
> becomes a maintenance nightmare. When there's a security problem, how
> do you identify which packages have a library with the problem? Which
> versions are affected? Does the same patch apply to them all? Who fixes
> it if it doesn't? 
I agree it is a packaging nightmare, but the problem is there is no
ideal solution.

Best solution, we try to continue what was started, which means
packaging Hadoop and friends with a lot of work (and patches) to make
things build with the newer libraries that are in Fedora.

The problem here is that even if we get everything packaged we already
apparently have feedback from the Big Data community that they don't
trust, and hence won't use the packages that people have put much
effort into making.  While it wasn't mentioned why this is the case, I
am guessing it is at least in part because nothing is tested with
those newer library versions, and the attitude on the Hadoop mailing
lists will be shrug shoulders and say download the "official" version
of Hadoop or whatever part of the ecosystem is being problematic.

Which is why the idea of simply creating a docker container is so
attractive.  You remove all the time spent either creating/maintaining
the above patched versions, or alternatively packaging and maintaining
older libraries.  We in essence throw the security issue to the
upstream developers, and simply update the docker container with every
Hadoop/Spark/etc release.

Is it a great solution?  Not really, but unless upstream changes there
are no great solutions.

[Tim St. Clair]

----- Original Message -----
>  From: "Gerald Henriksen" <ghenriks(a)gmail.com&gt;
>  To: "Fedora Big Data SIG" <bigdata(a)lists.fedoraproject.org&gt;
>  Sent: Saturday, August 1, 2015 10:33:04 PM
>  Subject: Re: Is it time to remove the existing Big Data packages from Fedora?
>  
>  On Sat, 1 Aug 2015 11:48:59 -0400, you wrote:
>  
>  >On Sat, Aug 01, 2015 at 09:32:22AM +0200, Javi Roman wrote:
>  >> The problem here is the big data tools, for example Apache Flume, or
>  >> Apache Spark are evolving quickly with new important features, however
>  >> some of the libraries used by those tools (for example Java artifacts in
>  >> Flume)
>  >> have frozen versions, because it's works for the developer. In Fedora
>  >> some of those
>  >> libraries are increasing their versions (the most updated versions
>  >> from the upstream project), and the big data tool affected breaks in
>  >> compilation time.
>  >
>  >This isn't a problem just with big data tools. Many developers want to
>  >do this. The problem is that having all of those multiple versions
>  >becomes a maintenance nightmare. When there's a security problem, how
>  >do you identify which packages have a library with the problem? Which
>  >versions are affected? Does the same patch apply to them all? Who fixes
>  >it if it doesn't?
>  
>  I agree it is a packaging nightmare, but the problem is there is no
>  ideal solution.
>  
>  Best solution, we try to continue what was started, which means
>  packaging Hadoop and friends with a lot of work (and patches) to make
>  things build with the newer libraries that are in Fedora.
>  
>  The problem here is that even if we get everything packaged we already
>  apparently have feedback from the Big Data community that they don't
>  trust, and hence won't use the packages that people have put much
>  effort into making.  While it wasn't mentioned why this is the case, I
>  am guessing it is at least in part because nothing is tested with
>  those newer library versions, and the attitude on the Hadoop mailing
>  lists will be shrug shoulders and say download the "official" version
>  of Hadoop or whatever part of the ecosystem is being problematic.
>  
>  Which is why the idea of simply creating a docker container is so
>  attractive.  You remove all the time spent either creating/maintaining
>  the above patched versions, or alternatively packaging and maintaining
>  older libraries.  We in essence throw the security issue to the
>  upstream developers, and simply update the docker container with every
>  Hadoop/Spark/etc release.
>  
>  Is it a great solution?  Not really, but unless upstream changes there
>  are no great solutions. 
IMHO it is the only tenable solution.  



>  _______________________________________________
>  bigdata mailing list
>  bigdata(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/bigdata
>   
-- 
Cheers,
Timothy St. Clair
Red Hat Inc.

[Gerald Henriksen]

On Mon, 3 Aug 2015 09:54:12 -0400 (EDT), you wrote:

> ----- Original Message -----
> > From: "Gerald Henriksen" <ghenriks(a)gmail.com&gt; 
> > Is it a great solution?  Not really, but unless upstream changes
> there
> > are no great solutions.
>
> IMHO it is the only tenable solution.   
Just to note that a similiar issue has just come up on the Fedora Java
mailing list where the person who brought Wildfly into Fedora has
found it impossible to keep it up to date thanks to both time
constraints and the issue that being Java it has required over 100
packages.

[Christopher]

On Sat, Aug 1, 2015 at 3:32 AM, Javi Roman <jroman.espinar(a)gmail.com&gt; wrote:
>  On Sat, Aug 1, 2015 at 1:22 AM, Christopher
> <ctubbsii-fedora(a)apache.org&gt; wrote:
> > On Fri, Jul 31, 2015 at 10:27 AM, Gerald Henriksen <ghenriks(a)gmail.com&gt; wrote:
[snip]
>  The problem here is the big data tools, for example Apache Flume, or
>  Apache Spark are evolving quickly with new important features, however
>  some of the libraries used by those tools (for example Java artifacts in Flume)
>  have frozen versions, because it's works for the developer. In Fedora
>  some of those
>  libraries are increasing their versions (the most updated versions
>  from the upstream project), and the big data tool affected breaks in
>  compilation time. 
That's true of many upstream projects. That problem is not unique to
Big Data, though I will concede it seems to be a larger problem with
Java projects, in general. This is where the downstream, like Fedora,
can provide a huge benefit to users: it can ship versions of projects
that have been updated with the latest known security and stability
fixes, and it can provide dependency convergence to ensure
compatibility between components, and API stability/reliability that
upstream doesn't necessarily care about.

Where the upstream project breaks when using up-to-date versions of
libraries, Fedora has an opportunity (and, perhaps, a responsibility)
to give back to the upstream community.

>  So we have two options:
>
>  1. The package maintainer (of the tools) must to patch the tool over
>  and over, in order to adapt the code to the new interfaces, sometimes
>  very complex task. And very difficult to follow the Fedora release policy cycle.
>
>  2. Try to convince the upstream developer for update the version of
>  the library of his tool. This is very improbable to achieve, because
>  the developer is thinking in the new features and he is not thinking
>  about change the way the tools are writing in the log (log4j), for
>  example.
>
>  I think we can not solve easily this situation without multi-version
>  packages. On the other hand we can help with things such as FHS,
>  environment variables, weird shell scripts and so forth. 
To some extent, Fedora has employed each of these strategies already:
repeated patching, convincing upstream, and supporting multiple
versions. If a particular package is problematic, or the maintainer
isn't up to the task to decide which combination of strategies is
best... well, that's where co-maintainers can be a really big help.
I'd only support dropping if maintainers can't be found who are
willing to support the packages to the extent they need. Isn't that
how things typically work in Fedora?

[Peter MacKinnon]

On 8/3/15 12:38 PM, Christopher wrote:
>  To some extent, Fedora has employed each of these strategies
> already:
>  repeated patching, convincing upstream, and supporting multiple
>  versions. If a particular package is problematic, or the maintainer
>  isn't up to the task to decide which combination of strategies is
>  best... well, that's where co-maintainers can be a really big help.
>  I'd only support dropping if maintainers can't be found who are
>  willing to support the packages to the extent they need. Isn't that
>  how things typically work in Fedora? 
Will had an excellent and succinct rationalization for dropping Apache 
Spark in the other related thread.

"I'm actually planning to retire Spark because the upstream has diverged 
too far from what we can actually support in Fedora -- too many things 
depend on bundled-but-modified libraries, nonstandard or older versions, 
etc., and even in the 0.9.1 package we had to carry a lot of patches to 
make everything work. I think that, given the developement direction the 
Spark community has taken since Spark entered Fedora, the most sensible 
way to have Spark integrated with Fedora going forward is in a Software 
Collection. This is a bummer because it was a lot of work to get Spark 
in to Fedora, but my concern is that having a (necessarily) limited 
Spark that is integrated with Fedora is not going to be sufficiently 
compelling for end-users to warrant not just using upstream binaries, 
and it's a lot of bandwidth for volunteers to take on. I'm open to 
discussion on this matter, but I think this is the best way forward."

The Fedora upstream feedback virtues you describe are laudable but the 
game has changed. That model works well for packages that have both 
limited scope and dependency sets. But these projects are massive, under 
rapid development, and IMHO the upstreams aren't interested in a 
continuous and steady stream of PR or JIRA to upgrade versions, 
unbundle, etc. for the sake of Fedora. We already went down that road 
with Hadoop for example and JIRAs just languished as we continually 
rebased them for a time. It wasn't for want of effort downstream. 
Neither does that make the upstreams "bad" or "evil".

My $0.02,
\Pete

[Haïkel]

+1 Pete

> From re-reading the whole thread, I understand that some people
> don't 
realize that the Big Data SIG already tried very hard to push upstream
their work.

It's not a matter of trying, it's just upstream have different priorities and we
don't have enough weight to change these.
You can put 10x/20x/30x more people on Big Data packages, it won't
change the fundamental
issue that it's a neverending task, and our packages are not trusted.

---

The discussion is not about accepting or not divergence from Fedora Packaging
guidelines for Big Data packages. If we want to ship Big Data
components in Fedora
and play a role in this industry, upstream and the big data folks
leaves us no choice in
the first place.
Worse, remaining outsiders mean that we have even less weight to
influence these projects.

The discussion is:
1. Do we want Fedora be a key player in Big Data or not <= and I can
list many other
important fields other than Big Data that are in the same position.
2. How to make a place for "products" that otherwise do not fit within
the traditional
Fedora model without compromising with our Foundations.

First question is about our distribution strategy, the second one is
organizational.

Let me be honest, if we refuse to evolve, Fedora has no future in the
current ecosystem,
and we're leaving an open road to other distribution that are more
flexible than us which
ironically includes Debian or worse Ubuntu.


It's up to you folks, hate it or not, we don't make the rules.


Regards,
H.

[gil]

Il 04/08/2015 16:25, Haïkel ha scritto:
>  +1 Pete
>
>   From re-reading the whole thread, I understand that some people don't
>  realize that the Big Data SIG already tried very hard to push upstream
>  their work.
>
>  It's not a matter of trying, it's just upstream have different priorities and we
>  don't have enough weight to change these.
>  You can put 10x/20x/30x more people on Big Data packages, it won't
>  change the fundamental
>  issue that it's a neverending task, and our packages are not trusted.
>
>  ---
>
>  The discussion is not about accepting or not divergence from Fedora Packaging
>  guidelines for Big Data packages. If we want to ship Big Data
>  components in Fedora
>  and play a role in this industry, upstream and the big data folks
>  leaves us no choice in
>  the first place.
>  Worse, remaining outsiders mean that we have even less weight to
>  influence these projects.
>
>  The discussion is:
>  1. Do we want Fedora be a key player in Big Data or not <= and I can
>  list many other
>  important fields other than Big Data that are in the same position.
>  2. How to make a place for "products" that otherwise do not fit within
>  the traditional
>  Fedora model without compromising with our Foundations.
>
>  First question is about our distribution strategy, the second one is
>  organizational.
>
>  Let me be honest, if we refuse to evolve, Fedora has no future in the
>  current ecosystem,
>  and we're leaving an open road to other distribution that are more
>  flexible than us which
>  ironically includes Debian or worse Ubuntu. 
For me is a regression, considering
that
are the same old bad habits of those who develop these (c#r@a§p) softwares
"we" must not "necessarily" acquire even "us".
regards
sorry for the noise
gil
>  It's up to you folks, hate it or not, we don't make the rules.
>
>
>  Regards,
>  H.
>  _______________________________________________
>  bigdata mailing list
>  bigdata(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/bigdata

[Robert Rati]

On 08/04/2015 09:05 AM, Peter MacKinnon wrote:
>  The Fedora upstream feedback virtues you describe are laudable but
> the
>  game has changed. That model works well for packages that have both
>  limited scope and dependency sets. But these projects are massive, under
>  rapid development, and IMHO the upstreams aren't interested in a
>  continuous and steady stream of PR or JIRA to upgrade versions,
>  unbundle, etc. for the sake of Fedora. We already went down that road
>  with Hadoop for example and JIRAs just languished as we continually
>  rebased them for a time. 
I'll chime in with my support for this view, and I think the hadoop 
packaging effort has a prime example of this situation.  When we 
packaged hadoop we had to port the version of jetty used from 6->8 and 
then eventually to 9.  Changing minor jetty versions is non-trivial at 
times, and changing major versions takes quite a bit of work many times 
requiring diving into the implementations of api changes to make sure 
the new api did what the old one used to.  We carried this changes 
through multiple hadoop version updates as well.  Jetty is core to 
hadoop and the upstream project would add more jetty related code in 
each release which meant we had to do the same inspection process each 
time we wanted to update versions in Fedora.

We provided these patches to the upstream project, but they never went 
anywhere.  I even started a discussion on the hadoop mailing list about 
the situation, and the end result was that there was no way they were 
going to take the patch.  There were multiple valid reasons why, but at 
the core was that they had different priorities than Fedora.  The 
situation was made worse by an upstream branching/support/development 
strategy that really couldn't handle these kinds of changes easily.  In 
the case of jetty, any change/bugfix in a release version of hadoop on 
an older jetty would require the same kind of machinations we did in 
Fedora in order to port into any development stream with a newer jetty. 
  That is a lot of non-trivial work.  Newer jetty versions also needed a 
version of java hadoop didn't officially support, which is a non-starter.

I haven't been following the hadoop development mailing list as close as 
I used to, but last I saw they still didn't have a process to handle 
this kind of situation.  They recognized the problem and were trying to 
improve things (like moving to a newer version of java), but the core 
problem remained.

Rob

[Christopher]

On Tue, Aug 4, 2015 at 9:05 AM, Peter MacKinnon <pmackinn(a)redhat.com&gt; wrote:
>  On 8/3/15 12:38 PM, Christopher wrote:
> >
> > To some extent, Fedora has employed each of these strategies already:
> > repeated patching, convincing upstream, and supporting multiple
> > versions. If a particular package is problematic, or the maintainer
> > isn't up to the task to decide which combination of strategies is
> > best... well, that's where co-maintainers can be a really big help.
> > I'd only support dropping if maintainers can't be found who are
> > willing to support the packages to the extent they need. Isn't that
> > how things typically work in Fedora?
>
>
>  Will had an excellent and succinct rationalization for dropping Apache Spark
>  in the other related thread.
>
>  "I'm actually planning to retire Spark because the upstream has diverged too
>  far from what we can actually support in Fedora -- too many things depend on
>  bundled-but-modified libraries, nonstandard or older versions, etc., and
>  even in the 0.9.1 package we had to carry a lot of patches to make
>  everything work. I think that, given the developement direction the Spark
>  community has taken since Spark entered Fedora, the most sensible way to
>  have Spark integrated with Fedora going forward is in a Software Collection.
>  This is a bummer because it was a lot of work to get Spark in to Fedora, but
>  my concern is that having a (necessarily) limited Spark that is integrated
>  with Fedora is not going to be sufficiently compelling for end-users to
>  warrant not just using upstream binaries, and it's a lot of bandwidth for
>  volunteers to take on. I'm open to discussion on this matter, but I think
>  this is the best way forward."
>
>  The Fedora upstream feedback virtues you describe are laudable but the game
>  has changed. That model works well for packages that have both limited scope
>  and dependency sets. But these projects are massive, under rapid
>  development, and IMHO the upstreams aren't interested in a continuous and
>  steady stream of PR or JIRA to upgrade versions, unbundle, etc. for the sake
>  of Fedora. We already went down that road with Hadoop for example and JIRAs
>  just languished as we continually rebased them for a time. It wasn't for
>  want of effort downstream. Neither does that make the upstreams "bad" or
>  "evil".
>
>  My $0.02,
>  \Pete
>  
Perhaps it makes sense to drop Spark (I don't have a vested interest
in Spark), and some others. Perhaps it's even worth it to drop the
idea of a Big Data SIG (or mutate it to reflect the proposed focus on
Docker, or something else). However, I still have an interest in some
of the Big Data packages (ZooKeeper, Accumulo, Hadoop, etc.), and am
willing to do some effort to keep them alive as standard packages, and
I hope I can count on other Fedora packers to support this, as needed,
even if it is no longer the focus of the Big Data SIG.

[Gerald Henriksen]

On Fri, 31 Jul 2015 19:22:50 -0400, you wrote:

> On Fri, Jul 31, 2015 at 10:27 AM, Gerald Henriksen
> <ghenriks(a)gmail.com&gt; wrote:
> > So the discussion about Spark has revealed:
> >
> > 1) the Big Data community doesn't trust the Fedora packages
>
> From my perspective, I see a lot of things happening in the "Big Data
> community" which indicates not so much distrust for Fedora packages
> (or RHEL, for that matter), but more a complete lack of understanding
> about the benefits of packaging for these projects. 
As posted in the Spark thread, the feedback to Red Hat employee(s) is
specifically that the packages in Fedora are not trusted.

In may well be a lack of understanding, or it may be that they don't
like the changes required to package for a distro that has newer
libraries than they use, but the end result is the same.

> > 2) it seems a consensus that the realities of the Big Data
> ecosystem
> > don't work very well with making packages for Fedora
>
> I don't think that's necessarily true. I think a lot of the community
> is following the "Hadoop Way", where developers are just trying to
> mimic the way Hadoop does things, but Hadoop seems to be pretty poorly
> packaged and has had a lot of problems over time, from complete lack
> of dependency convergence, lack of packaging standards, improper use
> of the Linux filesystem hierarchy, confusing use of shell scripts and
> environment variables set all over the place, etc. 
No arguement, Hadoop and some of its ecosystem have what I would agree
are significant issues.

> A lot of this is where downstream packagers can bring good things
> back
> to the upstream communities. So, I don't think it's that the ecosystem
> doesn't work well with Fedora packaging... I think it's more that they
> haven't had sufficient feedback from those communities to push them to
> make their software more easily integrated. (Part of this is a Java
> thing more than a big data thing... let's face it... Java packaging
> hasn't really had a lot of good support until quite recently, with
> xmvn and related tools.) 
I have followed the Hadoop devel list for over 6 months now, and the
perhaps incorrect impression I have is that the Hadoop community is
largely being governed by feedback from places that are using Hadoop.
And those places seem to have made it clear while they want bug fixes
and new features, they want it done so in a way that creates the least
amount of disruption (and hence testing) required.

It will be very difficult for a downstream like Fedora to overcome
that sort of influence.

> > 3) result of 2, move towards newer (Docker) style of making Big
> Data
> > available on Fedora, perhaps with a side benefit of it then also being
> > possible to have it available on RHEL.
>
> Docker still has some limitations, and is not necessarily suitable to
> everybody's situation. I'd still prefer Fedora RPMs over Docker-based
> packaging. I think the community could benefit from both. 
I am only getting started with Docker, but I agree it is not an ideal
solution.  I too would prefer proper rpm packages in Fedora, but it
just doesn't seem to be feasible without significant compromises
leading me to think that Docker containers offer best of a bunch of
bad options.

> > Should we then also have a discussion, or just do it, of removing
> the
> > existing packages from Fedora 23?
>
> I'd be against that. It's one thing if there's no maintainer, but it's
> another to decide to oust a whole class of projects. 
I was thinking of a case where Fedora switched to providing Hadoop
using Docker containers, in which case also having older versions of
the software installable via rpm could cause issues.

Anyway, we are already going to lose Spark because it no longer builds
and I would guess it is only a matter of time before a library update
means the rest of the Big Data stack also fails to build.

[Denis Arnaud]

Reading the whole thread, I now get a better view of the pros and cons, and of the
suggested solutions (or, rather, work-arounds).

You surely know that Cloudera ships their own Hadoop distribution, which is based on a
RedHat/CentOS Linux distribution(!), and which is made of exactly the same components that
we intend to ship with Fedora, including Apache Spark.
So, why couldn't we just use the (surely hard) work Cloudera is doing to build their
distribution?
I have not looked at the details, but from what I understand, their distribution of Hadoop
(and Spark, and Impala) relies only on open source components. They have proprietary
software, for instance the Cloudera Manager (used to configure, deploy and administer a
cluster; but it can be replaced by Apache Ambary, which is open source).

Note that there are other Hadoop distributions, such as Hortonworks (I believe those
latter even ship only 100% open source components).
I do not mention MapR, as their stack is mainly based on proprietary software.

Joining our forces with Cloudera (and/or Hortonworks) would help all of us in front of the
upstream developers. For sure, Cloudera and Hortonworks face the same issues as us when
packaging the components and when dealing with the Hadoop upstream eco-system.

Just food for thoughts.

Kind regards

Denis

[Denis Arnaud]

Still another option would be to build a Big Data Spin.
That corresponds to the SCL (Software Collection) suggestion, though more complete.

The Big Data Spin would allow to deploy stand-alone as well as full clusters.
Note that the Big Data Spin would be very similar to the classical Hadoop distributions
(Cloudera, Hortonworks).
Hence, by the way, I am not sure Fedora would add much value here... I mean, why a user
would use a Fedora Big Data Spin rather than, say a CDH (Cloudera Hadoop) image?

So, not so sure it is a good idea, then...
----

Properly packaging Big Data software is something upstream developers should want more, as
not doing so costs a lot in maintenance. And I am sure that upstream (most, like
Databricks, Cloudera, or even Data Artisans, are commercial companies, which care about
development costs) would rather develop new features than maintain patches on old versions
of their bundled libraries.

By the way, that practice in Apache Spark causes headaches to a lot of users (search, for
instance, for "Spark NoSuchFieldError"), even experienced ones, as runtime
errors pop up out of nowhere, and debugging them is quite difficult. The dependency graph
of a typical Spark-based application lists dozens of libraries with not always compatible
versions, most being duplicated, part of them being bundled and patched.
Fixing that kind of headache should not be the work of a user. To my belief, that is
exactly what Linux distributions are done for: ship an ecosystem of components, for which
the versions are known to work well together.

[Gerald Henriksen]

On Mon, 11 Jan 2016 18:13:53 -0000, you wrote:

> Properly packaging Big Data software is something upstream developers
> should want more, as not doing so costs a lot in maintenance. And I am sure that upstream
> (most, like Databricks, Cloudera, or even Data Artisans, are commercial companies, which
> care about development costs) would rather develop new features than maintain patches on
> old versions of their bundled libraries. 
The problem is that there is also a development cost to upgrading to
newer libraries, both in the work to update the code base and then the
resulting testing.

Despite efforts by Fedora packagers (as well as others on occasion on
the Hadoop mailing lists) the upstream developers have no desire to
spend the time and effort to update libraries until they are forced
to.

For that matter, the only fully approved version of Java for Hadoop is
the official Oracle Java 7, which is now 9 month beyond any publicly
available updates for security issues.  They do not support OpenJDK or
Java 8.

In short, as previously discussed, Hadoop is like so much of the
software developed in the last decade where the need to constantly
update with the language and/or libraries is viewed as a bug and
alternative methods have been developed to allow the project to exist
in its own little bubble - in the case of Hadoop its Maven but even
the recently open sourced Swift is developing its own package system
just like Python, Perl, Go, etc.

As long as they can stay in that bubble there is little to no
development cost to not updating the code base.

As long as that continues Hadoop is impossible to package properly in
rpm form for Fedora.

[obviously one can wrap up the jar files into an rpm, as some
commercial Hadoop companies do.  But they could never be in Fedora
proper, and if you are going to go through the hassle to wrap a jar
into an rpm you may as well move with the present and do a Docker
container instead]

> By the way, that practice in Apache Spark causes headaches to a lot of
> users (search, for instance, for "Spark NoSuchFieldError"), even experienced
> ones, as runtime errors pop up out of nowhere, and debugging them is quite difficult. The
> dependency graph of a typical Spark-based application lists dozens of libraries with not
> always compatible versions, most being duplicated, part of them being bundled and patched.
I have no experience with Spark, so I cannot comment on your example
other than to say fixing that sort of issue relies on buy in from
upstream to keep updating the code base, and upstream may not be
willing.  The secondary issue is that even if the Spark (or any other
Hadoop related project) developers want to do things correctly they
are still stuck with the decisions made by Hadoop.

> Fixing that kind of headache should not be the work of a user. To my
> belief, that is exactly what Linux distributions are done for: ship an ecosystem of
> components, for which the versions are known to work well together. 
Two things:

1) the distributions simpy don't have the available manpower to do it.
At this point there are so many different projects with different
library needs it would take years by a lot of programmers/packagers
just to get everybody up do date, and with too many upstreams
unwilling to update their codebases with the changes it would remain
an ongoing problem.

2) the users don't care - as posted by one of the Red Hat(?) members
of the list the users don't trust a properly packaged Hadoop.

Part of the problem is the Java is too much of a nightmare to package,
it results in far too many rpm files that to a user it looks too
complicated compared to downloading 1 jar file.

But in the end it doesn't matter, as no user support means it is
difficult to justify the hours of work required.

[Mohd Izhar Firdaus Ismail]

adding a bit .. been using Hortonworks Hadoop for a while now .. yes they
bundle everything into /usr/hadoop and the packages are not compatible with
fedora packaging guideline .. but rather packaged in a way that Ambari can
handle them easily ..

On Tue, Jan 12, 2016 at 7:50 AM, Gerald Henriksen <ghenriks(a)gmail.com&gt;
wrote:

>  On Mon, 11 Jan 2016 18:13:53 -0000, you wrote:
>
>  >Properly packaging Big Data software is something upstream developers
>  should want more, as not doing so costs a lot in maintenance. And I am sure
>  that upstream (most, like Databricks, Cloudera, or even Data Artisans, are
>  commercial companies, which care about development costs) would rather
>  develop new features than maintain patches on old versions of their bundled
>  libraries.
>
>  The problem is that there is also a development cost to upgrading to
>  newer libraries, both in the work to update the code base and then the
>  resulting testing.
>
>  Despite efforts by Fedora packagers (as well as others on occasion on
>  the Hadoop mailing lists) the upstream developers have no desire to
>  spend the time and effort to update libraries until they are forced
>  to.
>
>  For that matter, the only fully approved version of Java for Hadoop is
>  the official Oracle Java 7, which is now 9 month beyond any publicly
>  available updates for security issues.  They do not support OpenJDK or
>  Java 8.
>
>  In short, as previously discussed, Hadoop is like so much of the
>  software developed in the last decade where the need to constantly
>  update with the language and/or libraries is viewed as a bug and
>  alternative methods have been developed to allow the project to exist
>  in its own little bubble - in the case of Hadoop its Maven but even
>  the recently open sourced Swift is developing its own package system
>  just like Python, Perl, Go, etc.
>
>  As long as they can stay in that bubble there is little to no
>  development cost to not updating the code base.
>
>  As long as that continues Hadoop is impossible to package properly in
>  rpm form for Fedora.
>
>  [obviously one can wrap up the jar files into an rpm, as some
>  commercial Hadoop companies do.  But they could never be in Fedora
>  proper, and if you are going to go through the hassle to wrap a jar
>  into an rpm you may as well move with the present and do a Docker
>  container instead]
>
>  >By the way, that practice in Apache Spark causes headaches to a lot of
>  users (search, for instance, for "Spark NoSuchFieldError"), even
>  experienced ones, as runtime errors pop up out of nowhere, and debugging
>  them is quite difficult. The dependency graph of a typical Spark-based
>  application lists dozens of libraries with not always compatible versions,
>  most being duplicated, part of them being bundled and patched.
>
>  I have no experience with Spark, so I cannot comment on your example
>  other than to say fixing that sort of issue relies on buy in from
>  upstream to keep updating the code base, and upstream may not be
>  willing.  The secondary issue is that even if the Spark (or any other
>  Hadoop related project) developers want to do things correctly they
>  are still stuck with the decisions made by Hadoop.
>
>  >Fixing that kind of headache should not be the work of a user. To my
>  belief, that is exactly what Linux distributions are done for: ship an
>  ecosystem of components, for which the versions are known to work well
>  together.
>
>  Two things:
>
>  1) the distributions simpy don't have the available manpower to do it.
>  At this point there are so many different projects with different
>  library needs it would take years by a lot of programmers/packagers
>  just to get everybody up do date, and with too many upstreams
>  unwilling to update their codebases with the changes it would remain
>  an ongoing problem.
>
>  2) the users don't care - as posted by one of the Red Hat(?) members
>  of the list the users don't trust a properly packaged Hadoop.
>
>  Part of the problem is the Java is too much of a nightmare to package,
>  it results in far too many rpm files that to a user it looks too
>  complicated compared to downloading 1 jar file.
>
>  But in the end it doesn't matter, as no user support means it is
>  difficult to justify the hours of work required.
>  _______________________________________________
>  bigdata mailing list
>  bigdata(a)lists.fedoraproject.org
>  http://lists.fedoraproject.org/admin/lists/[e-mail]
>  


-- 
Mohd Izhar Firdaus Bin Ismail / KageSenshi
Fedora Malaysia Contributor & Ambassador
http://blog.kagesenshi.org
92C2 B295 B40B B3DC 6866  5011 5BD2 584A 8A5D 7331

[Mohd Izhar Firdaus Ismail]

oops .. */usr/hadoop -> /usr/hdp

[Denis Arnaud]

2016-01-12 1:50 GMT+02:00 Gerald Henriksen <ghenriks(a)gmail.com&gt;:

>  On Mon, 11 Jan 2016 18:13:53 -0000, you wrote:
>
>  >Properly packaging Big Data software is something upstream developers
>  should want more, as not doing so costs a lot in maintenance. And I am sure
>  that upstream (most, like Databricks, Cloudera, or even Data Artisans, are
>  commercial companies, which care about development costs) would rather
>  develop new features than maintain patches on old versions of their bundled
>  libraries.
>
>  The problem is that there is also a development cost to upgrading to
>  newer libraries, both in the work to update the code base and then the
>  resulting testing.
>
>  Despite efforts by Fedora packagers (as well as others on occasion on
>  the Hadoop mailing lists) the upstream developers have no desire to
>  spend the time and effort to update libraries until they are forced
>  to.
>
>  For that matter, the only fully approved version of Java for Hadoop is
>  the official Oracle Java 7, which is now 9 month beyond any publicly
>  available updates for security issues.  They do not support OpenJDK or
>  Java 8.
>
>  In short, as previously discussed, Hadoop is like so much of the
>  software developed in the last decade where the need to constantly
>  update with the language and/or libraries is viewed as a bug and
>  alternative methods have been developed to allow the project to exist
>  in its own little bubble - in the case of Hadoop its Maven but even
>  the recently open sourced Swift is developing its own package system
>  just like Python, Perl, Go, etc.
>
>  As long as they can stay in that bubble there is little to no
>  development cost to not updating the code base.
>
>  As long as that continues Hadoop is impossible to package properly in
>  rpm form for Fedora.
>
>  [obviously one can wrap up the jar files into an rpm, as some
>  commercial Hadoop companies do.  But they could never be in Fedora
>  proper, and if you are going to go through the hassle to wrap a jar
>  into an rpm you may as well move with the present and do a Docker
>  container instead]
>  
Ok, thanks. All what you say is clear and makes sense, unfortunately... So,
a is a little despairing: we should wait until the big data development
ecosystem implodes under its own complexity, and then we hope that the big
data developer community will be wise enough to pick the best practices,
which have allowed the Linux distributions to succeed so far.

> By the way, that practice in Apache Spark causes headaches to a lot
> of
>  users (search, for instance, for "Spark NoSuchFieldError"), even
>  experienced ones, as runtime errors pop up out of nowhere, and debugging
>  them is quite difficult. The dependency graph of a typical Spark-based
>  application lists dozens of libraries with not always compatible versions,
>  most being duplicated, part of them being bundled and patched.
>
>  I have no experience with Spark, so I cannot comment on your example
>  other than to say fixing that sort of issue relies on buy in from
>  upstream to keep updating the code base, and upstream may not be
>  willing.  The secondary issue is that even if the Spark (or any other
>  Hadoop related project) developers want to do things correctly they
>  are still stuck with the decisions made by Hadoop.
>  
Yes, you are right.



>  >Fixing that kind of headache should not be the work of a user. To
> my
>  belief, that is exactly what Linux distributions are done for: ship an
>  ecosystem of components, for which the versions are known to work well
>  together.
>
>  Two things:
>
>  1) the distributions simply don't have the available manpower to do it.
>  At this point there are so many different projects with different
>  library needs it would take years by a lot of programmers/packagers
>  just to get everybody up do date, and with too many upstreams
>  unwilling to update their codebases with the changes it would remain
>  an ongoing problem.
>  
The Linux distribution packagers could all unite. But given the above (and
below), it is worthless until the upstream developer community drastically
change their perception about the value of packaging.


2) the users don't care - as posted by one of the Red Hat(?) members
>  of the list the users don't trust a properly packaged Hadoop.
Part of the problem is the Java is too much of a nightmare to package,
>  it results in far too many rpm files that to a user it looks too
>  complicated compared to downloading 1 jar file.
>
>  But in the end it doesn't matter, as no user support means it is
>  difficult to justify the hours of work required.
>  
Ok. I am of course biased as a user (and was inclined to believe that more
users would be like me (willing clean big data packages within a standard
Linux distribution)) :)

Thanks again for the clarification.


So, back to the initial question: what do we do? Should we drop Hadoop
altogether?

I would favor to continue trying to keep a working version of Hadoop, as
up-to-date as possible, on Fedora (and hopefully on CentOS). It would be a
shame to throw away the man.years of investment/work done by the Fedora
packagers already.

[By the way, thanks to gil, Hadoop builds again on Rawhide (I still have to
apply his patch, though, so that it finds way into the Git repository]

Kind regards

-da

[Peter MacKinnon]

On 1/10/16 6:16 PM, Denis Arnaud wrote:
>  Reading the whole thread, I now get a better view of the pros and
> cons, and of the suggested solutions (or, rather, work-arounds).
>
>  You surely know that Cloudera ships their own Hadoop distribution, which is based on a
> RedHat/CentOS Linux distribution(!), and which is made of exactly the same components that
> we intend to ship with Fedora, including Apache Spark.
>  So, why couldn't we just use the (surely hard) work Cloudera is doing to build their
> distribution?
>  I have not looked at the details, but from what I understand, their distribution of
> Hadoop (and Spark, and Impala) relies only on open source components. They have
> proprietary software, for instance the Cloudera Manager (used to configure, deploy and
> administer a cluster; but it can be replaced by Apache Ambary, which is open source).
Pretty sure they assemble their RPMs directly from the jar artifacts 
directly derived from a source+patches maven build, without mapping on 
to officially packaged "system" jars as we have done in the past for 
Hadoop and other related Big Data projects in Fedora. Those RPMs violate 
existing Fedora packaging rules. That is an essential part of this debate.

>  Note that there are other Hadoop distributions, such as Hortonworks (I believe those
> latter even ship only 100% open source components).
>  I do not mention MapR, as their stack is mainly based on proprietary software.
>
>  Joining our forces with Cloudera (and/or Hortonworks) would help all of us in front of
> the upstream developers. For sure, Cloudera and Hortonworks face the same issues as us
> when packaging the components and when dealing with the Hadoop upstream eco-system.
>
>  Just food for thoughts.
>
>  Kind regards
>
>  Denis
>  _______________________________________________
>  bigdata mailing list
>  bigdata(a)lists.fedoraproject.org
>  http://lists.fedoraproject.org/admin/lists/[e-mail]