Re: Packaging Hadoop in the Fedora community

[Tim St. Clair]

Bruno -

Feel free to join the dev-lists and IRC channels(fedora-devel, fedora-bigdata), as there are numerous items we are still hashing out here (https://fedoraproject.org/wiki/Features/Hadoop), and we would be happy to coordinate.  For example, getting parity across the tests is a stubborn foe.  Once Hadoop we've tackled the main package we can eval and coordinate other ecosystem packages.

Cheers,
Tim

----- Original Message -----
> From: "Bruno Mahé" 
> To: "Matthew Farrellee" 
> Cc: [e-mail], [e-mail]
> Sent: Wednesday, June 12, 2013 1:56:57 AM
> Subject: Re: Packaging Hadoop in the Fedora community
>
> +Tim
>
> Hi Matthew and Tim,
>
> I noticed Tim's email on the Apache Bigtop/Hadoop mailing list
>   and was wondering how we could all collaborate and work together to
> improve the state of Apache Hadoop (and its ecosystem) on
> Fedora/CentOS/RHEL.
>
> Thanks,
> Bruno
>
> On 04/26/2013 10:38 AM, Bruno Mahé wrote:
> > On 04/25/2013 06:49 AM, Matthew Farrellee wrote:
> > > On 04/25/2013 01:55 AM, Bruno Mahé wrote:
> > > > On 04/24/2013 05:40 AM, Matthew Farrellee wrote:
> > > > > Hello Big Data SIG!
> > > > >
> > > > > A few of us in the Fedora community have been working to package Apache
> > > > > Hadoop for Fedora. The feature page is,
> > > > >
> > > > > https://fedoraproject.org/wiki/Features/Hadoop
> > > > >
> > > > > You'll find what we plan to package, our progress working through all
> > > > > the dependencies, a process for how we're attacking dependencies and a
> > > > > pre-Fedora repo of the packages in progress.
> > > > >
> > > > > It's a big effort, but will form a foundation for packaging more Hadoop
> > > > > ecosystem packages for Fedora.
> > > > >
> > > > > We're at a point where we need a better place to engage the community
> > > > > and the Big Data SIG is a natural choice. I hope you will welcome us to
> > > > > the list and IRC, and maybe lend a hand!
> > > > > _______________________________________________
> > > > > bigdata mailing list
> > > > > [e-mail]
> > > > > https://admin.fedoraproject.org/mailman/listinfo/bigdata
> > > >
> > > > Hi,
> > > >
> > > > I am not CC-ing the Bigdata SIG as to not spam the list.
> > > >
> > > > One of the project I work on intersects closely with this SIG. This
> > > > project is Apache Bigtop (http://bigtop.apache.org/).
> > > > The goal of Apache Bigtop is three folds:
> > > > 1/ Provide top notch packages for Apache Hadoop related projects
> > > > 2/ Provide a point of integration and testing for all these projects
> > > > 3/ Provide means to reliably deploy a complete stack.
> > > >
> > > > So there is quite a bit of overlapping between Apache Bigtop, this SIG
> > > > and your efforts.
> > > >
> > > > Would you see any way we could collaborate?
> > > >
> > > > Please, see this previous message for an explanation about Apache Bigtop
> > > > with regards to this SIG and Fedora:
> > > > http://lists.fedoraproject.org/pipermail/bigdata/2013-March/000012.html
> > > >
> > > > Thanks,
> > > > Bruno
> > >
> > > Definitely. I hold an assumption that Bigtop is a good place to -
> > >
> > > 0) drive consistency across the Hadoop ecosystem
> > >   1) provide deliverables that are integrated into various OS platforms
> > >
> > > For (0), our packaging effort is finding old and inconsistent (different
> > > versions of the same) dependencies across projects.
> > >
> > > For (1), at a minimum I assume that the packaging work done for Fedora
> > > can be contributed to Bigtop. This would provide integration with Fedora
> > > at a level that does not currently exist.
> > >
> > > Beyond that. I'm interested in the testing framework that Bigtop
> > > provides. Immediately, for the Fedora packaging effort, we need a way to
> > > make sure the packages we provide aren't breaking core functionality. I
> > > don't think TestDFSIO and some GridMix2 will be sufficient.
> > >
> > > Best,
> > >
> > > matt
> >
> > Hi,
> >
> > For (0), we are already working on this. We also spend a lot of efforts
> > helping verifying that release candidates do not break their downstream
> > projects. So the more people helping on that front, the merrier!
> >
> > For (1), we would definitely gladly accepts all contributions! And we
> > would look at the best way to integrate your work.
> > The sticky point may be related to Fedora specific macros. Our spec file
> > works across a bunch of RPM distributions, so Fedora specific macros
> > would have to be inside some ifdef or encapsulated or any other solution
> > (even using Apache Bigtop as a common effort and forking the specs to
> > add the Fedora specific macros). But in any case that would be
> > implementation details and we can look at it whenever you feel to
> > contribute your work.
> >
> > Apache Bigtop has some smoke tests, package tests (verify they can be
> > installed, removed, upgraded, permissions are rights...) as well as
> > integration tests. So I believe that could be leveraged by you whether
> > or not you decide to put in common the efforts to work on packaging or
> > not. If you have any issue using the tests, feel free to ping me or to
> > send an email to the mailing list.
> >
> > Thanks,
> > Bruno