Packaging Hadoop in the Fedora community

[Tim St Clair]

Bruno - 

Feel free to join the dev-lists and IRC channels(fedora-devel, fedora-bigdata), as there are numerous items we are still hashing out here (https://fedoraproject.org/wiki/Features/Hadoop), and we would be happy to coordinate.  For example, getting parity across the tests is a stubborn foe.  Once Hadoop we've tackled the main package we can eval and coordinate other ecosystem packages.  

Cheers,
Tim

----- Original Message -----
> From: "Bruno Mahé" <bruno at bmahe.net>
> To: "Matthew Farrellee" <matt at redhat.com>
> Cc: puntogil at libero.it, tstclair at redhat.com
> Sent: Wednesday, June 12, 2013 1:56:57 AM
> Subject: Re: Packaging Hadoop in the Fedora community
> 
> +Tim
> 
> Hi Matthew and Tim,
> 
> I noticed Tim's email on the Apache Bigtop/Hadoop mailing list
>   and was wondering how we could all collaborate and work together to
> improve the state of Apache Hadoop (and its ecosystem) on
> Fedora/CentOS/RHEL.
> 
> Thanks,
> Bruno
> 
> On 04/26/2013 10:38 AM, Bruno Mahé wrote:
> > On 04/25/2013 06:49 AM, Matthew Farrellee wrote:
> >> On 04/25/2013 01:55 AM, Bruno Mahé wrote:
> >>> On 04/24/2013 05:40 AM, Matthew Farrellee wrote:
> >>>> Hello Big Data SIG!
> >>>>
> >>>> A few of us in the Fedora community have been working to package Apache
> >>>> Hadoop for Fedora. The feature page is,
> >>>>
> >>>>     https://fedoraproject.org/wiki/Features/Hadoop
> >>>>
> >>>> You'll find what we plan to package, our progress working through all
> >>>> the dependencies, a process for how we're attacking dependencies and a
> >>>> pre-Fedora repo of the packages in progress.
> >>>>
> >>>> It's a big effort, but will form a foundation for packaging more Hadoop
> >>>> ecosystem packages for Fedora.
> >>>>
> >>>> We're at a point where we need a better place to engage the community
> >>>> and the Big Data SIG is a natural choice. I hope you will welcome us to
> >>>> the list and IRC, and maybe lend a hand!
> >>>> _______________________________________________
> >>>> bigdata mailing list
> >>>> bigdata at lists.fedoraproject.org
> >>>> https://admin.fedoraproject.org/mailman/listinfo/bigdata
> >>>
> >>>
> >>> Hi,
> >>>
> >>> I am not CC-ing the Bigdata SIG as to not spam the list.
> >>>
> >>> One of the project I work on intersects closely with this SIG. This
> >>> project is Apache Bigtop (http://bigtop.apache.org/).
> >>> The goal of Apache Bigtop is three folds:
> >>> 1/ Provide top notch packages for Apache Hadoop related projects
> >>> 2/ Provide a point of integration and testing for all these projects
> >>> 3/ Provide means to reliably deploy a complete stack.
> >>>
> >>> So there is quite a bit of overlapping between Apache Bigtop, this SIG
> >>> and your efforts.
> >>>
> >>> Would you see any way we could collaborate?
> >>>
> >>>
> >>> Please, see this previous message for an explanation about Apache Bigtop
> >>> with regards to this SIG and Fedora:
> >>> http://lists.fedoraproject.org/pipermail/bigdata/2013-March/000012.html
> >>>
> >>>
> >>> Thanks,
> >>> Bruno
> >>
> >> Definitely. I hold an assumption that Bigtop is a good place to -
> >>
> >>   0) drive consistency across the Hadoop ecosystem
> >>   1) provide deliverables that are integrated into various OS platforms
> >>
> >> For (0), our packaging effort is finding old and inconsistent (different
> >> versions of the same) dependencies across projects.
> >>
> >> For (1), at a minimum I assume that the packaging work done for Fedora
> >> can be contributed to Bigtop. This would provide integration with Fedora
> >> at a level that does not currently exist.
> >>
> >> Beyond that. I'm interested in the testing framework that Bigtop
> >> provides. Immediately, for the Fedora packaging effort, we need a way to
> >> make sure the packages we provide aren't breaking core functionality. I
> >> don't think TestDFSIO and some GridMix2 will be sufficient.
> >>
> >> Best,
> >>
> >>
> >> matt
> >>
> >
> > Hi,
> >
> > For (0), we are already working on this. We also spend a lot of efforts
> > helping verifying that release candidates do not break their downstream
> > projects. So the more people helping on that front, the merrier!
> >
> > For (1), we would definitely gladly accepts all contributions! And we
> > would look at the best way to integrate your work.
> > The sticky point may be related to Fedora specific macros. Our spec file
> > works across a bunch of RPM distributions, so Fedora specific macros
> > would have to be inside some ifdef or encapsulated or any other solution
> > (even using Apache Bigtop as a common effort and forking the specs to
> > add the Fedora specific macros). But in any case that would be
> > implementation details and we can look at it whenever you feel to
> > contribute your work.
> >
> > Apache Bigtop has some smoke tests, package tests (verify they can be
> > installed, removed, upgraded, permissions are rights...) as well as
> > integration tests. So I believe that could be leveraged by you whether
> > or not you decide to put in common the efforts to work on packaging or
> > not. If you have any issue using the tests, feel free to ping me or to
> > send an email to the mailing list.
> >
> >
> >
> > Thanks,
> > Bruno
> 
>