bigdata July 2013

bigdata@lists.fedoraproject.org

8 participants
16 discussions

xmlenc on Fedora 20
by Robert Rati 16 Jul '13

16 Jul '13

The xmlenc package in rawhide/F20 will no longer resolve the xmlenc:xmlenc group/artifact id. The version of xmlenc in rawhide is the same as in F19, but has been converted to use the xmvn macros and is missing the mapping for xmlenc:xmlenc. The F20 package only has a mapping for org.znerd:xmlenc. Is this an intentional change? I believe a mvn_alias line like the following would provide the same linkage: %mvn_alias "org.znerd:xmlenc" "xmlenc:xmlenc? Rob

1 0

Fwd: protobuf 2.5.0 failure should be known by wiki
by Tim St. Clair 10 Jul '13

10 Jul '13

FYI, has anyone seen *this on F19 yet, I saw 2.5.X in Koji :-/ Cheers, Tim ----- Forwarded Message ----- > From: "Akira AJISAKA" <ajisakaa(a)oss.nttdata.co.jp> > To: common-dev(a)hadoop.apache.org > Sent: Tuesday, July 9, 2013 7:56:45 PM > Subject: protobuf 2.5.0 failure should be known by wiki > > Hi, > > I've installed ProtocolBuffer 2.5.0 according to [[wiki: > HowToContribute]]. And that's why I failed to build hadoop and I had to > downgrade protobuf to 2.4.1. > > Now I know HADOOP-9346 and HADOOP-9440 for enabling protobuf 2.5.0, but > these issues have been left for months. So, it should be commented to > wiki that protobuf 2.5.0 fails to build and we recommend using protobuf > 2.4.1. > > I tried to edit wiki but it seems that I don't have the permission. > How can I edit? > > - Akira >

2 1

Re: Bringing Hadoop to Fedora
by Tim St. Clair 09 Jul '13

09 Jul '13

Thanks Alejandro. comments inline below ----- Original Message ----- > From: "Alejandro Abdelnur" <tucu(a)cloudera.com> > To: common-dev(a)hadoop.apache.org > Sent: Monday, July 8, 2013 4:06:37 PM > Subject: Re: Bringing Hadoop to Fedora > > [moving bigtop to bcc] > > Tim, > > Except from HADOOP-9680 which has significant code changes and some false > changes (which I did not go thru), all other changes seem OK. > > * Have you had a change to run ALL Hadoop testcases with them applied to > make sure there are not regression? > Yes for everything, except (https://issues.apache.org/jira/browse/HADOOP-9680 / 9623). > * Have you look at the output of running 'mvn dependency:tree' without/with > the patches to verify unwanted dependencies are not sneaking in? > We've been tracking the dep-graph with a pretty close eye, but it would always be good to have another set of eyes gander at the delta. > On HADOOP-9650 (not included in the above list): Done on purpose at this point because the shift in Fedora 19 was just recently absorbed, and we are running through tests and use cases to verify we didn't break the world ;-). > > Moving from Jetty6 to Jetty9 should be OK if there are not regressions. I > was chatting with ToddL about the issues we had before in and according to > Jetty guys they've been fixed in newer Jetty versions. Also, we are not > using Jetty for shuffle anymore (we use Netty), so Jetty is not 'stress' > that much anymore as it is used for the web UIs and for NN-SNN > checkpointing. > > From the patch some code changes are required, once that is take care we > should repeat the bullet item above for this patch. > > In addition, we have to see how this will play with other projects like > HBase that are using HttpServer from hadoop-common. I'll forward this part > of the email to their dev@ so they can watch/jump-in if necessary in the > JIRA. Agreed, which is part of the reason why we wanted to include BIGTOP on this thread, as they would likely know more about the stack dependencies then we would. > > Thanks. > > > On Mon, Jul 8, 2013 at 11:28 AM, Tim St Clair <tstclair(a)redhat.com> wrote: > > > Arun, > > > > I realize you're probably pretty busy (or on vacation), but I figured I > > would re-ping this thread to inquire about the status of the patch set > > listed below.?.? > > > > https://issues.apache.org/jira/browse/HADOOP-9594 > > https://issues.apache.org/jira/browse/HADOOP-9610 > > https://issues.apache.org/jira/browse/HADOOP-9611 > > https://issues.apache.org/jira/browse/HADOOP-9613 > > https://issues.apache.org/jira/browse/HADOOP-9680 / 9623 > > > > Cheers, > > Tim > > > > ----- Original Message ----- > > > From: "Arun C Murthy" <acm(a)hortonworks.com> > > > To: yarn-dev(a)hadoop.apache.org > > > Cc: common-dev(a)hadoop.apache.org, mapreduce-dev(a)hadoop.apache.org, > > hdfs-dev(a)hadoop.apache.org, dev(a)bigtop.apache.org > > > Sent: Tuesday, June 11, 2013 12:54:11 PM > > > Subject: Re: Bringing Hadoop to Fedora > > > > > > Tim, > > > > > > This is great! I'll take a look at some of these patches, welcome! > > > > > > Arun > > > > > > On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote: > > > > > > > Greetings Hadoop Development Community, > > > > > > > > As some of you are aware, we've been looking to bring the upstream 2.X > > > > series into the Fedora channels. We believe that there are several > > > > benefits that this can bring to the community: > > > > > > > > - Updated & streamlined rpm packaging > > > > - Deeper level of system integration > > > > - System managed dependencies (security + defect tracking) > > > > - Proving ground for OpenJDK7 > > > > - Greater exposure for the community (both adoption and defect > > tracking) > > > > ... > > > > > > > > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop > > ), > > > > we've come across some build issues as they relate to Fedora packaging > > > > guidelines (http://fedoraproject.org/wiki/Packaging:Java) Most of > > these > > > > issues (listed below) are very minor dependency changes, and we were > > > > wondering if there are folks who would be willing to review. > > > > > > > > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594 > > > > fedora-patch-collections: > > https://issues.apache.org/jira/browse/HADOOP-9610 > > > > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611 > > > > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613 > > > > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623 > > > > (uber deprecated) > > > > > > > > There are other patches we have in the queue but they still need to be > > > > flushed out, and in an effort to follow the KISS principles we figure > > this > > > > would be a good start. > > > > > > > > Cheers, > > > > Tim > > > > > > -- > > > Arun C. Murthy > > > Hortonworks Inc. > > > http://hortonworks.com/ > > > > > > > > > > > > > > > -- > Alejandro >

1 0

Moving from Features/Hadoop to Changes/Hadoop
by Matthew Farrellee 08 Jul '13

08 Jul '13

I'm about to move the Features/Hadoop page to Changes/Hadoop. I'll do two passes and send out a note when I'm done. The first pass should catch the bulk of the content. The second will catch anything that was updated during the first pass (I know Pete is actively editing) and change Features/Hadoop to point to Changes/Hadoop. If anyone else is actively editing the Features/Hadoop page let me know on irc (#fedora-bigdata). Best, matt

1 1

Rebase to Hadoop 2.0.5 alpha
by Peter MacKinnon 08 Jul '13

08 Jul '13

FYI, We have completed rebasing the Fedora Hadoop branches (patches, integration, test) from 2.0.2-alpha to 2.0-5-alpha. This is of course the most recent tarball release from Apache upstream. The git branches can be examined at https://github.com/fedora-bigdata/hadoop-common/tree/fedora-* and the feature page references have also been updated at https://fedoraproject.org/wiki/Features/Hadoop. Nothing major to report at this time. I will be doing another sweep of the source test suite to update the table at https://fedoraproject.org/wiki/Features/Hadoop#Test_Suite to identify or remove test changes. Many of the test-based issues in 2.0.2a (e.g., GuiceServletConfig instantiation) are still in play. \Pete -- Peter MacKinnon MRG Grid/Big Data Red Hat Inc. Raleigh, NC

3 3

Fwd: [jira] [Commented] (HADOOP-9680) Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials
by Tim St. Clair 01 Jul '13

01 Jul '13

----- Forwarded Message ----- > From: "Robert Gibbon (JIRA)" <jira(a)apache.org> > To: tstclair(a)redhat.com > Sent: Monday, July 1, 2013 12:22:20 PM > Subject: [jira] [Commented] (HADOOP-9680) Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security > Credentials > > > [ > https://issues.apache.org/jira/browse/HADOOP-9680?page=com.atlassian.jira.p… > ] > > Robert Gibbon commented on HADOOP-9680: > --------------------------------------- > > I took a look at your patch in HADOOP-9623. Some comments: > > * Bucket keyspace listings running over a s3-native fs will be broken by your > patch, they make use of the method > org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(String > key) when querying S3 for the given URI; if the URI does not correspond to a > single key in a bucket (ie a single object) an exception is thrown. In the > above mentioned method, the exception's Message property was being parsed > for the string "ResponseCode=404" to interpret that the URI is not a single > key. If the condition is met, it returns null. Its a horrible piece of code > and a very poorly defined contract with the calling party. It is also broken > by jets3t 0.9.0, which doesn't pass back that message anymore in that > situation. I adapted it to look at the ResponseCode property for the integer > 404 instead, but someone who knows that code better than me would do a good > deed to fix it more sustainably. > > * I needed to upgrade jets3t to 0.9.0 because I need support for AWS IAM > federated access tokens (temporary, time limited access credentials, tied to > a session ticket). I don't see any support for that in the patch in > HADOOP-9623, for me its of no value unless it supports temp security tokens. > > I think we're aligned on the need for an uprev of the jets3t implementation > in hadoop. > HTH > > > Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security > > Credentials > > ------------------------------------------------------------------------------ > > > > Key: HADOOP-9680 > > URL: https://issues.apache.org/jira/browse/HADOOP-9680 > > Project: Hadoop Common > > Issue Type: Improvement > > Components: fs/s3 > > Reporter: Robert Gibbon > > Priority: Minor > > Attachments: s3fs-temp-iam-creds.diff.patch > > > > > > Here is a patch in unified diff format to enable Amazon Web Services IAM > > Temporary Security Credentials secured interactions with S3 from Hadoop. > > It bumps the JetS3t release version up to 0.9.0. > > To use a temporary security credential set, you need to provide the > > following properties, depending on the implementation (s3 or s3native): > > fs.s3.awsAccessKeyId or fs.s3n.awsAccessKeyId - the temporary access key id > > issued by AWS IAM > > fs.s3.awsSecretAccessKey or fs.s3n.awsSecretAccessKey - the temporary > > secret access key issued by AWS IAM > > fs.s3.awsSessionToken or fs.s3n.awsSessionToken - the session ticket issued > > by AWS IAM along with the temporary key > > fs.s3.awsTokenFriendlyName or fs.s3n.awsTokenFriendlyName - any string > > -- > This message is automatically generated by JIRA. > If you think it was sent incorrectly, please contact your JIRA administrators > For more information on JIRA, see: http://www.atlassian.com/software/jira >

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

bigdata July 2013