F21 Self Contained Change: Apache Spark

[Jaroslav Reznik]

= Proposed Self Contained Change: Apache Spark =
https://fedoraproject.org/wiki/Changes/ApacheSpark

Change owner(s): William Benton <willb at redhat.com>

Apache Spark is a fast and general engine for large-scale data processing. 
This change brings Spark to Fedora, allowing easy deployment and development 
of Spark applications on Fedora. 

== Detailed Description ==
Apache Spark is a fast and general engine for large-scale data processing. It 
supports developing custom analytic processing applications over large data 
sets or streaming data. Because it has the capability to cache intermediate 
results in cluster memory and schedule DAGs of computations, Spark programs 
can run up to 100x faster than equivalent Hadoop MapReduce jobs. Spark 
applications are easy to develop, parallel, fast, and resilient to failure, 
and they can operate on data from in-memory collections, local files, a Hadoop-
compatible filesystem, or from a variety of streaming sources. Spark also 
includes libraries for distributed machine learning and graph algorithms. 

== Scope ==
* Proposal owners: Currently our Spark package has been accepted into Fedora 
[1]. It features nearly all of the functionality available from the upstream 
release. (The missing features -- specifically, Python bindings, the Spark 
REPL, Kryo-based serialization, primitives for approximate cardinalities of 
very large sets, and Mesos integration -- were missing from the initial 
packages due to unavailable dependencies and bundling issues; we're working to 
close the gap with upstream as quickly as possible.) This work depended upon 
Fedora 21's improved support for the Scala ecosystem [2]. 
* Other developers: N/A (not a System Wide Change) 
* Release engineering: N/A (not a System Wide Change) 
* Policies and guidelines: N/A (not a System Wide Change) 

[1] http://pkgs.fedoraproject.org/cgit/spark.git
[2] https://fedoraproject.org/wiki/Changes/ImprovedScalaEcosystem