F21 Self Contained Change: Apache Spark
jreznik at redhat.com
Fri Mar 21 12:28:00 UTC 2014
= Proposed Self Contained Change: Apache Spark =
Change owner(s): William Benton <willb at redhat.com>
Apache Spark is a fast and general engine for large-scale data processing.
This change brings Spark to Fedora, allowing easy deployment and development
of Spark applications on Fedora.
== Detailed Description ==
Apache Spark is a fast and general engine for large-scale data processing. It
supports developing custom analytic processing applications over large data
sets or streaming data. Because it has the capability to cache intermediate
results in cluster memory and schedule DAGs of computations, Spark programs
can run up to 100x faster than equivalent Hadoop MapReduce jobs. Spark
applications are easy to develop, parallel, fast, and resilient to failure,
and they can operate on data from in-memory collections, local files, a Hadoop-
compatible filesystem, or from a variety of streaming sources. Spark also
includes libraries for distributed machine learning and graph algorithms.
== Scope ==
* Proposal owners: Currently our Spark package has been accepted into Fedora
. It features nearly all of the functionality available from the upstream
release. (The missing features -- specifically, Python bindings, the Spark
REPL, Kryo-based serialization, primitives for approximate cardinalities of
very large sets, and Mesos integration -- were missing from the initial
packages due to unavailable dependencies and bundling issues; we're working to
close the gap with upstream as quickly as possible.) This work depended upon
Fedora 21's improved support for the Scala ecosystem .
* Other developers: N/A (not a System Wide Change)
* Release engineering: N/A (not a System Wide Change)
* Policies and guidelines: N/A (not a System Wide Change)
More information about the devel-announce