spark

org.hammerlab.sbt:spark

addSbtPlugin("org.hammerlab.sbt" % "spark" % "5.0.0")

SBT configs for projects that use Apache Spark; (source):

spark

This adds:

• provided dependencies on Spark (2.4.0) and Hadoop (2.7.3)
• a test-dep on hammerlab/spark-tests (2.4.0)
• a dep on Kryo (2.24.0)
• adds a Scala 2.11 cross-build to the default 2.12-only build inherited from the org.hammerlab.sbt:scala plugin

Version DSL

Relevant versions can be set via dedicated settings:

 spark      .version := "2.2.1"
 spark.tests.version := "2.3.3"
hadoop      .version := "2.7.3"
  kryo      .version := "2.24.0"

(these values are the defaults)

Versions can also be set via the versions helper:

versions(
   spark       → "2.4.0",
   spark.tests → "2.4.0",
  hadoop       → "2.7.3",
    kryo       → "2.24.0"
)

The Spark and Hadoop versions can also be overriden via the system parameters spark.version and hadoop.version.

Spark submodules

In the above, spark defaults to spark.core; other modules are also available:

dep(
  spark.graphx,
  spark.mllib,
  spark.sql
)