Spark MLlib Module

A Spark MLlib Module is a Spark module intended as a data analytics platform (a machine learning library for Spark-based cluster computing jobs).

• Context:
  • It can be part of a Distributed Machine Learning System.
  • It can have been superseded by an Spark.ML module.
  • It can be used by a Spark ML Program.
  • It can be accessed via a pyspark API.
  • It can contain libraries, such as:
    • org.apache.spark.mllib.classification.LogisticRegressionModel, and LogisticRegressionWithLBFGS (logistic regression system).
• Example(s):
  • MLlib 2.2.0.
  • MLlib 1.1.
  • …
• Counter-Example(s):
  • SparkR.
  • scikit-learn.
  • Mahout.
  • Samsung Veles.
  • TensorFlow,
  • GraphLab,
  • Vowpal Wabbit Open Source Project,
  • Apache Spark-Kafka Oryx 2.
• See: Scala Library, Scipy Library, Python, Pytorch, TensorFlow, Numpy.

---

---

References

2016a

• (Github, 2016) ⇒ http://lintool.github.io/bigdata-2016w/
  • QUOTE: ... One might break down the "big data" stack in the manner shown on the right. At the bottom resides the execution infrastructure, which is responsible for coordinating computations across a cluster (examples include MapReduce and Spark). In the middle resides analytics infrastructure, which implements data mining and machine learning algorithms on top of the execution infrastructure (an example would be MLlib in Spark). At the top are the tools data scientists use to generate insights, built on top of the analytics infrastructure.

2016b

• (Meng et al., 2016) ⇒ Xiangrui Meng, Joseph Bradley, Burak Yavuz, Evan Sparks, Shivaram Venkataraman, Davies Liu, Jeremy Freeman, DB Tsai, Manish Amde, Sean Owen, Doris Xin, Reynold Xin, Michael J. Franklin, Reza Zadeh, Matei Zaharia, and Ameet Talwalkar. (2016). “MLlib: Machine Learning in Apache Spark.” In: The Journal of Machine Learning Research, 17. ISBN:1938-7228 arXiv:1505.06807
  • QUOTE: In this work we present MLlib, Spark’s distributed machine learning library, and the largest such library. The library targets large-scale learning settings that benefit from data-parallelism or model-parallelism to store and operate on data or models. MLlib consists of fast and scalable implementations of standard learning algorithms for common learning settings including classification, regression, collaborative filtering, clustering, and dimensionality reduction. It also provides a variety of underlying statistics, linear algebra, and optimization primitives. Written in Scala and using native (C++ based) linear algebra libraries on each node, MLlib includes Java, Scala, and Python APIs, and is released as part of the Spark project under the Apache 2.0 license.

2015

• (Apache Spark, 2015) ⇒ https://spark.apache.org/docs/latest/mllib-guide.html
  • QUOTE: MLlib is Spark’s scalable machine learning library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as underlying optimization primitives, as outlined below:
    • Data types
    • Basic statistics.
      • summary statistics.
      • correlations.
      • stratified sampling.
      • hypothesis testing.
      • random data generation.
    • Classification and regression.
      • linear models (SVMs, logistic regression, linear regression)
      • naive Bayes.
      • decision trees.
      • ensembles of trees (Random Forests and Gradient-Boosted Trees)
    • Collaborative filtering.
      • alternating least squares (ALS).
    • Clustering.
      • k-means.
    • Dimensionality reduction.
      • singular value decomposition (SVD).
      • principal component analysis (PCA).
    • Feature extraction and transformation.
    • Optimization (developer)
      • stochastic gradient descent.
      • limited-memory BFGS (L-BFGS)

2014

• (Apache Spark, 2014) ⇒ https://spark.apache.org/docs/1.1.0/mllib-guide.html
  • MLlib is Spark’s scalable machine learning library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as underlying optimization primitives, as outlined below:

2014b

• (Apache Spark, 2014) ⇒ https://spark.apache.org/docs/latest/mllib-classification-regression.html
  • MLlib supports various methods for binary classification, multiclass classification, and regression analysis. The table below outlines the supported algorithms for each type of problem.
    • Binary Classification linear SVMs, logistic regression, decision trees, naive Bayes.
    • Multiclass Classification decision trees, naive Bayes.
    • Regression linear least squares, Lasso, ridge regression, decision trees

---