Streaming data usually possesses the following four characteristics: Data is continuously generated; often originates from more than one source; is unavailable as a complete data set; requires incremental processing.

Console and Memory are data sink options that are not fault-tolerant and that are recommended only for debugging.

Apache Spark Structured Streaming processes a data stream with the Spark SQL engine using DataFrame orDatasetAPIs.

You can download the GraphFrames package from the spark-packages.org website.

A directed graph contains edges with a single direction between two vertices, indicating a one-way relationship, illustrated using lines with arrows.

Undirected graphs have edges representing a relationship without a direction, illustrated using lines without arrows.

In the context of Apache Spark, graphs are a construct that contains a set of vertices with pairwise edges that connect one vertex to another.

Graph theory, in the context of Apache Spark, is the mathematical study of modeling pairwise relationships between objects.

Watermarking is the process that manages late data.

Watermarking enables the inclusion of late-arriving data stream processing.

Watermarking updates results after initial data processing.

GraphFrames Comes with popular built-in graph algorithms for use with the edge and vertex DataFrames.

GraphFrames provides one DataFrame for graph vertices and one DataFrame for edges that can be used with SparkSQL for analysis.

GraphFrames performs Motif finding, which searches the graph for structural patterns. Motif finding is supported in GraphFrames with the `find()` method that uses domain specific language (DSL) to specify the search query in terms of edges and vertices.

GraphFrames is ideal for modeling data with connecting relationships and computes relationship strength and direction.

Spark supports extracting data via JDBC

Spark supports extracting data from Parquet.

Spark supports extracting data from Apache ORC.

A machine learning system applies a specific machine learning algorithm to train data models. After training the model, the system infers or “predicts” results on previously unseen data.

Spark ML utilities help during the intermediate steps of data processing, cleaning, and building models.

Spark ML inbuilt utilities includes the Feature module.

Spark ML inbuilt utilities includes a linear algebra package.

Spark ML inbuilt utilities includes a statistics package.

Spark ML supports both feature vector and label column data.

LIBSVM loads the ”libsvm” data files and creates a DataFrame with two columns including the feature vector and label.

The Spark ML library provides the spark.ml.classification library for classifications.

The Spark ML model predicts each object’s target category or “class.”

Producing a prediction from a discrete set of possible outcomes from the task is called classificati

Regression is a form of an implicit function approximation where the model predicts real valued outputs for a given input.

Examples of regression analysis include Weather predictions, stock market price predictions, house value estimation, and others.

Regression predicted values are usually in the form of a continuous real number, such as a float or integer.

Unsupervised learning does not require explicit labels mapped to features.

Unsupervised learning is a subset of machine learning algorithms.

Unsupervised learning automatically learns patterns and latent spaces in the data.

In order to perform clustering using Spark ML, follow the following steps: First load the data. Next, create the model and train it. And finally, perform predictions on the test data.

The Spark MLlib provides a clustering library located at (spark.ml.clustering).

The Spark MLlib provides functions for k-means.

The Spark MLlib provides functions for Gaussian Mixture Models.

The Spark MLlib provides functions for Latent Dirichlet Allocation.