Spark SQL Statement

The Spark SQL Statement task executes a Spark SQL script on a Spark enabled connection.

As is the case with the SQL Statement task, this task does not show output and is intended for creating and modifying database tables.

Currently the Spark SQL Statement task type only supports execution on Databricks connections. For general Hadoop Spark invocation use the Spark task type.

Provide Spark SQL Statement

Once the Spark SQL task type is selected, all that’s required for the Spark SQL task is a valid connection and the statement to run.

For Databricks connections you will be required to select the cluster definition or existing cluster to use for the statement.

The task type accepts any valid Spark SQL statement, and will submit the statement as part of the task. Currently Spark SQL Statement do not have the ability to be validated so Data Governor cannot check the syntax of your statements.