Options and configuration

Spark options are useful not only to control how the connector reads to and writes from a Neo4j database, but also to manage multiple connections and manage Neo4j driver options.

Multiple connections

The connector allows you to use more than one connection within a single Spark Session. For example, you can read data from a database and write it to another database in the same session.

Reading from a database and writing to a different one
import org.apache.spark.sql.{SaveMode, SparkSession}

val spark = SparkSession.builder().getOrCreate()

val df = spark.read.format("org.neo4j.spark.DataSource")
  .option("url", "neo4j://first.host.com:7687")
  .option("labels", "Person")
  .load()

df.write.format("org.neo4j.spark.DataSource")
  .mode(SaveMode.Append)
  .option("url", "neo4j://second.host.com:7687")
  .option("labels", "Person")
  .save()

Another case to use multiple connections is when you want to merge two data sources.

Merge data from two databases
import org.apache.spark.sql.{SaveMode, SparkSession}

val spark = SparkSession.builder().getOrCreate()

val dfOne = spark.read.format("org.neo4j.spark.DataSource")
  .option("url", "neo4j://first.host.com:7687")
  .option("labels", "Person")
  .load()

val dfTwo = spark.read.format("org.neo4j.spark.DataSource")
  .option("url", "neo4j://second.host.com:7687")
  .option("labels", "Person")
  .load()

val dfJoin = dfOne.join(dfTwo, dfOne("name") === dfTwo("name"))

Authentication

The connector supports the none, basic, kerberos, custom, bearer and oidc authentication types, and authentication types of your own. See Authentication.

Neo4j driver options

Under the covers, the Spark connector uses the official Neo4j Java Driver. When using the connector, you can set any valid Neo4j driver option using the option method:

import org.apache.spark.sql.{SaveMode, SparkSession}

val spark = SparkSession.builder().getOrCreate()

val df = spark.read.format("org.neo4j.spark.DataSource")
        .option("url", "neo4j://localhost:7687")
        .option("authentication.type", "basic")
        .option("authentication.basic.username", "myuser")
        .option("authentication.basic.password", "neo4jpassword")
        .option("labels", "Person")
        .load()

Alternatively, you can specify a global configuration in the Spark Session to avoid retyping connection options every time. You can set any Neo4j Connector option adding the neo4j. prefix.

For example, if you want to set the authentication.type option for the session, you have to add neo4j.authentication.type. Here is a full example:

import org.apache.spark.sql.{SaveMode, SparkSession}

val spark = SparkSession.builder()
    .config("neo4j.url", "neo4j://localhost:7687")
    .config("neo4j.authentication.type", "basic")
    .config("neo4j.authentication.basic.username", "myuser")
    .config("neo4j.authentication.basic.password", "neo4jpassword")
    .getOrCreate()

val dfPerson = spark.read.format("org.neo4j.spark.DataSource")
        .option("labels", "Person")
        .load()

val dfProduct = spark.read.format("org.neo4j.spark.DataSource")
        .option("labels", "Product")
        .load()

The following table captures the most common configuration settings to use with the Neo4j driver. For full documentation on all possible configuration options for Neo4j Drivers, see the Neo4j Java Driver manual.

Table 1. List of available configuration settings
Setting name Description Default value Required

Driver options

url

The url of the Neo4j instance to connect to.

When provided with a comma-separated list of URIs, the resolver function feature of the driver will be activated. The first URI will be used as original host while the rest are treated as resolver function outputs.

(none)

Yes

authentication.type

The authentication methods to be used:

See Authentication for more information.

basic

No

authentication.basic.username

Username to use for basic authentication type

(Neo4j Driver default)

No

authentication.basic.password

Password to use for basic authentication type

(Neo4j Driver default)

No

authentication.kerberos.ticket

Kerberos Auth Ticket

(Neo4j Driver default)

No

authentication.custom.principal

This is used to identify who this token represents

(Neo4j Driver default)

No

authentication.custom.credentials

These are the credentials authenticating the principal

(Neo4j Driver default)

No

authentication.custom.realm

This is the "realm" string specifying the authentication provider

(Neo4j Driver default)

No

authentication.bearer.token

This is the token to provide for the bearer authentication scheme

(Neo4j Driver default)

No

encryption.enabled

Specify if encryption should be enabled. This setting is ignored if you use a URI scheme with +s or +ssc

false

No

encryption.trust.strategy

Set certificate trust strategy, it is ignored if the connection URI uses +s or +ssc as suffix. Available values are:

  • TRUST_SYSTEM_CA_SIGNED_CERTIFICATES

  • TRUST_CUSTOM_CA_SIGNED_CERTIFICATES

  • TRUST_ALL_CERTIFICATES.

(Neo4j Driver default)

No

encryption.ca.certificate.path

Set certificate path for TRUST_CUSTOM_CA_SIGNED_CERTIFICATES trust strategy

(Neo4j Driver default)

No

connection.max.lifetime.msecs

Connection lifetime in milliseconds

(Neo4j Driver default)

No

connection.liveness.timeout.msecs

Liveness check timeout in milliseconds

(Neo4j Driver default)

No

connection.acquisition.timeout.msecs

Connection acquisition timeout in milliseconds

(Neo4j Driver default)

No

connection.timeout.msecs

Connection timeout in milliseconds

(Neo4j Driver default)

No

db.transaction.timeout

Transaction timeout in milliseconds

(Neo4j Driver default)

No

db.transaction.metadata

Tag custom transaction metadata to the driver, works on read and write, details below

(empty)

No

Session options

database

Database name to connect to. The driver allows to define the database in the URL, yet in case you set this option, it has the priority compared to the one defined in the URL.

(Neo4j Driver default)

No

access.mode

Possible values are:

  • read

  • write

Only affects how reads are routed. No effect on writes. In a cluster, read routes read requests to the followers, while write routes them to the leader. Writes always go to the leader, regardless of the value of this option.

read

No

Driver transaction metadata
Introduced in 6.0

You can use transaction metadata to label your transactions in the target Neo4j database query log, to help identifying them as Cypher® queries generated by the Spark connector.

Metadata is sent as a object/map in the driver, and you populate it using options prefix db.transaction.metadata. Keys are dot-separated and interpreted right after the prefix:

Example
val df = spark.read
    .format("org.neo4j.spark.DataSource")
    .option("query", "RETURN 42 AS example")
    .option("db.transaction.metadata.fromMySparkApp", true)
    .option("db.transaction.metadata.anotherMetadata", "free text goes here")
    .load()
Example
df = (
    spark.read.format("org.neo4j.spark.DataSource")
    .option("query", "RETURN 42 AS example")
    .option("db.transaction.metadata.fromMySparkApp", True)
    .option("db.transaction.metadata.withMessage", "free text goes here")
    .load()
)
Send the object to the driver via transaction metadata.
{
  "fromMySparkApp": true,
  "anotherMetadata": "free text goes here"
}

You can create nested structures by assigning more options. Each dot represents one step deeper into the nested map/object.

Example
val df = spark.read
    .format("org.neo4j.spark.DataSource")
    .option("query", "RETURN 42 AS example")
    .option("db.transaction.metadata.fromMySparkApp", true)
    .option("db.transaction.metadata.example.id", 55)
    .option("db.transaction.metadata.example.message", "another message")
    .load()
Example
df = (
    spark.read.format("org.neo4j.spark.DataSource")
    .option("query", "RETURN 42 AS example")
    .option("db.transaction.metadata.fromMySparkApp", True)
    .option("db.transaction.metadata.example.id", 55)
    .option("db.transaction.metadata.example.message", "another message")
    .load()
)
Send the object to the driver via transaction metadata.
{
  "fromMySparkApp": true,
  "example": {
    "id": 55,
    "message": "another message"
  }
}