Reader options

Table 1. List of available read options
Setting name Description Default value Required

query

Cypher® query to read the data

(none)

Yes*

labels

List of node labels separated by colon. The first label is to be the primary label.

(none)

Yes*

relationship

Type of a relationship

(none)

Yes*

schema.flatten.limit

Number of records to be used to create the Schema (only if APOC is not installed, or for custom Cypher queries provided via query options).

10

No

schema.strategy

Strategy used by the connector in order to compute the Schema definition for the Dataset. Possible values are string, sample. When string is set, it coerces all the properties to String, otherwise it tries to sample the Neo4j’s dataset.

sample

No

pushdown.filters.enabled

Enable or disable the PushdownFilters support.

true

No

pushdown.columns.enabled

Enable or disable the PushdownColumn support.

true

No

pushdown.aggregate.enabled

Enable or disable the PushdownAggregate support.

true

No

pushdown.limit.enabled

Enable or disable the PushdownLimit support.

true

No

pushdown.topN.enabled

Enable or disable the PushDownTopN support.

true

No

partitions

This defines the parallelization level while pulling data from Neo4j.

Note: as more parallelization does not mean better query performance, tune wisely in according to your Neo4j installation.

1

No

type.conversion

Data conversion logic. When set to legacy, timestamps, intervals and byte arrays are processed the same way as they were before 5.4.0. See Data type mapping for more information.

default

No

transaction.retries

Number of times a partition may retry its read query after a transient failure. See Read retries.

3

No

transaction.retry.timeout

The time in milliseconds to wait before each retry.

0

No

Query specific options

query.count

Query count is used only in combination with query option. This is a query that returns a count field like the following:

MATCH (p:Person)-[r:BOUGHT]->(pr:Product)
WHERE pr.name = 'An Awesome Product'
RETURN count(p) AS count

or a simple number that represents the number of records returned by query. Consider that the number passed by this value represents the volume of the data pulled off Neo4j, so use it carefully.

(empty)

No

Relationship specific options

relationship.nodes.map

If it’s set to true, source and target nodes are returned as Map<String, String>, otherwise we flatten the properties by returning every single node property as column prefixed by source or target

false

No

relationship.source.labels

List of source node labels separated by colon.

(empty)

Yes

relationship.target.labels

List of target node labels separated by colon.

(empty)

Yes

Cypher options [role=label—​new-6.0]

cypher.version

Set a custom Cypher language version to a custom value. The expected value is the version number itself, for example 5 or 25.

5

No

cypher.tuning.*

Set any cypher query tuning parameter with this prefix. See more details on how to properly use this feature on /docs/spark/current/performance/cypher-tuning.

(emtpy)

No

Optimization options

script

Apply a custom Cypher query which will run once before any write operations. Use this option to apply optimizations such as the use of a custom index and constraint queries. Can not be used together with the numbered script.N option.

(empty)

No

script.N

Similar to script, replace N with any positive integer, put as many as you like. The scripts are executed in succession and in numbered order. Can not be used together with the unnumbered script option.

(empty)

No

* Just one of the options can be specified at the time.

Read retries

Each partition runs its own query in its own transaction. If that query fails with a transient error, the partition retries it, up to transaction.retries times, waiting transaction.retry.timeout milliseconds between attempts.

An error is transient when the Neo4j driver marks it as retryable, which covers a lost connection (ServiceUnavailableException), a server that can no longer serve the request (SessionExpiredException), transient server errors such as deadlocks (TransientException), and expired authorization.

A retry re-runs the query from the beginning. The connector does not track how far the partition had already read, so rows the failed attempt had already produced are read and emitted again. If a read fails partway through and is retried, the partition can therefore contain duplicate rows.

Set transaction.retries to 0 if you would rather have the read fail and be retried as a whole Spark task, which does not have this problem.

Retries also interact with partitioned reads. With partitions greater than 1, each partition reads a different SKIP/LIMIT window in a separate transaction, so the partitions do not share a consistent snapshot of the graph. If the data changes while the read is in progress, a retried partition sees the newer state, and rows can be missed or read twice. When you need a consistent result, read from a database that is not being written to for the duration of the read.