Data type mapping

Neo4j and Cypher® provide a type system that describes how values are stored in the database, but these types do not always exactly match what Spark provides.

In some cases, there are data types that Neo4j provides that Spark does not have an equivalent for, and vice versa.

Data type mappings

Table 1. Spark to Neo4j type mapping reference
Neo4j type Spark type Notes

String

string

Example: "Hello"

Integer

long, short, byte

Example: 12345

Float

double

Example: 3.141592

String

decimal

New in 5.4.0 Example: "16725.77423461" This mapping only applies when writing to Neo4j. If you need to read a spark decimal, parse the string representation in an appropriate way for your use case.

Boolean

boolean

Example: true

Point

struct { type: string, srid: integer, x: double, y: double, z: double }

For more information on spatial types in Neo4j, see Spatial values

Date

date

Example: 2020-09-11

Time

struct { type: string, value: string }

Example: [offset-time, 12:14:08.209Z]

LocalTime

struct { type: string, value: string }

Example: [local-time, 12:18:11.628]

ZonedDateTime

timestamp

New in 5.4.0 Example: 2020-09-11 12:17:39.192+02:00 Spark timestamps (TIMESTAMP) are written as ZonedDateTime in UTC.

LocalDateTime

timestamp

Deprecated since 5.4.0 Set data.conversion to legacy to continue using it.

LocalDateTime

timestamp_ntz

New in 5.4.0 Example: 2020-09-11 12:14:49.081 Spark local timestamps (TIMESTAMP_NTZ) are written as LocalDateTime.

Integer

timestamp_ntz

Deprecated since 5.4.0 Set data.conversion to legacy to continue using it.

Duration

struct { type: string, months: long, days: long, seconds: long, nanoseconds: integer, value: string }

See Temporal functions: duration.

Duration

duration or period objects

New in 5.4.0 Example: java.time.Duration.ofDays(42) or java.time.Period.ofMonths(5).

Long

duration or period objects

Deprecated since 5.4.0 Set data.conversion to legacy to continue using it.

Duration

Spark SQL INTERVAL types

New in 5.4.0 Example: INTERVAL '10 05:30' DAY TO MINUTE, INTERVAL '4-5' YEAR TO MONTH or timestamp('2025-01-02 18:30:00.454') - timestamp('2024-01-01 00:00:00').

Long

Spark SQL INTERVAL types

Deprecated since 5.4.0 Set data.conversion to legacy to continue using it.

Node

struct { <id>: long, <labels>: array[string], (PROPERTIES) }

Nodes in Neo4j are represented as property containers; that is they appear as structs with properties corresponding to whatever properties were in the node. For ease of use it is usually better to return individual properties than a node from a query.

Relationship

struct { <rel.id>: long, <rel.type>: string, <source.id>: long, <target.id>: long, (PROPERTIES) }

Relationships are returned as maps, identifying the source and target of the relationship, its type, along with properties (if any) of the relationship. For ease of use it is usually better to return individual properties than a relationship from a query.

Path

string

Example: path[(322)←[20280:AIRLINE]-(33510)]. For ease of use it is recommended to use path functions to return individual properties/aspects of a path from a query.

ByteArray

binary, array[byte]

New in 5.4.0 Binary writes are treated specifically as Neo4j’s ByteArray type.

List<Integer>

binary, array[byte]

Deprecated since 5.4.0 Set data.conversion to legacy to continue using it.

[Array of same type]

array[element]

In Neo4j, arrays must be consistently typed (for example, an array must contain only Float values). The inner Spark type matches the type mapping above.

Complex data types

Spark does not natively support all Neo4j data types (for example Point, Time, Duration). Such types are transformed into Struct types containing all the useful data.

Table 2. Complex data type conversion
Neo4j type Spark Struct

Duration

 
Struct(Array(
    ("type", DataTypes.StringType, false),
    ("months", DataTypes.LongType, false),
    ("days", DataTypes.LongType, false),
    ("seconds", DataTypes.LongType, false),
    ("nanoseconds", DataTypes.IntegerType, false),
    ("value", DataTypes.StringType, false)
  ))

Point

 
Struct(Array(
    ("type", DataTypes.StringType, false),
    ("srid", DataTypes.IntegerType, false),
    ("x", DataTypes.DoubleType, false),
    ("y", DataTypes.DoubleType, false),
    ("z", DataTypes.DoubleType, true),
  ))

Time

 
Struct(Array(
    ("type", DataTypes.StringType, false),
    ("value", DataTypes.StringType, false)
  ))

Map type or Struct type

The connector treats Spark’s maps and structs in the same way except for when a struct matches the struct-specific data representations for complex types, mentioned above. For the sake of simplicity, this section covers maps only in code examples. You can assume structs has equivalent functionality.

Since Neo4j does not support arbitrary map or struct objects as values for properties, the connector needs to flatten the maps before writing to Neo4j. To demonstrate how the flattening is done, consider this dataset:

id name lives_in

1

Andrea Santurbano

{address: 'Times Square, 1', city: 'NY', state: 'NY'}

2

Davide Fantuzzi

{address: 'Statue of Liberty, 10', city: 'NY', state: 'NY'}

The connector flattens the map or struct stored at column lives_in into three separate columns lives_in.address, lives_in.city, and lives_in.state:

id name lives_in.address lives_in.city lives_in.state

1

Andrea Santurbano

Times Square, 1

NY

NY

2

Davide Fantuzzi

Statue of Liberty, 10

NY

NY

For an end-to-end example, consider the following Spark job:

val data = Seq(
  ("A", 1, Map("inner" -> Map("key" -> "table value one"))),
  ("B", 2, Map("inner" -> Map("key" -> "table value two"))),
).toDF("id", "time", "table")

data.write
  .mode(SaveMode.Append)
  .format("org.neo4j.spark.DataSource")
  .option("url", "neo4j://localhost:7687")
  .option("labels", ":Object")
  .save()

Because of the map or struct flattening in the connector you will find Neo4j nodes with label Object represented like this in the end:

Object {
    id: 'A',
    time: 1,
    `table.inner.key`: 'table value one'
}
Object {
    id: 'B',
    time: 1,
    `table.inner.key`: 'table value two'
}

Duplication issue with map or structs

It is possible to fall into problematic situations when the flattened maps end up as duplicated columns of other flattened maps. Consider these two maps:

val data = Seq(
  ("A", 1, Map("key.inner" -> Map("key" -> "table value one"), "key" -> Map("inner.key" -> "one"))),
  ("B", 2, Map("key.inner" -> Map("key" -> "table value two"), "key" -> Map("inner.key" -> "two"))),
).toDF("id", "time", "table")
data.write
  .mode(SaveMode.Append)
  .format("org.neo4j.spark.DataSource")
  .option("url", "neo4j://localhost:7687")
  .option("labels", ":Object")
  .save()

The intermediate data columns after flattening these keys will collide. This results in duplicated keys. In this scenario, the connector picks one of the associated values in a non-deterministic way. To demonstrate, this is what the flattened intermediate structure looks like:

id time table.key.inner.key

'A'

1

'table value one' OR 'one'

'B'

2

'table value two' OR 'two'

The resulting nodes in Neo4j are the following, where the values picked are not guaranteed:

Object {
    id: 'A',
    time: 1,
    `table.key.inner.key`: 'table value one' // OR 'one'
}
Object {
    id: 'B',
    time: 2,
    `table.key.inner.key`: 'table value two' // OR 'two'
}

Prevent duplication issue with map or structs

You can use the schema.map.group.duplicate.keys option to avoid the duplicated key problem. The default value for the option is false. When set to true, all the values with the same keys are grouped into an array. In a scenario like this:

val data = Seq(
  ("A", 1, Map("key.inner" -> Map("key" -> "innerValue0"), "key" -> Map("inner.key" -> "value0"))),
  ("B", 1, Map("key.inner" -> Map("key" -> "innerValue1"), "key" -> Map("inner.key" -> "value1"))),
).toDF("id", "time", "table")
data.write
  .mode(SaveMode.Append)
  .format("org.neo4j.spark.DataSource")
  .option("url", "neo4j://localhost:7687")
  .option("labels", ":Object")
  .option("schema.map.group.duplicate.keys", true)
  .save()

You would instead get the Neo4j node representation where each duplicated value from the flattened maps are saved as arrays. The ordering is not guaranteed.

Object {
    id: 'A',
    time: 1,
    `table.key.inner.key`: ['innerValue0', 'value0']
}
Object {
    id: 'B',
    time: 1,
    `table.key.inner.key`: ['innerValue1', 'value1']
}

In order for schema.map.group.duplicate.keys to work, every "final" value of the maps or structs (the value that ends up being collected in the list) must be of one single type. This correlates to the fact that in Neo4j, arrays stored as properties must be homogeneous. If you break this, you get a Cypher exception.

Read/write round trips for Map, Struct or Arrays
Updated in 6.1

In an ideal scenario, if you read a map, struct or array from Spark, write it to Neo4j, then read it again from Spark, you should expect the same value back. However, there are some considerations regarding a full round trip where the connector handles both the write and the read.

For example, you may get different types writing from Spark to Neo4j and reading from Neo4j back to Spark. See Complex data types and Map type or Struct type for more details.

As described, it is possible for a struct to become a Duration, Point or Time in Neo4j given the right shape. Nested maps or structs will become a simple dot-notated column-name with plain values. Finally, it is possible maps or structs to become an Array if option schema.map.group.duplicate.keys is true.

Due to a long-living bug since connector version 5.1.0, fixed in connector version 5.6.0, round trips for flattened maps would yield spark NULL on read. If you experience flaky NULL values for nested structs and you are running a connector pre-5.6.0, consider upgrading your connector.

Once the value is written in Neo4j see the Spark to Neo4j type mapping reference for the expected type mappings. The table can be read bidirectionally.

Constraint type mapping

Table 3. Spark to Cypher constraint type mapping
Spark type Neo4j Type

BooleanType

BOOLEAN

StringType

STRING

IntegerType

INTEGER

LongType

INTEGER

FloatType

FLOAT

DoubleType

FLOAT

DateType

DATE

TimestampType

LOCAL DATETIME

Custom pointType as: Struct { type: string, srid: integer, x: double, y: double, z: double }

POINT

Custom durationType as: Struct { type: string, months: long, days: long, seconds: long, nanoseconds: integer, value: string }

DURATION

DataTypes.createArrayType(BooleanType, false)

LIST<BOOLEAN NOT NULL>

DataTypes.createArrayType(StringType, false)

LIST<STRING NOT NULL>

DataTypes.createArrayType(IntegerType, false)

LIST<INTEGER NOT NULL>

DataTypes.createArrayType(LongType, false)

LIST<INTEGER NOT NULL>

DataTypes.createArrayType(FloatType, false)

LIST<FLOAT NOT NULL>

DataTypes.createArrayType(DoubleType, false)

LIST<FLOAT NOT NULL>

DataTypes.createArrayType(DateType, false)

LIST<DATE NOT NULL>

DataTypes.createArrayType(TimestampType, false)

LIST<LOCAL DATETIME NOT NULL>

DataTypes.createArrayType(pointType, false)

LIST<POINT NOT NULL>

DataTypes.createArrayType(durationType, false)

LIST<DURATION NOT NULL>

For the arrays in particular we use the version without null elements as Neo4j does not allow to have them in arrays.