maandag 16 november 2026

Databricks series : ODCS Data contracts in Databricks (Part X)

Introduction

In the previous blogpost I wrote about contracts. Now there is also an open standard available : Open Data Contract Standard (ODCS). An video on Youtube makes it a lot clearer. In this blogpost I'll investigate the data contracts according to the ODCS in more detail. 


In modern data environments, reliable data is essential. However, the transfer of data between producers and consumers often encounters issues due to unclear agreements or unexpected changes. 


An ODCS contract (based on the Open Data Contract Standard) provides a solution to this. It is a standardized, machine-readable data contract (typically drafted in YAML) that serves as a formal agreement between the team providing the data and the teams or systems consuming it. Rather than relying on disparate documentation, an ODCS contract establishes exactly what the consumer can expect, all in one central location. This includes details such as data structure (schema), quality requirements, service-level agreements (SLAs), ownership, and usage policies. Consequently, errors can be prevented early on, changes become easier to manage, and a shared standard is established for both human users and automated data pipelines.

We are building upon what we already have :


ODCS contracts

We are transitioning our existing Databricks solution to a formal ODCS (Open Data Contract Standard) version 3.2.0 contract.

We will retain the elements that are already functional:
  • The Databricks Job with the 'entity' parameter.
  • The CSV files stored in Unity Catalog Volumes.
  • The Python DataFrame validation.
  • The 'overwrite', 'append', and 'merge' loading strategies.
  • The validation check ensuring that mandatory fields cannot be empty.
We will replace our custom JSON-based contract with a YAML-based contract that adheres to the official ODCS structure.

The new project structure 

Let s change the project structure..

databricks-transfer/
├── databricks.yml
├── resources/
│   └── transfer_job.yml
├── src/
│   ├── main.py
│   └── transfer/
│       ├── config_loader.py
│       ├── contract_validator.py
│       └── processor.py
├── config/
│   └── entities.json
├── contracts/
│   └── customers.yaml
├── data/
│   ├── customers.csv
│   └── customers_update.csv
└── requirements.txt


From now on, we will use 'customers.yaml' instead of 'customers.json'.


2. Create the ODCS contract

First, we create a new file: contracts/customers.yaml and replace the content with:

apiVersion: v3.2.0
kind: DataContract

id: urn:datacontract:databricks:customers
name: customers
version: 1.0.0
status: active

description:
  purpose: Customer master data
  usage: Used by the Databricks data transfer pipeline
  limitations: Customer identifiers must be unique

team:
  name: Data Engineering

servers:
  - server: databricks_dev
    type: databricks
    description: Databricks development environment

schema:
  - name: customers
    physicalName: main.demo.customers
    physicalType: table
    description: Customer master data

    properties:
      - name: customer_id
        logicalType: string
        physicalType: STRING
        primaryKey: true
        required: true
        description: Unique customer identifier

      - name: first_name
        logicalType: string
        physicalType: STRING
        required: true
        description: Customer first name

      - name: last_name
        logicalType: string
        physicalType: STRING
        required: true
        description: Customer last name

      - name: country
        logicalType: string
        physicalType: STRING
        required: true
        description: Customer country


What is the meaning of the fields:




Modify config/entities.json

We do not need to change the source files or loading strategies. Only the reference to the contract changes. This is now the new content of the file: config/entities.json


{
  "customers": {
    "source": "/Volumes/main/demo/raw/customers.csv",
    "target": "main.demo.customers",
    "format": "csv",
    "mode": "overwrite",
    "contract": "contracts/customers.yaml"
  },
  "customers_update": {
    "source": "/Volumes/main/demo/raw/customers_update.csv",
    "target": "main.demo.customers",
    "format": "csv",
    "mode": "merge",
    "merge_key": "customer_id",
    "contract": "contracts/customers.yaml"
  }
}


Both entities therefore use the same ODCS contract. This makes sense: they both describe the same dataset, even though they employ different loading strategies.


Adding YAML support

Since we are now reading YAML, we need PyYAML. We add an entry in the file: requirements.txt


PyYAML==6.0.2


The requirements.txt file defines the Python dependencies your project needs to run. In other words, it tells Python which external libraries must be installed in addition to the standard Python libraries


Execute the requirements.txt to make sure the packages are installed :


pip install -r requirements.txt


Also ensure that the dependency is available in the Job's serverless Python environment. Change the file: resources/transfer_job.yml and add dependencies under `spec`, if not already present:


environments:
  - environment_key: default
    spec:
      environment_version: "2"
      dependencies:
        - PyYAML==6.0.2


Keep the rest of your existing Job configuration.


Why both? Because installing PyYAML locally does not automatically install it in the Databricks serverless environment. The idea is to keep your dependencies explicit and reproducible, so someone else can set up the same Python project without guessing which libraries are needed.


Loading the contract in main.py

We are now changing the JSON parser to a YAML parser. replace the entire content of the file src/main.py with :


import argparse
from pathlib import Path

import yaml
from pyspark.sql import SparkSession

from transfer.config_loader import load_entity_config
from transfer.processor import transfer_data


def get_project_root():
    current_dir = Path.cwd()

    for candidate in [current_dir, *current_dir.parents]:
        if (
            (candidate / "config" / "entities.json").exists()
            and (candidate / "contracts").exists()
        ):
            return candidate

    raise FileNotFoundError(
        f"Could not locate project root. Current directory: {current_dir}"
    )


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--entity", required=True)
    args = parser.parse_args()

    spark = SparkSession.builder.getOrCreate()

    project_root = get_project_root()

    config_path = project_root / "config" / "entities.json"
    config = load_entity_config(config_path, args.entity)

    contract_path = project_root / config["contract"]

    with open(contract_path, "r", encoding="utf-8") as file:
        contract = yaml.safe_load(file)

    transfer_data(spark, config, contract)


if __name__ == "__main__":
    main()


Here, we once again deliberately use our 'get_project_root()' function, which we modified earlier. This allows us to avoid the issue with '__file__'.


Adapting the validator for ODCS

This is the most significant change. The previous validator expected:


contract["columns"]


But ODCS uses:


contract["schema"][0]["properties"]


Replace the content of the file: src/transfer/contract_validator.py with 


from pyspark.sql.functions import col, trim
from functools import reduce

TYPE_MAPPING = {
    "string": "string",
    "integer": "int",
    "long": "bigint",
    "number": "double",
    "boolean": "boolean",
    "date": "date",
    "timestamp": "timestamp"
}


def validate_dataframe(df, contract):
    errors = []

    # Read ODCS metadata
    contract_name = contract.get("name")
    contract_version = contract.get("version")

    if not contract_name:
        raise ValueError(
            "Invalid ODCS contract: missing required field 'name'."
        )

    if not contract_version:
        raise ValueError(
            "Invalid ODCS contract: missing required field 'version'."
        )

    print(f"Validating ODCS contract: {contract_name}")
    print(f"ODCS version: {contract['apiVersion']}")
    print(f"Contract version: {contract_version}")

    # This example uses one schema object
    schema_objects = contract["schema"]

    if len(schema_objects) != 1:
        raise ValueError(
            "This example supports exactly one ODCS schema object"
        )

    schema_object = schema_objects[0]
    properties = schema_object["properties"]

    expected = {
        item["name"]: item
        for item in properties
    }

    # Check columns
    actual_columns = set(df.columns)
    expected_columns = set(expected.keys())

    missing = expected_columns - actual_columns
    unexpected = actual_columns - expected_columns

    if missing:
        errors.append(f"Missing columns: {sorted(missing)}")

    # Strict schema matching is a pipeline policy in this example
    if unexpected:
        errors.append(f"Unexpected columns: {sorted(unexpected)}")

    if errors:
        raise ValueError(
            "Data contract schema validation failed:\n"
            + "\n".join(errors)
        )

    validated_df = df

    # Validate required fields and datatypes
    for name, definition in expected.items():

        logical_type = definition["logicalType"]
        required = definition.get("required", False)

        if logical_type not in TYPE_MAPPING:
            errors.append(
                f"Unsupported logicalType '{logical_type}' "
                f"for column {name}"
            )
            continue

        source_col = col(name)

        if required:
            invalid = df.filter(
                source_col.isNull() |
                (trim(source_col) == "")
            ).limit(1).count()

            if invalid:
                errors.append(
                    f"Column {name} cannot be NULL or empty"
                )

        spark_type = TYPE_MAPPING[logical_type]

        if logical_type != "string":
            converted = source_col.cast(spark_type)

            invalid = df.filter(
                source_col.isNotNull() &
                (trim(source_col) != "") &
                converted.isNull()
            ).limit(1).count()

            if invalid:
                errors.append(
                    f"Invalid {logical_type} values in column {name}"
                )

            validated_df = validated_df.withColumn(
                name,
                converted
            )

    # Validate primary key
    primary_keys = [
        item["name"]
        for item in properties
        if item.get("primaryKey", False)
    ]

    if not primary_keys:
        errors.append("No primary key defined in ODCS contract")
    else:
        pk_invalid_condition = reduce(
            lambda left, right: left | right,
            [
                col(name).isNull() | (trim(col(name)) == "")
                for name in primary_keys
            ]
        )

        invalid_pk = validated_df.filter(
            pk_invalid_condition
        ).limit(1).count()

        if invalid_pk:
            errors.append(
                f"Primary key columns cannot be NULL or empty: "
                f"{primary_keys}"
            )

        duplicates = (
            validated_df
            .groupBy(*primary_keys)
            .count()
            .filter(col("count") > 1)
            .limit(1)
            .count()
        )

        if duplicates:
            errors.append(
                f"Duplicate primary key values found: {primary_keys}"
            )

    if errors:
        raise ValueError(
            "Data contract validation failed:\n"
            + "\n".join(errors)
        )

    print(
        f"ODCS contract {contract_name} "
        f"version {contract_version} PASSED"
    )

    return validated_df


The validator now performs the following steps:

1. Reads the ODCS metadata.

2. Retrieves the columns from 'schema.properties'.

3. Checks for missing or unexpected columns.

4. Validates 'required: true' constraints.

5. Verifies data types via 'logicalType'.

6. Checks the primary key for null values ​​and duplicates.

The strict check for additional columns is a deliberate design choice in our Python pipeline; these are distinct from ODCS fields.


processor.py does not need to change

At first I thought it was not necessary to change the processor.py. But after some debuging with AI I noticed that the merge didn't work well and I needed to replace the file.


from delta.tables import DeltaTable
from transfer.contract_validator import validate_dataframe

def transfer_data(spark, config, contract=None):

    source = config["source"]
    target = config["target"]
    file_format = config["format"]
    mode = config["mode"]

    print(f"Reading source: {source}")
    print(f"Target table: {target}")
    print(f"Loading mode: {mode}")

    # Read everything as strings first

    df = (
        spark.read
        .format(file_format)
        .option("header", "true")
        .option("inferSchema", "false")
        .load(source)
    )

    # Validate before writing

    if contract is None:
        raise ValueError(
            "No data contract configured for this entity"
        )

    df = validate_dataframe(df, contract)

    record_count = df.count()

    print(f"Validated records: {record_count}")

    # Apply loading strategy

    if mode == "overwrite":

        (
            df.write
            .format("delta")
            .mode("overwrite")
            .saveAsTable(target)
        )

    elif mode == "append":

        (
            df.write
            .format("delta")
            .mode("append")
            .saveAsTable(target)
        )

 
    elif mode == "merge":

        merge_key = config["merge_key"]

        # Read primary keys from the ODCS contract
        primary_keys = [
            item["name"]
            for item in contract["schema"][0]["properties"]
            if item.get("primaryKey") is True
        ]

        print(f"ODCS primary keys: {primary_keys}")
        print(f"Configured merge key: {merge_key}")

        if merge_key not in primary_keys:
            raise ValueError(
                f"Merge key {merge_key} is not defined "
                f"as a contract primary key. Found: {primary_keys}"
            )

        if spark.catalog.tableExists(target):

            delta_table = DeltaTable.forName(
                spark, target
            )

            (
                delta_table.alias("target")
                .merge(
                    df.alias("source"),
                    f"target.{merge_key} = source.{merge_key}"
                )
                .whenMatchedUpdateAll()
                .whenNotMatchedInsertAll()
                .execute()
            )

        else:

            (
                df.write
                .format("delta")
                .mode("overwrite")
                .saveAsTable(target)
            )


    else:
        raise ValueError(
            f"Unsupported loading mode: {mode}"
        )

    print(
        f"Successfully processed {record_count} records"
    )


The processor does not need to know whether the contract originates from JSON or YAML; that is the responsibility of 'main.py'.


8. Testing in Databricks

In your VS Code terminal, run:


databricks bundle validate -t dev


Then

databricks bundle deploy -t dev


and :


databricks bundle run transfer_job -t dev --params entity=customers


Resulting in :




and execute :


databricks bundle run transfer_job -t dev --params entity=customers_update


Resulting in :



Hennie

Geen opmerkingen:

Een reactie posten