Introduction
In the previous blogpost I wrote about contracts. Now there is also an open standard available : Open Data Contract Standard (ODCS). An video on Youtube makes it a lot clearer. In this blogpost I'll investigate the data contracts according to the ODCS in more detail.
We are building upon what we already have :
- Databricks series : Starting a new python project in Visual Studio Code (Part I)
- Databricks series : setting up a connection to Databricks from local IDE (Part II)
- Databricks series : Setting up a Databricks project in Visual Studio Code (part III)
- Databricks series : Deployment from Visual Studio Code (Part IV)
- Databricks series : Execute the job in Databricks (Part V)
- Databricks series : A configrable framework in Databricks (Part VI)
- Databricks series : Loading strategies in Databricks (Part VII)
- Databricks series : Data contracts in Databricks (Part VIII)
- Databricks series : Debugging Python code in Visual Studio Code with arguments (Part IX)
ODCS contracts
- The Databricks Job with the 'entity' parameter.
- The CSV files stored in Unity Catalog Volumes.
- The Python DataFrame validation.
- The 'overwrite', 'append', and 'merge' loading strategies.
- The validation check ensuring that mandatory fields cannot be empty.
The new project structure
databricks-transfer/ ├── databricks.yml ├── resources/ │ └── transfer_job.yml ├── src/ │ ├── main.py │ └── transfer/ │ ├── config_loader.py │ ├── contract_validator.py │ └── processor.py ├── config/ │ └── entities.json ├── contracts/ │ └── customers.yaml ├── data/ │ ├── customers.csv │ └── customers_update.csv └── requirements.txt
From now on, we will use 'customers.yaml' instead of 'customers.json'.
2. Create the ODCS contract
First, we create a new file: contracts/customers.yaml and replace the content with:
apiVersion: v3.2.0
kind: DataContract
id: urn:datacontract:databricks:customers
name: customers
version: 1.0.0
status: active
description:
purpose: Customer master data
usage: Used by the Databricks data transfer pipeline
limitations: Customer identifiers must be unique
team:
name: Data Engineering
servers:
- server: databricks_dev
type: databricks
description: Databricks development environment
schema:
- name: customers
physicalName: main.demo.customers
physicalType: table
description: Customer master data
properties:
- name: customer_id
logicalType: string
physicalType: STRING
primaryKey: true
required: true
description: Unique customer identifier
- name: first_name
logicalType: string
physicalType: STRING
required: true
description: Customer first name
- name: last_name
logicalType: string
physicalType: STRING
required: true
description: Customer last name
- name: country
logicalType: string
physicalType: STRING
required: true
description: Customer country
What is the meaning of the fields:
Modify config/entities.json
We do not need to change the source files or loading strategies. Only the reference to the contract changes. This is now the new content of the file: config/entities.json
{
"customers": {
"source": "/Volumes/main/demo/raw/customers.csv",
"target": "main.demo.customers",
"format": "csv",
"mode": "overwrite",
"contract": "contracts/customers.yaml"
},
"customers_update": {
"source": "/Volumes/main/demo/raw/customers_update.csv",
"target": "main.demo.customers",
"format": "csv",
"mode": "merge",
"merge_key": "customer_id",
"contract": "contracts/customers.yaml"
}
}
Both entities therefore use the same ODCS contract. This makes sense: they both describe the same dataset, even though they employ different loading strategies.
Adding YAML support
Since we are now reading YAML, we need PyYAML. We add an entry in the file: requirements.txt
PyYAML==6.0.2
The requirements.txt file defines the Python dependencies your project needs to run. In other words, it tells Python which external libraries must be installed in addition to the standard Python libraries
Execute the requirements.txt to make sure the packages are installed :
pip install -r requirements.txt
Also ensure that the dependency is available in the Job's serverless Python environment. Change the file: resources/transfer_job.yml and add dependencies under `spec`, if not already present:
environments: - environment_key: default spec: environment_version: "2" dependencies: - PyYAML==6.0.2
Keep the rest of your existing Job configuration.
Why both? Because installing PyYAML locally does not automatically install it in the Databricks serverless environment. The idea is to keep your dependencies explicit and reproducible, so someone else can set up the same Python project without guessing which libraries are needed.
Loading the contract in main.py
We are now changing the JSON parser to a YAML parser. replace the entire content of the file src/main.py with :
import argparse
from pathlib import Path
import yaml
from pyspark.sql import SparkSession
from transfer.config_loader import load_entity_config
from transfer.processor import transfer_data
def get_project_root():
current_dir = Path.cwd()
for candidate in [current_dir, *current_dir.parents]:
if (
(candidate / "config" / "entities.json").exists()
and (candidate / "contracts").exists()
):
return candidate
raise FileNotFoundError(
f"Could not locate project root. Current directory: {current_dir}"
)
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--entity", required=True)
args = parser.parse_args()
spark = SparkSession.builder.getOrCreate()
project_root = get_project_root()
config_path = project_root / "config" / "entities.json"
config = load_entity_config(config_path, args.entity)
contract_path = project_root / config["contract"]
with open(contract_path, "r", encoding="utf-8") as file:
contract = yaml.safe_load(file)
transfer_data(spark, config, contract)
if __name__ == "__main__":
main()
Here, we once again deliberately use our 'get_project_root()' function, which we modified earlier. This allows us to avoid the issue with '__file__'.
Adapting the validator for ODCS
This is the most significant change. The previous validator expected:
contract["columns"]
But ODCS uses:
contract["schema"][0]["properties"]Replace the content of the file: src/transfer/contract_validator.py with
from pyspark.sql.functions import col, trim
from functools import reduce
TYPE_MAPPING = {
"string": "string",
"integer": "int",
"long": "bigint",
"number": "double",
"boolean": "boolean",
"date": "date",
"timestamp": "timestamp"
}
def validate_dataframe(df, contract):
errors = []
# Read ODCS metadata
contract_name = contract.get("name")
contract_version = contract.get("version")
if not contract_name:
raise ValueError(
"Invalid ODCS contract: missing required field 'name'."
)
if not contract_version:
raise ValueError(
"Invalid ODCS contract: missing required field 'version'."
)
print(f"Validating ODCS contract: {contract_name}")
print(f"ODCS version: {contract['apiVersion']}")
print(f"Contract version: {contract_version}")
# This example uses one schema object
schema_objects = contract["schema"]
if len(schema_objects) != 1:
raise ValueError(
"This example supports exactly one ODCS schema object"
)
schema_object = schema_objects[0]
properties = schema_object["properties"]
expected = {
item["name"]: item
for item in properties
}
# Check columns
actual_columns = set(df.columns)
expected_columns = set(expected.keys())
missing = expected_columns - actual_columns
unexpected = actual_columns - expected_columns
if missing:
errors.append(f"Missing columns: {sorted(missing)}")
# Strict schema matching is a pipeline policy in this example
if unexpected:
errors.append(f"Unexpected columns: {sorted(unexpected)}")
if errors:
raise ValueError(
"Data contract schema validation failed:\n"
+ "\n".join(errors)
)
validated_df = df
# Validate required fields and datatypes
for name, definition in expected.items():
logical_type = definition["logicalType"]
required = definition.get("required", False)
if logical_type not in TYPE_MAPPING:
errors.append(
f"Unsupported logicalType '{logical_type}' "
f"for column {name}"
)
continue
source_col = col(name)
if required:
invalid = df.filter(
source_col.isNull() |
(trim(source_col) == "")
).limit(1).count()
if invalid:
errors.append(
f"Column {name} cannot be NULL or empty"
)
spark_type = TYPE_MAPPING[logical_type]
if logical_type != "string":
converted = source_col.cast(spark_type)
invalid = df.filter(
source_col.isNotNull() &
(trim(source_col) != "") &
converted.isNull()
).limit(1).count()
if invalid:
errors.append(
f"Invalid {logical_type} values in column {name}"
)
validated_df = validated_df.withColumn(
name,
converted
)
# Validate primary key
primary_keys = [
item["name"]
for item in properties
if item.get("primaryKey", False)
]
if not primary_keys:
errors.append("No primary key defined in ODCS contract")
else:
pk_invalid_condition = reduce(
lambda left, right: left | right,
[
col(name).isNull() | (trim(col(name)) == "")
for name in primary_keys
]
)
invalid_pk = validated_df.filter(
pk_invalid_condition
).limit(1).count()
if invalid_pk:
errors.append(
f"Primary key columns cannot be NULL or empty: "
f"{primary_keys}"
)
duplicates = (
validated_df
.groupBy(*primary_keys)
.count()
.filter(col("count") > 1)
.limit(1)
.count()
)
if duplicates:
errors.append(
f"Duplicate primary key values found: {primary_keys}"
)
if errors:
raise ValueError(
"Data contract validation failed:\n"
+ "\n".join(errors)
)
print(
f"ODCS contract {contract_name} "
f"version {contract_version} PASSED"
)
return validated_df
The validator now performs the following steps:
1. Reads the ODCS metadata.
2. Retrieves the columns from 'schema.properties'.
3. Checks for missing or unexpected columns.
4. Validates 'required: true' constraints.
5. Verifies data types via 'logicalType'.
6. Checks the primary key for null values and duplicates.
The strict check for additional columns is a deliberate design choice in our Python pipeline; these are distinct from ODCS fields.
processor.py does not need to change
At first I thought it was not necessary to change the processor.py. But after some debuging with AI I noticed that the merge didn't work well and I needed to replace the file.
from delta.tables import DeltaTable
from transfer.contract_validator import validate_dataframe
def transfer_data(spark, config, contract=None):
source = config["source"]
target = config["target"]
file_format = config["format"]
mode = config["mode"]
print(f"Reading source: {source}")
print(f"Target table: {target}")
print(f"Loading mode: {mode}")
# Read everything as strings first
df = (
spark.read
.format(file_format)
.option("header", "true")
.option("inferSchema", "false")
.load(source)
)
# Validate before writing
if contract is None:
raise ValueError(
"No data contract configured for this entity"
)
df = validate_dataframe(df, contract)
record_count = df.count()
print(f"Validated records: {record_count}")
# Apply loading strategy
if mode == "overwrite":
(
df.write
.format("delta")
.mode("overwrite")
.saveAsTable(target)
)
elif mode == "append":
(
df.write
.format("delta")
.mode("append")
.saveAsTable(target)
)
elif mode == "merge":
merge_key = config["merge_key"]
# Read primary keys from the ODCS contract
primary_keys = [
item["name"]
for item in contract["schema"][0]["properties"]
if item.get("primaryKey") is True
]
print(f"ODCS primary keys: {primary_keys}")
print(f"Configured merge key: {merge_key}")
if merge_key not in primary_keys:
raise ValueError(
f"Merge key {merge_key} is not defined "
f"as a contract primary key. Found: {primary_keys}"
)
if spark.catalog.tableExists(target):
delta_table = DeltaTable.forName(
spark, target
)
(
delta_table.alias("target")
.merge(
df.alias("source"),
f"target.{merge_key} = source.{merge_key}"
)
.whenMatchedUpdateAll()
.whenNotMatchedInsertAll()
.execute()
)
else:
(
df.write
.format("delta")
.mode("overwrite")
.saveAsTable(target)
)
else:
raise ValueError(
f"Unsupported loading mode: {mode}"
)
print(
f"Successfully processed {record_count} records"
)
The processor does not need to know whether the contract originates from JSON or YAML; that is the responsibility of 'main.py'.
8. Testing in Databricks
In your VS Code terminal, run:
databricks bundle validate -t dev
Then
databricks bundle deploy -t dev
and :
databricks bundle run transfer_job -t dev --params entity=customers
Resulting in :
and execute :
databricks bundle run transfer_job -t dev --params entity=customers_update
Resulting in :
Hennie








.png)









