Databricks series : Deployment from Visual Studio Code (Part IV)

Introduction 

I've written a couple of blogposts about a small project to create a Databricks project. This blogpost is the fourth in the series. Now the fun part: we don’t have to manually create the Job in the Databricks interface. We’ll use the Databricks CLI for this!

Previous blogposts : 

• Databricks series : Starting a new python project in Visual Studio Code (Part I) 
• Databricks series : setting up a connection to Databricks from local IDE (Part II)
• Databricks series : Setting up a Databricks project in Visual Studio Code (part III)

Deploy the code

Open a terminal window in Visual Studio Code and authenticate with databricks auth login

 > databricks auth login --host https://dbc-xxxxxx-d21b.cloud.databricks.com/
 --profile DBprofile

Validate the bundle

Next step is validate the code with databricks bundle validate

> databricks bundle validate

results in : 

Name: DatabricksProject
Target: dev
Workspace:
  Host: https://dbc-xxxxxx-d21b.cloud.databricks.com
  User: xxxxxx@gmail.com
  Path: /Workspace/Users/xxxxx@gmail.com/.bundle/DatabricksProject/dev

Deploy the code

And deploy the code with databricks bundle deploy

> databricks bundle deploy

results in : 

Uploading bundle files to /Workspace/Users/xxxxr@gmail.com/.bundle/DatabricksProject/dev/files...
Created jobs.transfer_job
Files: 1 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged

Databricks now deploys:

• The Python code.
• The job configuration.
• The required project files.

Where is everything in Databricks?

In your Databricks Workspace, go to: Workspace → Users → your user → .bundle

There, you will typically find a structure such as:

Job-configuration

The Job itself is not necessarily found as a YAML file within the Workspace.

Go to: Workflows → Jobs & Pipelines

Search for: python-data-transfer

There, you can view:

• The Job configuration.
• The configured parameters.
• The Python Script Task.
• The execution history.
• The logs and results.

The YAML configuration remains part of your local DAB project in Visual Studio Code. Databricks uses that configuration to create or update the Job.

So that's it. The code is deployed to Databricks with DAB

Hennie

Databricks series : Setting up a Databricks project in Visual Studio Code (part III)

Introduction

In this blogpost I'll describe the Visual Studio Project in more detail. This blogpost is one in a series of blogposts. Here are the previous blogposts : 

• Databricks series : Starting a new python project in Visual Studio Code (Part I) 
• Databricks series : setting up a connection to Databricks from local IDE (Part II)

Project structure

Lets create a project structure where we can store our code and also the data files. I've created the following structure in Visual studio Code :

databricks-transfer/
│
├── databricks.yml
├── resources/
│   └── transfer_job.yml
│
├── src/
│   ├── __init__.py
│   └── main.py
│
├── data/
│   ├── customers.csv
│   └── transactions.csv
│
├── requirements.txt
└── README.md

What is the function of the different folders?

Some test data

For this blogseries, I've created a couple of testfiles with help of ChatGPT :

customers.csv

transactions.csv

Python code

The application performs three tasks:

1. Receives parameters from the Databricks Job.

2. Reads the specified CSV file using Spark.

3. Writes the data to a Delta table.

The advantage is that the Python code is independent of the specific dataset being processed.

import argparse
from pyspark.sql import SparkSession


def main():

    parser = argparse.ArgumentParser()

    parser.add_argument("--entity", required=True)
    parser.add_argument("--source-path", required=True)
    parser.add_argument("--target-table", required=True)

    args = parser.parse_args()

    spark = SparkSession.builder.getOrCreate()

    print(f"Processing entity: {args.entity}")

    df = (
        spark.read
        .option("header", "true")
        .option("inferSchema", "true")
        .csv(args.source_path)
    )

    (
        df.write
        .format("delta")
        .mode("overwrite")
        .saveAsTable(args.target_table)
    )

    print(f"Successfully processed {df.count()} records")


if __name__ == "__main__":
    main()

DAB

DAB will start with databricks.yml file. This is the content of the databricks.yml:

bundle:
  name: DatabricksProject

include:
  - resources/*.yml

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://dbc-xxxxxxx.cloud.databricks.com

This is an example of a serverless Python Job. The environment is linked to the task via environment_key. The Job has three parameters:

Create a transfer_job.yml in the resources folder :

resources:
  jobs:
    transfer_job:

      name: python-data-transfer

      parameters:
        - name: entity
          default: customers

        - name: source_path
          default: /Volumes/main/demo/raw/customers.csv

        - name: target_table
          default: main.demo.customers

      tasks:
        - task_key: transfer_data

          spark_python_task:
            python_file: ../src/main.py

            parameters:
              - --entity
              - "{{job.parameters.entity}}"
              - --source-path
              - "{{job.parameters.source_path}}"
              - --target-table
              - "{{job.parameters.target_table}}"

          environment_key: default

      environments:
        - environment_key: default
          spec:
            environment_version: "2"

The Job passes these parameters to main.py. Pay special attention to:"{{job.parameters.entity}}"

This is a dynamic reference to a Job parameter. Databricks populates this value when the Job is executed. This means we do not need to modify our Python code when we want to process a different dataset. It is also possible to perform an explicit sync action.

bundle:
  name: DatabricksProject

include:
  - resources/*.yml

sync:
  include:
    - src/**
    - requirements.txt
    
targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://dbc-xxxxxx-d21b.cloud.databricks.com

  ## Optionally, there could be 'staging' or 'prod' targets here.
  #
  # prod:
  #   workspace:
  #     host: https://dbc-xxxxxx-d21b.cloud.databricks.com

This allows you to specify which project files you want to synchronize. We deliberately exclude the `data/` directory: our CSV files are ultimately stored in a Unity Catalog Volume.

Hennie

Databricks series : setting up a connection to Databricks from local IDE (Part II)

Introduction

This is the second blogpost in a series about setting up an environment and perform a small project in Databricks. The first blogpost was about the installation of Databricks local IDE and this is the second blogpost in the series. In this blogpost we'll continue setting up the environment and make a connection to the workspace in Databricks.

This is the first blogpost in the series : Databricks series : Starting a new python project in Visual Studio Code (Part I)

These are the steps I've taken to setup the connection :

• Open the Databricks sidebar in Visual Studio Code and configure a new connection.
• Select your Databricks Workspace.
• Configure authentication, for example, using OAuth.
• Sign in through your browser.
• Select the desired compute resource.
• Setup the Python environment.

 

Create a configuration

First you need to setup a configuration in Visual Studio Code.

Fill the URL to the Databricks environment.

And select OAUTH (in my case)

And give the profile a proper name. I called it DBProfile.

Sign in with your account to Databricks.

A verification code is necessary to enter and if all is verified then you are authenticated.

And in Visual Studio Code you see the following configuration settings:

And there is more information about my Databricks Workspace there.

Next step is to select the compute. The option is to select Serverless and this is fine for now.

And I selected v5 as the version for the serverless compute.

Next there is the set up for the Python environment.

But I ran into an error. uv was not installed yet. 

I tried to install with Winget but that didn't succeed.

winget install --id=astral-sh.uv -e

But ran in an error because uv was not recognized as a valid command. 

I think it has something to do with the PATH variable in Windows. On Stackoverflow I found a script that I used to install in a terminal in Visual Studio Code: https://stackoverflow.com/questions/79474930/uv-command-not-found-in-visual-studio-code-on-windows

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

And in the output tab of Visual Studio it states now that the full intstallation of Python for Databricks  Connect is succeeded. 

Once configured, we now can work with our Databricks environment directly from Visual Studio Code.

Final thoughts

Quite some steps to setup the local environment for connecting to Databricks. Up to the next blogpost!

Hennie

Databricks series : Starting a new python project in Visual Studio Code (Part I)

Introduction

From local Python development to Databricks Jobs! In these series we w''ll start building  a Flexible Data Transfer Solution with Visual Studio Code and Databricks Asset Bundles.

Databricks offers extensive capabilities for processing data, but did you know that you don't have to develop all your Python code directly inside a Databricks Notebook?

You can also work locally with Visual Studio Code, structure your Python project, and deploy your code to Databricks. By using Databricks Asset Bundles (DAB), you can also manage your Databricks Job configurations as code.

In these series of blogposts, we will build a small end-to-end example. We will develop a local Python application that reads CSV files and writes the data to Delta tables in Databricks.

The interesting part is that we will use a single Python script to process multiple datasets. Which dataset gets processed is determined by parameters passed from a Databricks Job.

We will cover in these series the following  topics :

• Setting up Visual Studio Code (part I)
• Installing the Databricks Extension (part I)
• Creating a local Python project (part I)
• Preparing sample data 
• Developing the Python application
• Configuring Databricks Asset Bundles
• Deploying and running a Databricks Job

Lets first start with setting up the local environment :

• Visual Studio Code : Local development environment
• Python : Developing the application logic
• Databricks Extension : Integration between VS Code and Databricks
• Databricks CLI : Deployment and management
• Databricks Asset Bundles: Managing Jobs and resources as code

Visual Studio Code

The first part is installing Visual Studion Code. You can download the code from https://code.visualstudio.com/ and install visual studio code. It's quite straight forward!

Install python

Download Python from https://www.python.org/downloads/And also install the official Python extension from Microsoft in Visual Studio Code. 

Verify the installation using the terminal with python --version.

Installing the Databricks Extension

Open Visual Studio Code and navigate to Extensions (Ctrl + Shift + X). Search for the official Databricks extension and install it. The extension provides several capabilities, including:

• Connecting to a Databricks Workspace
• Creating Databricks projects.
• Managing Asset Bundles.
• Synchronizing and deploying Python code.
• Running and managing Databricks Jobs.

Nowadays you're also asked for installing the databricks AI tools :

Installing the Databricks CLI

We use the Databricks CLI to validate and deploy our Asset Bundle. Installation instructions:

https://docs.databricks.com/aws/en/dev-tools/cli/install

The instalation of Databricks CLI accroding to the instructions was not quite straight forward as you see below. Instead of upgrade I used install in the command :

Verify the installation of the Databricks CLI with Databricks version.

Final thoughts

This was a small blogpost about the installation of Databricks local development environment.

Hennie

Certified DBT Developer certification

Wow, I finally managed to pass the exam of DBT Developer certification. I failed two times first because I didn't had much of handson experience with DBT. That will change in the future. So, I had to learn everything from the documentation and from a lot of Udemy tests. Although the tests available on the internet doesn't completely cover the questions on the real exam of DBT, it helped a lot to understand DBT. I did that by researching the answers, googling and reading the documentation of DBT. Last but not least I practiced a bit with DBT on a small mock-up project that I've created. 

I also created a summary of all the things I learned in PowerPoint slides. I do this now for a couple of years now and I have an extensive list of PowerPoints created on subjects like DevOps, Snowflake, Microsoft, Azure, Data modelling, Datavault, Git, Scrum, etc and now DBT. I helps me to understand and organize the information.

Here is my list of powerpoints of DBT. 

Here are interesting courses and tests on Udemy

And I really enjoyed following this course on Udemy (also because it's for Snowflake).

Qanalabs also has a good list of questions (and explanantions) that helps.

Hennie