Databricks series : Starting a new python project in Visual Studio Code (Part I)

Introduction

From local Python development to Databricks Jobs! In these series we w''ll start building  a Flexible Data Transfer Solution with Visual Studio Code and Databricks Asset Bundles.

Databricks offers extensive capabilities for processing data, but did you know that you don't have to develop all your Python code directly inside a Databricks Notebook?

You can also work locally with Visual Studio Code, structure your Python project, and deploy your code to Databricks. By using Databricks Asset Bundles (DAB), you can also manage your Databricks Job configurations as code.

In these series of blogposts, we will build a small end-to-end example. We will develop a local Python application that reads CSV files and writes the data to Delta tables in Databricks.

The interesting part is that we will use a single Python script to process multiple datasets. Which dataset gets processed is determined by parameters passed from a Databricks Job.

We will cover in these series the following  topics :

• Setting up Visual Studio Code (part I)
• Installing the Databricks Extension (part I)
• Creating a local Python project (part I)
• Preparing sample data 
• Developing the Python application
• Configuring Databricks Asset Bundles
• Deploying and running a Databricks Job

Lets first start with setting up the local environment :

• Visual Studio Code : Local development environment
• Python : Developing the application logic
• Databricks Extension : Integration between VS Code and Databricks
• Databricks CLI : Deployment and management
• Databricks Asset Bundles: Managing Jobs and resources as code

Visual Studio Code

The first part is installing Visual Studion Code. You can download the code from https://code.visualstudio.com/ and install visual studio code. It's quite straight forward!

Install python

Download Python from https://www.python.org/downloads/And also install the official Python extension from Microsoft in Visual Studio Code. 

Verify the installation using the terminal with python --version.

Installing the Databricks Extension

Open Visual Studio Code and navigate to Extensions (Ctrl + Shift + X). Search for the official Databricks extension and install it. The extension provides several capabilities, including:

• Connecting to a Databricks Workspace
• Creating Databricks projects.
• Managing Asset Bundles.
• Synchronizing and deploying Python code.
• Running and managing Databricks Jobs.

Nowadays you're also asked for installing the databricks AI tools :

Installing the Databricks CLI

We use the Databricks CLI to validate and deploy our Asset Bundle. Installation instructions:

https://docs.databricks.com/aws/en/dev-tools/cli/install

The instalation of Databricks CLI accroding to the instructions was not quite straight forward as you see below. Instead of upgrade I used install in the command :

Verify the installation of the Databricks CLI with Databricks version.

Final thoughts

This was a small blogpost about the installation of Databricks local development environment.

Hennie