Introduction
From local Python development to Databricks Jobs! In these series we w''ll start building a Flexible Data Transfer Solution with Visual Studio Code and Databricks Asset Bundles.
Databricks offers extensive capabilities for processing data, but did you know that you don't have to develop all your Python code directly inside a Databricks Notebook?
You can also work locally with Visual Studio Code, structure your Python project, and deploy your code to Databricks. By using Databricks Asset Bundles (DAB), you can also manage your Databricks Job configurations as code.
In these series of blogposts, we will build a small end-to-end example. We will develop a local Python application that reads CSV files and writes the data to Delta tables in Databricks.
The interesting part is that we will use a single Python script to process multiple datasets. Which dataset gets processed is determined by parameters passed from a Databricks Job.
We will cover in these series the following topics :
- Setting up Visual Studio Code (part I)
- Installing the Databricks Extension (part I)
- Creating a local Python project (part I)
- Preparing sample data
- Developing the Python application
- Configuring Databricks Asset Bundles
- Deploying and running a Databricks Job
- Visual Studio Code : Local development environment
- Python : Developing the application logic
- Databricks Extension : Integration between VS Code and Databricks
- Databricks CLI : Deployment and management
- Databricks Asset Bundles: Managing Jobs and resources as code
Visual Studio Code
Install python
Installing the Databricks Extension
- Connecting to a Databricks Workspace
- Creating Databricks projects.
- Managing Asset Bundles.
- Synchronizing and deploying Python code.
- Running and managing Databricks Jobs.
Installing the Databricks CLI
