Databricks series : Execute the job in Databricks (Part V)

Introduction

This blogpost describes the execution of the job in Databricks. In order to do that we need to do one thing and that is we need to place the files at the raw folder. Next is to execute job and as the final part we examine the results.

Previous blogposts : 

• Databricks series : Starting a new python project in Visual Studio Code (Part I) 
• Databricks series : setting up a connection to Databricks from local IDE (Part II)

Create a volume in databricks

First we need to create a folder structure at the location  :  /Volumes/main/demo/raw/

Open a notebook in Databricks and execute the following code :

%sql
CREATE CATALOG IF NOT EXISTS main;

CREATE SCHEMA IF NOT EXISTS main.demo;

CREATE VOLUME IF NOT EXISTS main.demo.raw;

This means : 

main                  ← Catalog

 └── demo              ← Schema

      └── raw          ← Volume

Upload the files to the location :

and here are the files stored at the locations. Everything is ready to execute the job.

Uitvoeren van de job

You can upload the files from our local data folder for this purpose.

Subsequently, we can run the same job twice.

entity = customers

source_path = /Volumes/main/demo/raw/customers.csv

target_table = main.demo.customers

entity = transactions

source_path  = /Volumes/main/demo/raw/transactions.csv

target_table = main.demo.transactions

The result should be: 

Job         Delta-table                 Records

Run 1 main.demo.customers 4

Run 2 main.demo.transactions 4

So, we have used one Python script and one Job for two different data streams.n.

Open de job and press on Run Now

The job is running now

Here you find information which code is executed in the job

More information is available in the Job execution window

And in the unity cata log we see the following information

Now execute the job with other parameters!

Enter the parameters and execute again.

And here information about the second run of the job

And this results in two tables with customer and transaction information.

Hennie