Data Engineering with Azure Databricks (English)

Description

In the training "Data Engineering with Azure Databricks" we will work with Databricks for two days to build a Data Lakehouse. The entire spectrum is covered here: architecture and design, Databricks setup, Implementation of transformations, orchestration of your tasks, version management and everything else that is needed for your Data Lakehouse. At the end of the training you will be able to independently set up a Data Lakehouse within Databricks.

Who is the Data Engineering with Azure Databricks training for?

This training is primarily aimed at Data Engineers and Data Warehouse developers or managers who have experience with Data Warehousing or other forms of (batch) data processing. Participants know that they want to learn more about Databricks and building a Data Lakehouse. Most participants already have some experience with cloud environments, but it is not mandatory: even if you are making the step from an on-premises Data Warehouse to a Data Lakehouse, this training fits in well.

Skills gained

After this training you:

• know what the architecture of a Data Lakehouse looks like and works
• understand the principles of Databricks, Data Lakehouses and Delta Lake
• can set up Databricks independently for a Data Lakehouse
• understand how Delta Lake Storage works and how it enables a Data Lakehouse
• can manage files in your Data Lake using Databricks
• can do orchestration within Databricks using jobs
• know which layers exist in a Data Lakehouse and how you can bring data from your practice here
• can do transformation and integration on the data in Databricks using PySpark and SparkSQL
• know how to deal with schedules and schedule evolution

Prerequisites to follow the Data Engineering with Azure Databricks training

To participate in this training it is important that you have prior knowledge of the following topics:

• Basic knowledge of SQL:
  • Query Concepts (SELECT, WHERE, GROUP BY, ORDER BY, LIMIT, and JOIN)
  • DDL (CREATE, ALTER, DROP of tables and databases)
  • DML (DELETE, INSERT, UPDATE, MERGE)
• Knowledge of data engineering in the cloud (VMs, storage accounts, AD accounts, etc.)
• Basic knowledge of Python (modules, reading data, simple operations)

Course outline

• Data Lakes and Data Warehouses
• Delta Lake
• ETL with Spark SQL
• Python and SparkSQL
• Incremental data processing with Structured Streaming and Auto Loader
• Data Lakehouse architecture: the "medallions" (gold, silver, bronze)
• Delta Live Tables
• Orchestration with Jobs
• Databricks SQL
• Rights management
• Bring dashboards and queries into production
• Dbfs: the DataBricks File System
• Manage your Data Lake from Databricks
• Transform data into a data lakehouse using PySpark
• Databricks tables: managed and unmanaged
• Hive Metastore
• Version management in Databricks
• Time travel
• Schedule enforcement
• Scheme evolution

Study material

In the training "Data Engineering with Azure Databricks" we work with official Databricks material. We make sure that you receive all the necessary material on time.