Tag: #python
-
Accelerating Change Data Capture with apply changes in Delta Live Tables (DLT): Simplifying SCD Type 1 & 2 Implementation
Introduction: Change Data Capture (CDC) is a crucial component of modern data engineering, enabling efficient tracking and processing of data changes from source systems. Traditionally, implementing CDC required complex and error-prone merge logic. With Delta Live Tables (DLT) in Databricks, CDC can now be implemented in a declarative, scalable, and reliable manner using the apply_changes…
-
A Unifying Tool For Deployment Of Databricks
Overview Databricks Asset Bundles are a way to develop, package, version, and deploy Databricks workspace artifacts (like notebooks, workflows, libraries, etc.) using YAML-based configuration files. This allows for CI/CD integration and reproducible deployments across environments (dev/test/prod). What are Databricks Asset Bundles Databricks Asset Bundles are an infrastructure-as-code (IaC) approach to managing your Databricks projects.…