Category: Databricks
-
Turning Notebooks into Dashboards with Databricks
Why Databricks Notebook Dashboards Stand Out In the world of data-driven decision-making, dashboards are essential for turning raw numbers into actionable insights. While most dashboards help you visualize numbers, Databricks takes it a step further by making the process smooth, flexible, and tightly integrated with your working environment. Databricks notebook dashboards offer a unique blend…
-
Triggering Azure Data Factory (ADF) Pipelines from Databricks Notebooks
Overview In modern data workflows, it’s common to combine the orchestration capabilities of Azure Data Factory (ADF) with the powerful data processing of Databricks. This blog demonstrates how to trigger an ADF pipeline directly from a Databricks notebook using REST API and Python. We’ll cover: Required configurations and widgets Azure AD authentication Pipeline trigger logic …
-
Delta Sharing: Let’s Share Seamlessly
Data became valuable the moment we started generating it at scale. As organizations began storing it by region — each with its own compliance rules, protocols, and security boundaries — the challenge shifted to: how do we share and consume data across regions securely, efficiently, and with minimal friction? Enter Delta Sharing: a modern, open, and cost-effective way to…
-
Battle of the Data Titans: Databricks vs Microsoft Fabric Notebooks
In this blog, we break down the key differences between Microsoft Fabric and Databricks notebooks— comparing their pricing, features, and capabilities — to help you choose the right platform for your business needs. In today’s world, data is the backbone of decision-making, innovation, and business growth. With the explosion of big data, companies need powerful…
-
-
Delta Lake Speed-Up: Z-Order on Single vs. Multiple Columns
Introduction As organizations ingest massive volumes of data into Delta Lake, query performance becomes critical, especially for dashboards, ad-hoc analysis, and downstream ETL jobs. One powerful technique to reduce query latency and improve data skipping is Z-Order Optimization. In this article, let’s cover: What Z-Ordering is How to apply it to single vs. multiple columns…
-
Accelerating Change Data Capture with apply changes in Delta Live Tables (DLT): Simplifying SCD Type 1 & 2 Implementation
Introduction: Change Data Capture (CDC) is a crucial component of modern data engineering, enabling efficient tracking and processing of data changes from source systems. Traditionally, implementing CDC required complex and error-prone merge logic. With Delta Live Tables (DLT) in Databricks, CDC can now be implemented in a declarative, scalable, and reliable manner using the apply_changes…
-
A Unifying Tool For Deployment Of Databricks
Overview Databricks Asset Bundles are a way to develop, package, version, and deploy Databricks workspace artifacts (like notebooks, workflows, libraries, etc.) using YAML-based configuration files. This allows for CI/CD integration and reproducible deployments across environments (dev/test/prod). What are Databricks Asset Bundles Databricks Asset Bundles are an infrastructure-as-code (IaC) approach to managing your Databricks projects.…
-
Accelerating Incremental Data Ingestion with Databricks Auto Loader and Delta Live Tables
Introduction: In today’s data-driven world, enterprises handle massive amounts of continuously arriving data from various sources. Traditional batch ETL jobs, while effective, often lead to inefficiencies, delays, and operational overhead. Databricks Auto Loader and Delta Live Tables (DLT) provide a powerful solution for incremental data ingestion and pipeline automation. Auto Loader simplifies real-time and batch…
-
Governance on AI Agents: Ensuring Control, Transparency, and Safety
What happens when an AI agent makes a decision that negatively impacts your business? According to a recent MIT study, 68% of organisations deploying autonomous AI agents report experiencing at least one significant incident related to misaligned agent behaviour in their first year of deployment. As AI agents become more capable and widespread, the need…