Databricks Clean Room — where shared insights meet uncompromised privacy
A Data clean Room is a secure space that enables businesses to work together on sensitive data without exposing or compromising it. By using robust protocols and advanced technologies it allows multiple parties to combine and analyse information while ensuring strict adherence to privacy regulations and compliance requirements. Let’s consider a scenario where two organizations […]
Handling CDC in Databricks: Custom MERGE vs. DLT APPLY CHANGES
Change data capture (CDC) is crucial for keeping data lakes synchronized with source systems. Databricks supports CDC through two main approaches: Custom MERGE operation (Spark SQL or PySpark) Delta Live Tables (DLT) APPLY CHANGES, a declarative CDC API This blog explores both methods, their trade-offs, and demonstrates best practices for production-grade pipelines in Databricks. Custom […]
End-to-End Ingestion of 400+ MySQL Tables with Databricks Delta Live Tables
Ingesting and managing data from more than 400 MySQL tables on recurring schedules is a complex challenge. Traditional approaches often lead to pipelines that are difficult to scale, hard to maintain, and prone to failure when handling schema changes or scheduling dependencies. To address these challenges, we designed and implemented a configuration-driven ingestion framework using […]
Streaming Made Simple with Databricks Debezium
Introduction In today’s fast-paced data-driven world, real-time data processing and change data capture (CDC) are crucial for businesses to make timely and informed decisions. Databricks, a powerful cloud-based analytics platform, combined with Debezium, an open-source CDC tool, enables seamless real-time data replication and transformation. This blog will explore Databricks and Debezium, detailing their integration and […]
Liquid Clustering in Databricks: The Future of Delta Table Optimization
Introduction — The Big Shift in Delta Optimization In the ever-evolving world of big data, performance tuning is no longer optional – it’s essential. As datasets grow exponentially, so does the complexity of keeping them optimized for querying. Databricks’ Liquid Clustering is a groundbreaking approach to data organization within Delta tables. Unlike traditional static partitioning, […]
Turning Notebooks into Dashboards with Databricks
Why Databricks Notebook Dashboards Stand Out In the world of data-driven decision-making, dashboards are essential for turning raw numbers into actionable insights. While most dashboards help you visualize numbers, Databricks takes it a step further by making the process smooth, flexible, and tightly integrated with your working environment. Databricks notebook dashboards offer a unique blend […]
Triggering Azure Data Factory (ADF) Pipelines from Databricks Notebooks
Overview In modern data workflows, it’s common to combine the orchestration capabilities of Azure Data Factory (ADF) with the powerful data processing of Databricks. This blog demonstrates how to trigger an ADF pipeline directly from a Databricks notebook using REST API and Python. We’ll cover: Required configurations and widgets Azure AD authentication Pipeline trigger logic […]
Delta Sharing: Let’s Share Seamlessly
Data became valuable the moment we started generating it at scale. As organizations began storing it by region — each with its own compliance rules, protocols, and security boundaries — the challenge shifted to: how do we share and consume data across regions securely, efficiently, and with minimal friction? Enter Delta Sharing: a modern, open, and cost-effective way to […]
Battle of the Data Titans: Databricks vs Microsoft Fabric Notebooks
In this blog, we break down the key differences between Microsoft Fabric and Databricks notebooks— comparing their pricing, features, and capabilities — to help you choose the right platform for your business needs. In today’s world, data is the backbone of decision-making, innovation, and business growth. With the explosion of big data, companies need powerful […]
A Secure & Scalable Oracle Connection Strategy in Databricks Using OJDBC and Azure Key Vault
