Visual Data Prep for the Modern Lakehouse
What If Building Data Pipelines Required Little to No Code?
Building reliable data pipelines is one of the most critical responsibilities of modern data teams. Whether the goal is analytics, reporting, machine learning, or AI, data must first be ingested, transformed, validated, and delivered — and traditionally, that means writing SQL, PySpark, or ETL code.
Databricks Lakeflow Designer changes that. It introduces a low-code, visual development experience that lets you design, transform, and orchestrate data pipelines through an intuitive drag-and-drop canvas — while keeping all the scalability, governance, and performance of the Databricks Lakehouse Platform intact.
What Is Lakeflow Designer?
Lakeflow Designer is a visual, no-code, AI-native data preparation tool built directly into the Databricks Data Intelligence Platform. It provides a drag-and-drop canvas where analysts, domain experts, and less technical users can build data transformation workflows without writing a single line of SQL or Python.
In Designer, you create Visual data preps — pipelines made up of a sequence of operators (Filter, Join, Aggregate, Transform, and more) arranged as a Directed Acyclic Graph (DAG). Critically, every visual transformation is backed by real, production-ready code under the hood so workflows can be version-controlled in Git, scheduled as recurring jobs, and integrated into larger engineering pipelines without being rebuilt from scratch.
With Lakeflow Designer, you can:
- Build transformation pipelines on a visual drag-and-drop canvas
- Transform data without code using built-in operators for filtering, aggregating, joining, and reshaping
- Use Genie Code to generate or refine transformations using natural language
- Apply built-in AI functions for sentiment analysis, classification, text extraction, summarization, and more
- Write results directly to Unity Catalog tables
- Schedule pipelines as production jobs and version them in Git
Why Lakeflow Designer? The Problem It Solves
The Traditional Bottleneck
In most data organizations, a stubborn divide exists between the people who understand the data and the people who can build pipelines. A business analyst identifies a need, documents requirements, hands off to a data engineer, and waits — only to find issues and repeat the cycle. It’s slow, expensive, and frustrating for both sides.
Why Existing Self-Service Tools Fall Short
Tools like Alteryx and Tableau Prep have tried to address this, but they bring their own problems:
- Governance gaps — Data moves outside the central platform, breaking audit trails and forcing IT to manage another system.
- Generic AI — No awareness of your organization’s schema, lineage, or business context.
- Productionization friction — Visual logic gets trapped in proprietary formats; engineers must rewrite it to go live.
- Seat-based licensing — Per-user costs limit who gets access before adoption even begins.
How Lakeflow Designer Is Different
Lakeflow Designer is built on Databricks, not bolted alongside it. This changes everything:
- Native on Databricks — Transformations run where data already lives. No data movement, no governance gaps.
- AI with real data context — Genie Code knows your schema, lineage, and business metadata — not just generic completions.
- No translation to production — Designer outputs are backed by real Python code. Nothing needs to be rewritten.
- No per-user licensing — Available to all users on Databricks workspaces without additional seat costs.
Getting Started with Lakeflow Designer
Prerequisites
- A Databricks workspace with Unity Catalog enabled
- CAN USE permission on at least one compute resource (serverless or all-purpose)
- Databricks AI Assistive features enabled (required for Genie Code)
Lakeflow Designer is enabled by default for all Free Edition, Premium, and Enterprise tier workspaces. If you do not see the Visual data prep option after clicking + New, contact your workspace administrator.
Building Your First Pipeline
Click + New in the left sidebar and select Visual data prep. From there, the workflow is straightforward:
Step 1 — Add a Source operator
Browse or search your Unity Catalog tables, drag in a local CSV/Excel file, or connect via Google Drive or SharePoint. Your data lands on the canvas immediately.
Step 2 — Build the transformation pipeline
Open the operator panel on the left and add operators — Filter, Join, Aggregate, Transform, and more. Click any operator to configure it, or simply describe what you want to Genie Code and let it configure the operator for you.
Step 3 — Preview at each step
Click any operator to see its output in the bottom pane — instantly, on a sample of up to 1,000 rows. Use the Data Profile to check null counts, distinct values, and distributions before moving on.
Step 4 — Write results and publish
Add an Output operator, configure the target Unity Catalog table, and click Run. Then schedule the pipeline directly from Designer or add it as a task in a Lakeflow Job for full production orchestration.
Genie Code: AI-Assisted Transformation
Genie Code is the AI assistant embedded directly in the Designer canvas. Type a prompt in the input bar at the bottom of the canvas, and Genie Code autonomously creates, configures, or modifies operators to match your description.
Because it has access to your data’s schema, column descriptions, lineage, and business context from the Databricks Data Intelligence Platform, suggestions are calibrated specifically to your data — not generic language model completions.
Example prompts you can use:
- “Add a filter to keep only rows where region is North America”
- “Change the join to a left join and add a null check on the key column”
- “Create a column for profit margin as revenue minus cost divided by revenue”
- “Write SQL to rank customers by total spend within each region”
Operators at a Glance
Lakeflow Designer ships with a comprehensive set of built-in operators organized by category. The operator menu supports both category browsing and natural language intent search — typing “average by month” surfaces the Aggregate operator without needing to know its exact name.
Transformation Operators
- Filter — Keep only rows matching one or more conditions using a graphical condition builder.
- Join — Combine two tables on matching column values. Supports Inner, Left, Right, and Full join types.
- Aggregate — Group rows and compute summary values (SUM, AVG, COUNT, MIN, MAX, MEDIAN, and more).
- Transform — Select, rename, reorder, or create columns. The custom column editor works both in natural language and in code.
- Pivot / Unpivot — Reshape data from rows to columns or columns to rows.
- SQL & Python — Write custom logic for anything not covered by built-in operators.
AI Function Operators
Apply AI-powered enrichment to any row of data — without leaving the canvas or writing model invocation code:
- ai_analyze_sentiment — Returns positive, negative, or neutral labels for text columns.
- ai_classify — Classifies text using custom labels you define.
- ai_extract — Extracts structured fields (names, dates, amounts) from unstructured text.
- ai_mask — Masks PII fields such as names, emails, and phone numbers.
- ai_summarize — Generates concise summaries of long-form text.
- ai_translate — Translates text to a target language of your choice.
From Canvas to Production
Scheduling
Click the Schedule button in the Designer toolbar to create a recurring job for your pipeline. You can configure run frequency, compute resource, and parameter overrides per schedule — for example, one schedule pointing to a test catalog and another pointing to production, using the same pipeline.
Alternatively, add the Visual data prep as a task inside an existing Lakeflow Job to combine it with notebooks, SQL queries, dbt models, and Python scripts in a larger orchestrated workflow.
Git Integration
Visual data prep files are stored as standard notebooks with the extension .designer.ipynb, which means they live natively in Git folders. Commit, push, branch, and review pipeline changes exactly as you would any other code — enabling proper environment promotion from dev to staging to production.
How Lakeflow Designer Fits the Databricks Platform
Lakeflow Designer is not a standalone product — it is one layer of a unified platform. Every piece connects:
- Unity Catalog — All source data, output tables, and lineage flow through Unity Catalog automatically. Fine-grained access control, data lineage tracking, and audit logs apply to everything Designer reads and writes.
- Lakeflow Jobs — Visual data preps deploy as tasks in the same orchestration engine used for production ETL and ML pipelines. No separate scheduling layer required.
- Lakeflow Connect — For sources not natively available in Designer (Salesforce, Workday, HubSpot, relational databases, cloud storage), Lakeflow Connect ingests data into Unity Catalog, making it immediately usable in Designer.
- Genie (BI) — After preparing data in Designer, analysts can use Genie to ask questions in natural language, build dashboards, and create visualizations — all within the same platform.
Lakeflow Designer vs. Traditional No-Code Tools
|
Capability |
Traditional No-Code Tools |
Lakeflow Designer |
|
Where data lives |
Copied to the tool’s environment |
Stays in Databricks (in-place) |
|
Governance |
Tool-specific, often manual |
Automatic via Unity Catalog |
|
AI assistance |
Generic (no data context) |
Genie Code with schema, lineage & business context |
|
Promotion to production |
Requires rewriting logic in code |
Generated code is already production-ready |
|
Version control |
Often proprietary formats |
Native Git via .designer.ipynb |
|
Licensing model |
Per-user seat licensing |
Included in Databricks workspace |
|
AI enrichment |
Limited or add-on |
Built-in AI Function operators |
Real-World Use Cases
Sales Reporting Pipeline
A sales analyst joins CRM opportunity data with invoicing data, filters to closed-won opportunities, aggregates revenue by region and quarter, and writes the result to a reporting table — scheduled to refresh nightly without any engineering involvement.
Customer Feedback Analysis
A product team uses the AI Function operators (ai_analyze_sentiment and ai_classify) to enrich support tickets with sentiment labels and topic categories. The enriched dataset is surfaced in a Genie dashboard for weekly review.
Data Quality and Cleansing
A governance team standardizes a raw ingested table in four operator steps: fix grammar in free-text fields with ai_fix_grammar, mask PII with ai_mask, deduplicate on a primary key using a Filter, and enforce a batch row limit with the Limit operator.
Self-Service Enrichment for Domain Teams
A data engineering team packages a proprietary scoring model as a User-Defined Operator. Domain analysts in the supply chain team can now apply the scoring as a single operator click — without understanding the underlying model code.
Conclusion
Lakeflow Designer addresses one of the most persistent challenges in data organizations: giving non-technical users the ability to build reliable, governed, production-ready data pipelines without creating a parallel, ungoverned data ecosystem.
Its core strengths at a glance:
- Visual + code-backed — The drag-and-drop interface generates real Python code, so nothing is lost when moving to production.
- AI-native — Genie Code and AI Function operators put AI where it is most useful, with full awareness of your data context.
- Deeply integrated — Unity Catalog, Lakeflow Jobs, Git, and Genie BI work together with no translation or handoff.
- Accessible by design — Enabled by default across all workspace tiers, with no per-user seat cost.
If your team is still translating analyst requirements into engineering tickets just to move data, Lakeflow Designer is worth a look.
