Deep Copy vs Shallow Copy in Databricks Delta Lake

Deep Copy vs Shallow Copy in Databricks Delta Lake

Deep Copy vs Shallow Copy in Databricks Delta Lake

When working with large-scale data in Databricks Delta Lake, it’s common to create copies of tables for testing, development, or archival purposes. However, not all copies are created equal. In Delta Lake, shallow copy and deep copy serve different purposes and have very different behaviors — both in terms of performance and data isolation.

In this post, we’ll break down:

  • What shallow and deep copies mean in Delta Lake
  • How to create them
  • When to use each
  • Common pitfalls

What is a Shallow Copy?

A shallow copy of a Delta table creates new Delta table metadata, but the data files are referenced, not duplicated. Think of it as a symbolic link to the original data — it’s fast, space-efficient, but not independent.

  • Only metadata is copied — data files are shared with the source table
  • Very fast to create (almost instant)
  • Minimal storage overhead
  • Both the source and the shallow copy point to the same physical data files
  • Ideal for:
    1. Read-only data exploration
    2. Schema evolution testing
    3. Quick development or experimentation
  • Not safe for destructive changes — changes in one table affect the other

Creating a Shallow copy

image 1 - delta lake
image 2 - delta lake
image 3 - delta lake

Or, with a specific version:

image 4 - delta lake
image 5 - delta lake
image 6 - delta lake
image 7 - delta lake

What is a Deep Copy?

A deep copy creates a complete and independent copy of both metadata and data files. It’s a true, physical duplication.

  • Metadata and data files are fully copied
  • Results in a completely independent copy
  • Slower to create (due to physical data duplication)
  • High storage usage
  • Provides full data isolation
  • Ideal for:
    1. Backup and recovery
    2. Sandboxed or destructive testing
    3. Migrating data across environments
  • Safe for modifying data — does not affect the source table

How to Create a Deep Copy:

image 9 - delta lake
image 10 - delta lake
image 12 - delta lake
Note: Deep clone is only available in Unity Catalog or with external locations in certain editions (e.g., Databricks Enterprise).

Conclusion:

Modifying a Shallow Copy:– If you delete or overwrite data in a shallow copy, it may impact the original table unless you create a deep copy first.

Misusing in Production:– Shallow copies are great for quick duplication, but using them in workflows that modify data or depend on isolation can cause unwanted side effects.

Real-World Use Cases

Use Shallow Copy When:

  • You need a fast copy for a quick query test
  • You want to explore the schema at a specific version
  • You’re developing read-only dashboards

Use Deep Copy When:

  • You’re archiving data
  • You want to test destructive transformations safely
  • You’re moving data across environments (e.g., dev → prod)

-Meka Mallikharjuna Reddy
Data Engineer