Most of us interact with Apache Iceberg using three deceptively simple commands:
From the outside, this feels like working with a traditional database. Clean, Familiar, and Comfortable. But behind these three lines lies a sophisticated metadata-driven architecture that makes Iceberg one of the most powerful innovations in modern data engineering.
This blog unpacks what’s happening under the hood: the files, the pointers, the immutable commits, and the layered design that makes it all work reliably at scale.
The Problem – Why DataLakes were broken before Iceberg
Data lakes started as a simple idea: dump files on cheap object storage, query them with a compute engine. But as workloads grew, the cracks showed fast.
Before Iceberg: Data stored as raw files that had no transaction support, concurrent writes could silently corrupt data, updates and deletes required full partition rewrites, and schema changes were risky and manual.
Hive Tables were the first attempt at a solution. They introduced a table abstraction over files and folder-based partitioning. But Hive metadata was fragile, the Metastore became a bottleneck, partition listing was slow, and there was no concept of transactions.
As data volumes exploded into petabytes and teams needed concurrent writers, safe schema evolution, and point-in-time queries, a new class of table formats emerged: Apache Iceberg, Delta Lake, and Apache Hudi.
What is an Iceberg
Apache Iceberg is an open-source, high-performance table format for massive analytical datasets, designed for data lakehouses. It sits between object storage and compute engines. It is the glue that gives files database like behaviour without touching the files themselves.
Iceberg does not move or store your data. It manages a metadata layer that tells compute engines exactly which files belong to a table, in what state, and at what point in time. The actual records stay in Parquet, ORC, or Avro files on your object storage, untouched and immutable.
Why Apache Iceberg
Iceberg was built to:
- Support large-scale analytics
- Enable safe concurrent writes
- Simplify schema evolution
- Support updates, deletes, and time travel
- Work across multiple engines
Data Engines and Cloud Platforms
One of the key strengths of Apache Iceberg is its ability to integrate with multiple data engines and cloud platforms. Iceberg acts as a common table layer between storage and compute engines, allowing different tools to read and write the same table safely and consistently. This engine-agnostic design eliminates vendor lock-in, enabling organizations to build flexible lakehouse architectures.
- Apache Iceberg integrates with several popular data processing and analytics engines, including Apache Spark, Apache Flink, Trino, Presto, Dremio, Snowflake, and Kafka-based streaming ecosystems.
- These engines can perform operations such as querying, batch processing, streaming ingestion, updates, and deletes on the same Iceberg table while maintaining consistency through Iceberg’s metadata layer.
- On the storage side, Iceberg works seamlessly with major cloud platforms such as AWS, Microsoft Azure, Google Cloud Platform (GCP), and Alibaba Cloud. Since Iceberg stores data as open file formats like Parquet on object storage, it can operate across different cloud environments without changing the table format itself.
This interoperability makes Apache Iceberg highly suitable for modern lakehouse architectures, where multiple engines and cloud services need to collaborate on the same large-scale datasets efficiently and reliably.
Core Architecture
Iceberg’s entire design revolves around one principle: separate data from metadata. Every operation, insert, update, or delete creates a new metadata snapshot without ever modifying existing files.
Here’s the full chain from a write operation to a data file:
Each layer serves a specific purpose. No layer is redundant. Let’s break each one down.
metadata.json – Table metadata
The top-level control file. Contains the table schema, partition spec, table properties, and a pointer to the current snapshot. Iceberg never updates this file; it creates a new version on every commit.
snap-*.avro – Snapshot
Each write creates a new snapshot representing the full state of the table at that moment. It records the operation type (insert, overwrite, delete) and points to a manifest list.
*-m0.avro – Manifest list
The “index of indexes.” Lists all manifest files for a snapshot, along with file-level statistics and partition-level metadata. Used by the query planner for file pruning.
*.avro – Manifest file
Contains the actual data file references like paths, record counts, min/max column values, and partition values. This is how Iceberg knows which Parquet files belong to the table.
In small tables, Iceberg may embed manifest entries directly in the manifest list, skipping the separate manifest file layer. This means:
Snapshot → Manifest List → Data files (directly).
This is expected behaviour, not a missing component.
Data files
At the bottom of the chain are Parquet (or ORC/Avro) files, the actual rows. These are completely immutable. Iceberg never modifies a data file in place. Every insert or update produces a new file, and a new snapshot points to it.
Real-world Example
Let’s walk through a practical scenario. Suppose you’ve just created an Orders table in your Unity Catalog-enabled Databricks workspace and inserted a few records into it.
Create the table
You run a simple CREATE TABLE. To you, it looks like any SQL table. Behind the scenes, Databricks creates a metadata.json file in the external storage account and registers the table in Unity Catalog.
Insert your first batch of records
You push the first day’s order events. Iceberg writes immutable Parquet files to the external storage account and creates a new snapshot, which is a versioned commit with a full record of what changed.
Verify where the table actually lives
If you have to know where this data is physically stored and what format it is in, you run DESCRIBE DETAIL and scroll to the key fields.
What Iceberg is doing under the hood at each step
- CREATE TABLE – Iceberg writes a metadata.json to the ADLS path. No data files yet, just the schema, partition spec, and table properties. This is the root of the metadata chain.
- INSERT – Iceberg creates immutable Parquet files, then generates a new snap-*. avro snapshot, a manifest list, and manifest files pointing to those Parquet files. The old snapshot is preserved, and nothing is overwritten.
- Location – The abfss:// path is Azure Data Lake Storage Gen2. This is where all Iceberg metadata and data files physically live. It is managed and governed by Unity Catalog.
- MANAGED – Unity Catalog owns the data lifecycle. If you drop the table, the underlying ADLS files are deleted too. No orphaned files, no manual cleanup.
- Provider: Iceberg – Databricks is using Apache Iceberg as the open table format. This means any other engine, like Spark, Trino, or Flink, can read the same table directly from ADLS.
After you ran CREATE TABLE orders and inserted data, Databricks created three folders inside the table’s GUID path on ADLS. NX holds the actual Parquet data files, _iceberg is where all the Iceberg metadata lives, _delta_log is created by Databricks Unity Catalog for governance purposes, and it is not part of Iceberg, and Iceberg never reads from it. This three-folder structure is proof that Iceberg and Delta coexist at the storage level, but they operate independently.
Drilling into _iceberg/ reveals a single metadata folder. This confirms Iceberg’s design and everything Iceberg needs to understand the table (schema, snapshots, manifests) lives inside this one metadata directory. There are no data files here. Iceberg keeps metadata and data completely separate.
This is the most important folder. You can see four files: two *.gz.meta files which are the versioned metadata.json snapshots (one created at table creation, one after the insert), one *-m0.avro file which is the manifest list, and one snap-*.avro file which is the snapshot. This is the exact metadata chain in action
metadata.json → snapshot → manifest list — physically visible as real files on ADLS.
NX folder contains the actual Parquet data file a single part-00000-*.parquet file, This is where your inserted rows physically live. The file is immutable and Iceberg will never modify it. Any future updates or deletes will create new Parquet files alongside this one, and a new snapshot will point to the updated set.
How Iceberg Achieves Acid, Without a Database
Iceberg’s metadata-first design makes ACID transactions possible on plain object storage, which has no native locking or transaction primitives.
Atomic – A commit either succeeds fully or entirely. There is no partial state; metadata is swapped atomically.
Consistent – The table is always in a valid state. Readers always see a complete, consistent snapshot.
Isolated – Concurrent writes don’t interfere. Each job works on its own snapshot and commits independently.
Durable – Once a snapshot is committed, the metadata persists safely in object storage, as durable as the storage itself.
Schema & Partitioning
Safe schema evolution and intelligent partitioning
One of the most painful problems in traditional data lakes was schema evolution. Renaming a column in Hive could break every downstream query. Iceberg solves this elegantly.
Iceberg tracks columns by a stable column_id, not by position or name. You can add, rename, reorder, or drop columns; safely, existing data files are unaffected because they’re read by ID, not by position.
Partitioning is equally innovative. Unlike Hive’s folder-based partitioning, in which partition values are made into directory names and are impossible to change, Iceberg’s partitioning is logical and declarative. The partition spec can be time, and users never need to include partition columns in their queries;ries, the engine handles pruning automatically.
Time travel and versioning
Because every write produces a new immutable snapshot, querying any historical state of the table is straightforward:
— Query the table as it was yesterday
SELECT * FROM orders TIMESTAMP AS OF ’2024-01-14 00:00:00′;
— Query a specific snapshot by ID
SELECT * FROM orders VERSION AS OF 7392847293421;
— Roll back to a previous snapshot
CALL iceberg.system.rollback_to_snapshot( ‘db.orders’, 7392847293421 );
Performance
Why Iceberg queries are fast
Iceberg’s metadata chain isn’t just for corr;ctness, it’s a query acceleration system. Before a single data file is read, the query planner uses three levels of pruning:
Partition pruning
The manifest list contains partition-level metadata. Entire partitions are eliminated before any manifest file is opened.
File-level statistics
Each manifest entry stores min/max column values per file. Files whose ranges don’t overlap the query predicate are skipped entirely.
Row-group pruning
Within Parquet files, Iceberg leverages column statistics at the row-group level to skip blocks of rows without reading them.
Minimal I/O
In a well-organized table, a highly selective query might scan less than 1% of the physical and data; the metadata does the heavy lifting.
Limitations
When Iceberg is the wrong choice
Iceberg is powerful, but it adds overhead. The metadata layer has real costs; more files, more indirection, more complexity. There are scenarios where simpler is better:
- Very small datasets where metadata overhead outweighs the benefits
- Simple batch workloads with a single writer and no history requirements
- No need for updates, deletes, schema evolution, or time travel
- Teams without the engineering capacity to operate and maintain the catalog
Databricks note:
If you’re on Databricks, you’ll notice a _delta_log/ directory alongside _iceberg/metadata/. The Delta log is created for Unity Catalog governance; however, it is not used by Iceberg. Iceberg reads only from its own metadata directory.
Conclusion
Apache Iceberg is much more than just a table format for data lakes. It is a powerful metadata-driven architecture that brings database-like reliability, consistency, and scalability to modern analytical workloads while still preserving the flexibility of file-based storage systems.
By understanding how Iceberg manages metadata, snapshots, manifests, and immutable data files internally, we gain a deeper perspective on how modern lakehouse platforms are built to support large-scale analytics efficiently. Features such as ACID transactions, schema evolution, partition evolution, and time travel are not just high-level capabilities;bilities they are the result of a carefully designed layered metadata structure working behind the scenes.
Understanding these concepts not only helps in using Iceberg effectively but also provides valuable insight into the future direction of modern data architectures.
Iceberg vs Other Table Formats
Key Takeaways
What to remember
- Iceberg is a metadata layer, not a database or storage system. It manages tables on top of files without owning the files.
- The metadata chain: metadata.json → Snapshot → Manifest List → Manifest → Data files is the core design. Understanding this chain means understanding Iceberg.
- Immutability is the secret. No file is ever modified in place. Every write produces new files and a new snapshot, enabling ACID, time travel, and safe concurrent writes.
- Column IDs (not names or positions) enable safe schema evolution — one of the biggest improvements over Hive.
- Iceberg is engine-agnostic. The same table can be read and written by Spark, Flink, Trino, and others simultaneously — the metadata layer provides the coordination.
- Metadata pruning, partition pruning, and min/max statistics make Iceberg queries fast by design — not by accident.