What is Apache Iceberg?
To paraphrase the official project definition: Apache Iceberg is a high-performance format for huge analytic tables.
It is an open-source standard that brings SQL-like reliability and simplicity to big data, while making it possible for multiple compute engines (like Spark, Trino, Flink, and Dremio) to work with the exact same data simultaneously without stepping on each other's toes.
Crucially, Apache Iceberg is not a query engine (it doesn't process data) and it is not a file format (like Parquet or CSV). It is an Open Table Format—a metadata layer that sits between your query engines and your storage files, acting as an intelligent directory that tells engines exactly where data lives, how it is partitioned, and what version of the data they should read.
Why Iceberg Was Created: The Fall of the Hive Metastore
To understand why Iceberg is a massive breakthrough, we must understand the problem it solved. Before Iceberg, the Hadoop and big data ecosystem relied on the Apache Hive Metastore.
Hive defined a table as a physical directory in a file system, and partitions as subdirectories (e.g., `/data/year=2026/month=05/`). When you queried a Hive table, the engine had to perform a "directory listing" to find all the files inside the relevant folder. On cloud object storage like Amazon S3, directory listing is incredibly slow. As tables grew to millions of files, queries began to time out just trying to figure out which files to read.
Worse, the directory model made safe concurrent writes impossible. If a Spark job crashed halfway through writing data into a directory, half the files were there, and any query running at that moment would read partial, corrupted data.
Netflix engineers created Apache Iceberg to solve these exact problems. They realized that by tracking data at the file level instead of the directory level, they could eliminate directory listings entirely and provide true database-like transactions.
The Core Abstraction: File-Level Tracking
The core architectural difference between legacy systems and Apache Iceberg is File-Level Metadata Tracking. Iceberg maintains an explicit, hierarchical list of every single file that belongs to a table.
When an engine queries an Iceberg table, it doesn't look at folders. It reads Iceberg's metadata. If a Parquet file exists in the cloud storage bucket but is not explicitly listed in Iceberg's metadata, the query engine completely ignores it. This abstraction is the secret behind all of Iceberg's powerful capabilities.
graph TD
subgraph "The Iceberg Stack"
Engine[Query Engine: Dremio, Spark, Flink]
Cat[Iceberg Catalog]
Meta[Iceberg Metadata Tree]
Files[(Data Files: Parquet, ORC, Avro)]
end
Engine -->|1. Asks for current state| Cat
Cat -->|2. Returns pointer to| Meta
Engine -->|3. Reads index & filters| Meta
Meta -->|4. Points to specific| Files
Engine -->|5. Downloads ONLY needed| Files
style Engine fill:#fef08a,stroke:#ca8a04
style Cat fill:#dbeafe,stroke:#2563eb
style Meta fill:#e0f2fe,stroke:#0284c7
style Files fill:#dcfce7,stroke:#22c55e
Key Capabilities
1. ACID Transactions (Serializable Isolation)
Because Iceberg tracks explicit files, writing to a table is an atomic operation. A writer creates new data files in the background, but readers can't see them yet. Once the write is finished, the writer performs an atomic swap in the catalog to update the metadata pointer. The table state changes instantaneously. Readers either see the table before the write, or after the write—they never see a partial state.
2. Schema Evolution
In the Hive era, renaming a column or changing its type was a nightmare that often required rewriting petabytes of data. Iceberg solves this using unique ID tracking for columns. You can add, drop, rename, or reorder columns instantly as a simple metadata operation. If you drop a column, it just stops being read. If you rename a column, its underlying ID stays the same, so historical data is perfectly preserved.
3. Hidden Partitioning
Partitioning in Hive required the user to physically add partition columns to their data and explicitly query them (e.g., `WHERE event_date = '2026-05-15'`). If the user forgot, they would trigger a full table scan, costing thousands of dollars.
Iceberg introduces Hidden Partitioning. The partition logic is defined in the metadata, not the data. If you partition by `day(timestamp)`, users simply query `WHERE timestamp = '2026-05-15 10:00:00'`, and Iceberg automatically translates that under the hood to prune the correct daily partitions. Furthermore, you can change the partition strategy (e.g., from daily to hourly) on the fly without rewriting historical data!
4. Time Travel and Rollbacks
Because Iceberg creates a new "snapshot" every time the table is modified, it natively supports Time Travel. You can query the table exactly as it looked last Tuesday. If an ETL job writes bad data, you can instantly rollback the table to the previous snapshot with a single command.
Where Iceberg Fits in the Lakehouse
Apache Iceberg is the crucial middle layer of the Data Lakehouse architecture. It bridges the gap between raw, cheap cloud storage and powerful, distributed compute engines.
By providing a unified, open standard, Iceberg prevents vendor lock-in. Because the metadata format is open source, any tool can implement a reader or writer for it. You can ingest data with Flink, transform it with Spark, and serve dashboards with Dremio—all pointing at the exact same Iceberg table, with zero data movement or duplication.
Iceberg vs. Plain Parquet Folders
| Feature | Plain Parquet in Hive/S3 | Apache Iceberg |
|---|---|---|
| Safe Concurrent Writes | No (Data corruption risk) | Yes (ACID Guarantees) |
| Schema Evolution | Painful / Requires Rewrites | Instant / In-Place |
| Partition Evolution | Impossible | Yes (Hidden Partitioning) |
| Time Travel | No | Yes |
| Performance | Slow (Directory Listings) | Fast (Metadata File Pruning) |
When NOT to use Iceberg
Iceberg is incredibly powerful, but it is specifically designed for huge analytic tables. You should reconsider using Iceberg if:
- You have tiny datasets: If your entire database is 50MB, the overhead of Iceberg's metadata tree will actually make queries slower than just reading a raw CSV or SQLite file.
- You need high-frequency, single-row OLTP transactions: Iceberg is an OLAP (analytics) format. If you need to process 10,000 individual single-row updates per second (like a banking application), you need a traditional operational database like PostgreSQL or Cassandra, not a lakehouse.
Conclusion
Apache Iceberg has won the table format war because it solved the hardest problems of data lakes (performance at scale, data correctness, and schema management) through an elegant, file-level metadata abstraction.
By standardizing how compute engines interact with data lakes, Iceberg has made the unified Data Lakehouse a reality, allowing organizations to achieve data warehouse performance at cloud object storage prices.