Hidden Pitfalls — Compaction and Partition Evolution in Apache Iceberg

Apache Iceberg offers partition evolution, allowing you to change how your data is partitioned over time without rewriting historical files. This is a major advantage over legacy file formats, but it also introduces new challenges—especially when it comes to compaction and query optimization.

In this post, we’ll explore how partition evolution can impact compaction, metadata management, and query performance—and how to avoid the most common pitfalls.

What Is Partition Evolution?

Partition evolution allows you to:

Add new partition fields
Drop old partition fields
Change partition transforms (e.g., from day(ts) to hour(ts))

Unlike traditional systems that enforce a single static layout, Iceberg lets you evolve the partitioning strategy without rewriting or invalidating historical data.

Example:

-- Original partitioning
ALTER TABLE sales ADD PARTITION FIELD day(order_date);

-- Later evolve to hourly
ALTER TABLE sales DROP PARTITION FIELD day(order_date);
ALTER TABLE sales ADD PARTITION FIELD hour(order_date);

Each snapshot will respect the partition spec that was active at the time the data was written.

The Pitfall: Compaction Across Partition Specs

When compaction jobs span files written under different partition specs, several challenges arise:

File Layout Inconsistency Compaction may combine files that don’t share a common layout, resulting in mixed partition values that reduce query pruning efficiency.
Reduced Predicate Pushdown Query engines rely on partition columns for efficient pruning. If files are mixed across specs, pruning may be incomplete, increasing scan cost.
Compaction Failures or Misbehavior Some engines may fail to rewrite or rewrite files improperly when specs conflict, especially in older versions of Iceberg libraries or poorly configured environments.

Best Practices to Manage Partition Evolution Safely

1. Compact Within Partition Spec Versions

Query the files metadata table to identify which files belong to which spec:

Copy
Edit
SELECT spec_id, COUNT(*) AS file_count
FROM my_table.files
GROUP BY spec_id;

Run compaction per spec_id to preserve consistency and avoid mixing files.

2. Track and Align Sorting and Clustering

When evolving partitions, ensure that sort orders are also updated. Mismatched sort and partition strategies can undermine clustering efforts.

SELECT spec_id, sort_order_id, COUNT(*) 
FROM my_table.files 
GROUP BY spec_id, sort_order_id;

3. Repartition Carefully and Gradually

Avoid abrupt changes like:

Switching from coarse to fine partitioning (e.g., day to minute)
Dropping too many partition fields at once
These can lead to over-fragmentation and more small files unless paired with compaction and sort order realignment.

4. Use Metadata Tables to Guide Evolution

Before evolving a partition spec:

Inspect query patterns (e.g., WHERE clauses)
Evaluate partition sizes and access frequencies
Use tools like Dremio’s catalog lineage and query analyzer if available

5. Communicate Changes Across Teams

If your tables are used across multiple teams or tools:

Document changes to partitioning logic
Include schema and partition spec history in data documentation
Coordinate compaction jobs after major partition changes

Summary

Partition evolution is one of Iceberg’s superpowers—but like all powerful features, it must be used wisely. To avoid performance and optimization issues:

Don’t mix files with different partition specs in compaction jobs
Update sort orders and clustering with partition changes
Monitor partition usage and access patterns continuously

In the next post, we’ll move from structural design to execution tuning—exploring how to scale compaction operations efficiently using parallelism, checkpointing, and fault tolerance.