Clustering on the write path: Iceberg results

Introduction

Open table formats are growing in popularity because they prevent customer data from being locked in by a single vendor. With these formats, customers can use different engines to process their data while storing it in object storage, in a format that has broad support across open-source engines and vendors. Among the existing table formats, Apache Iceberg stands out for its wide adoption and growing momentum. Iceberg is an Apache Software Foundation project with a very active, thriving community, and the industry is largely backing it; e.g., AWS offers native Iceberg support through S3 Tables, and Snowflake enables access to Iceberg tables in external cloud storage.

Given the relevance to the industry, it is critical for us at Qbeast to provide first-class support for Iceberg. In the previous post on the performance of clustering in the write path with Qbeast, we explained the importance of keeping data efficiently organized as it lands and how Qbeast makes it possible with indexing and clustering techniques. We ran experiments with Delta Lake, measuring Qbeast performance against other clustering strategies and the algorithm's scalability.

This post measures the same performance aspects, but now for Iceberg. Same Indexing algorithm, same incremental clustering, but different metadata and commit protocol underneath.

Set Up

We ran all experiments on AWS with the following specifications:

• Iceberg for the table format, version 1.10.1
• Spark 4.0.1
• 4 cores and 25 GB of RAM for the Spark driver
• 4 executors, each with 4 cores and 10 GB of RAM (dynamic allocation disabled)
• On-demand m6gd.4xlarge EC2 instances with 950 GB of NVMe disk

The setup is the same as in the previous article, so we would avoid repeating information going forward. You can consult more details about the experiment conditions and datasets here.

Evaluation

Clustering Comparison

For this and the rest of the experiments, we measured the Spark executor’s runtime, which is the total time a worker node spends actively processing a task, including both CPU compute time and the time required to fetch shuffle data.

The plot compares four executions, each one from a different write technique:

• unclustered — Plain Iceberg: no clustering, no indexing
• qbeast — Qbeast, clustered directly on append (the default)
• qbeast-clusterAfterAppend — Qbeast, clustered in a separate step after append
• zorder— Iceberg with Z-order, using the same dimensions as Qbeast

Qbeast finishes in ~51% of Z-Order's time, writing data 1.9x faster than the Iceberg implementation. However, compared with unclustered execution, it induces a 56% overhead.

Index Tuning

There are two main parameters that shape the Qbeast Layout: the columns used for indexing and clustering, and the target row count per file. Depending on how a user configures them, the clustering creates narrow or large file min-max intervals, changing the final state of the layout. A natural question that arises with this configuration is how this selection affects write performance.

For Index Tuning experiments, we used the TPC-DS Store Sales table (SF 1000).

Tuning Row Count per File

We used different row-count targets per file (100K, 500K, 1M, and 5M) and measured the executor runtime for each experiment.

Runtime is 20% faster when we increase the row count from 100K to 1M rows per file, but only another 5.8% from 1M to 5M. A smaller target (50K) costs 17.5% more than 100K, confirming the same trade-off between tree depth and file count.

Tuning Clustering Columns

Using the same Store Sales table from TPC-DS SF1000, we varied the number of clustered columns while keeping the row count per file fixed at 1M.

The pattern holds for Iceberg: elapsed time rises about 18% from 1 to 8 clustering columns, close to the ~16% observed with Delta.

Scalability

We measured how clustering speed changes with data volume using the TPC-DS Store Sales Table. Because TPC-DS ships with different scale factors, we clustered the datasets ranging from SF10 to SF1000 and captured several metrics, including CPU usage, executor time, and total elapsed time.

Results with Fixed Cluster

With a fixed cluster size, throughput scales linearly for SF10-SF300, then saturates at ~1M rows/s on SF300-SF1000. It reflects CPU saturation: cluster CPU utilization sits at 85–91% from SF300 onward, while GC overhead stays under 1%. Following the same reasoning as the last post, we would test scaling the number of executors alongside data volume to prove the algorithm scales with the system.

Results with Scaling Executors

We scaled the number of executors in proportion to the data, holding the work per core roughly constant at ~70M rows/core (1 executor at SF100, 3 at SF300, 10 at SF1000).

The results show that the algorithm scales linearly with data volume as we increase the number of executors.

Conclusions

This post evaluated the performance of Qbeast on the write path for Iceberg, following the analysis from our previous post on Delta.

• The additional overhead of clustering on write remains low. Indexing with Qbeast completes in about half the time of Iceberg's Z-order (1.9x faster) and adds ~56% overhead compared with unclustered Iceberg.
• Tuning row count per file matters more than clustering columns. Adding clustering columns induces a low overhead (~18% from 1 to 8), consistent with Qbeast on Delta. The target row count per file is the parameter to choose carefully.
• The implementation scales with data volume. On a fixed cluster, throughput grows with data and then saturates once every core is busy (85–91% CPU utilization from SF300 onward). When we scale the cluster with the data, throughput grows near-linearly, reaching ~2.59M rows/s at 40 cores.

These results confirm that Qbeast's indexing behaves consistently across table formats: the same Indexing and incremental Clustering approach shows similar performance characteristics, making the algorithm agnostic to the different metadata and commit protocols.

‍