QbeastTable.forPath(spark, tmpDir).
compact()
On high writing workloads, we usually end up appending short subsets of data to the table. This results in writing many small files on the Data Lake, which can cause performance issues. To address this, we implement the Compact operation, based on Delta Lake’s Optimize, that arranges the small files and compacts them into a single bigger file.