Smart Data Layout:
A new standard
for managed tables

We all like seeing complex problems being solved by simple and elegant solutions, I can’t verbalize what it is, but the good feeling that comes with it is undeniable.

At Qbeast, we want to tackle some of the difficulties in the field of Big Data, and we want to do it by introducing a powerful and open source data format that enables fast approximate and full precision analytics.

Sounds good? Before going into the details, let’s take a look at the problem first.

Creating and leveraging indexes is an essential feature in DBMS and data warehouses to improve query speed. When applying filters in a SQL query, the index is first consulted to locate the target data before any read. In this way, we avoid reading the entire table from the storage, reducing the data transfer involved.

For a query like the following, if the sales_table is indexed on the year column, we can directly access the data from 2020 and onwards, avoiding sequential scan on the entire table. Search algorithms such as Binary Search can be leveraged to achieve sub-linear time complexity by sorting the table by year.

Writing procedural programs to analyze data often comes less handy than a declarative approach, for one has to define the exact control flow of the program rather than simply stating the desired outcome. For example, it takes a lot more to write a simple groupby using Python than with SQL.

Other than this apparent inconvenience, there’s also a potential performance drop-off, given that procedural codes are treated as black boxes when it comes to automatic code optimization. On the other hand, the structure of SQL queries are transparent to the underlying query optimizer, and both rule and cost-based optimizations can be applied to improve performance.

One common problem that we heard about is the scalability of ML models. Even if you are using simple ML models like support vector machines, linear regression, or neural networks, when you use them at large scale they can be very slow. Luckily for us, we can use sampling as a trick to increase speed without compromising accuracy. This is what the Qbeast Format can help us with, and we’ll see how to take advantage of its sampling capability.

One of the uses of machine learning algorithms is detecting fraudulent transactions by feeding a model with historical transaction data.

Now that AI is gaining adoption in our day-to-day life, we need a data layout that enables the training of future MLs at scale.

With gradient-based data navigation that reduces time and cost, we can take full advantage of the distribution and enhance training on big chunks of data that sits on any Data Lake.

Contact us to build the next generation of AI or read more about our use cases.

Now that AI is gaining adoption in our day-to-day life, we need a data layout that enables the training of future MLs at scale.

With gradient-based data navigation that reduces time and cost, we can take full advantage of the distribution and enhance training on big chunks of data that sits on any Data Lake.

Contact us to build the next generation of AI or read more about our use cases.