In my previous blog posts—Ignite 3 Alpha: A Sneak Peek into the Future of Apache Ignite and Just Released: Apache Ignite 3, Alpha 2—I talked about the Apache Ignite community's journey toward Ignite 3, the next generation of the Apache Ignite database.
It's a tradition within the Ignite community to provide an Ignite 3 alpha release every few months. This tradition enables everyone interested in the Ignite 3 project to see the project in its current state, play with its new features, and get a feel for what is coming.
A few days ago, the most recent step in the Ignite 3 journey became available—Alpha 3. In this post, I’ll talk about Alpha 3’s new features, including a SQL engine that is based on Apache Calcite, a replication infrastructure that is based on RAFT, and LSM-tree storage.
SQL engine based on Apache Calcite
The SQL engine that is used by Ignite 2.x is based on the H2 database. The engine is powerful and feature-rich. However, because the engine is optimized for operational workloads, extra effort is required to support its analytical capabilities.
Apache Calcite to the rescue! Calcite is designed to act as a foundation for high-performance databases that are used for hybrid data processing (operational, analytical, and streaming). So, Calcite is a perfect fit for Ignite's architecture.
Calcite not only is more efficient than H2 in terms of CPU and memory consumption but also significantly widens the range of the SQL capabilities that you can support and the performance optimizations that you can apply. For example, distributed SQL queries with subselects are easy to do in Alpha 3!
Alpha 3 includes the first version of the Calcite-based SQL engine and of the JDBC driver that enables you to connect to a cluster and run queries. The release includes a code example that illustrates how the driver can be embedded into an application.
New unified client protocol
One of the goals of Ignite 3 is to unify the client APIs, to eliminate the need to deal with the ambiguity of thick and thin clients. To meet this goal, Ignite 3 will have a single binary client protocol that can be implemented in any programming language.
Alpha 3 features the first version of the protocol, along with implementations in Java and .NET. The release includes code examples for the Java client (.NET examples are coming soon).
LSM-tree storage based on RocksDB
For persisting data on disc, Ignite 2.x uses page-memory storage that is based on B+ tree. This implementation will be ported to Ignite 3 before the GA release.
However, by taking advantage of Ignite 3’s modular architecture, we can significantly increase flexibility. Persistent storage in Ignite 3 will be pluggable. This feature will allow developers to plug in multiple implementations based on their requirements.
For example, if your application is read-intensive, B+ tree is a great fit. However, for write-intensive workloads, LSM trees probably provide greater performance.
Alpha 3 implements the pluggable architecture for data storage; adds an LSM tree implementation that is based on RocksDB; and, on top, provides an additional in-memory layer.
Support for multiple engines enables users to mix and match types of storage, thus allowing different datasets to use different types of storage. This feature is a significant stepping stone toward a platform that is capable of processing hybrid workloads.
Raft-based data rebalancing
Raft-based replication infrastructure is the backbone of Ignite 3’s architecture. The infrastructure is (or will be) used by the components that are responsible for data synchronization, transactional processing, SQL query execution, computation runs, and so on.
Alpha 3 adds one more piece to this puzzle—data-rebalancing support that relies on the Raft snapshotting mechanism.
The rebalancing flow that Alpha 3 uses is illustrated in a code example.
What's next?
As usual, the best way to try the Alpha 3 build is to go through the Getting Started Guide and run the examples.
Please share your questions, concerns, and ideas in the dedicated mailing list thread.
Also, be sure to register for the upcoming virtual Ignite Community event on November 9. During the event, we will talk about the changes introduced in Alpha 3 and about plans for the future of the Ignite 3 project. Community members will have the opportunity to share their feedback and discuss proposed changes.
In the meantime, the Ignite 3 journey continues. The community is working on advancements in data-rebalancing mechanisms, baseline-topology management, the SQL engine, and the storage engine. Significantly improved transactional support is coming soon—via a new, fully-async API for explicit transactions.
Stay tuned!