Held on November 9, Ignite Summit Europe 2022 provided a closeup look into how Apache Ignite is being adapted for use in a variety of dynamic, demanding business settings and use cases. (Video of all sessions is available on-demand here.) Presenters were from organizations as diverse as a financial software tools, the largest food delivery company in Latin America, and the largest particle accelerator laboratory in the world.
Organized by GridGain and the Apache Ignite community, Ignite Summit attracts Igniters interested in learning and sharing their experiences about Apache Ignite. Ignite Summit Europe 2022 included a virtual “Ask an Ignite Expert” booth with Ignite committers and GridGain engineers available to answer questions about Ignite and the Apache Ignite Community.
Promoting Proven Use Cases vs. a Swiss Army Knife of Features
Nikita Ivanov, GridGain founder and CTO, delivered the summit keynote, titled “Is Apache Ignite a Swiss Army Knife? Not really….” Since 2016, Apache Ignite has grown with an array of added features, including a tiered memory architecture, machine learning, streaming and continuous queries, and the use of Ignite as a service grid. But Ivanov sees Ignite less as a collection of features than a world-class solution for three main use cases: using Ignite for data caching and acceleration, as a data integration hub, and for co-located computations.
For caching and acceleration, “Ignite is the best framework for runtime in existence today,” said Ivanov. “99% of application latency is due to accessing external network and storage resources. Optimizing caching, despite how trivial and mundane it may sound, is still one of the best ways to bring top-level acceleration to your application. Apache Ignite has the best feature set for that use case.”
The second use case championed by Ignite is the data integration hub where “you put an Ignite cluster in front of the data sources with unified, high-performance API access.” To Ivanov “Ignite is the perfect middleware to construct a hub. I think this use case will see the most growth in the coming years.”
Finally, Ivanov talked about collocated computations as a client-server architecture in reverse that provides highly scalable, high-performance computing. “You move computations to where the data is located in nodes in the cluster. It minimizes unnecessary data movement. Ignite has the number one industry implementation of this idea.”
The European Organization for Nuclear Research Deploys Ignite
Tiago Oliveira, DevOps software engineer on the Accelerator Controls team at CERN, the European Organization for Nuclear Research, with the largest particle physics laboratory in the world, reported on the Swiss organization’s adoption of Apache Ignite for the CERN Control and Monitoring Platform (C2MON). Highly-availability and fault tolerant, C2MON handles data acquisition, monitoring, and control of computing and industrial systems throughout the organization via 200,000 endpoints that generate up to two million messages a day.
Using Apache Ignite as a data acquisition layer and standard APIs, Oliveira’s team deployed two Ignite clusters to ensure high performance with low latency, redundancy for rolling upgrades, smooth failover, and zero downtime. Most production software at CERN is run on bare metal. The Ignite clusters can also be run on Docker containers, which CERN IT has found useful for integration testing. "With C2MON, scalability, consistency of data, and rolling upgrades are crucial," said Oliveira. "And the integration with Apache Ignited helped us achieve those goals." He added that the fine-grained configurations provided by Ignite make it easy to optimize and finetune application behavior.
A Look at Apache Ignite 3.0
Stanislav Lukyanov, GridGain Technical Director and long-time Apache Ignite enthusiast, gave an overview of the new features and changes coming in Apache Ignite 3.0, a major departure from Ignite 2.0. Legacy APIs and CLIs have been modernized or removed. Some static, cumbersome approaches to configuration have been improved. Certain protocols have been replaced with industry-standard libraries that have better features (like SQL transactions and split-brain protection out of the box).
Lukyanov described key changes to Ignite 3.0 in three categories: architecture, APIs, and installation and management. He urged developers to start exploring and playing with Ignite 3.0 Beta 1. It's important to note that Ignite 3.0 Beta 1 is not a production ready release and works best on a stable topology. Join the discussion about Ignite 3.0 Beta 1 via the community dev list.
GridGain architect Alexey Scherbakov showed how the newer architecture enhances transactions for different types of workloads (e.g., read-write, write-only, read-only). With more modern, distributed protocols in Ignite 3.0, a global partition map exchange is no longer needed because each replication group works independently and a node failure affects only a small subset of data. Split-brain protection is available out of the box. A new transaction protocol was designed.
Software engineer Mikhail Pochatkin of Unison Technologies, described the distribution model available in Ignite 3.0 Beta 1. He showed how to transition to the new model and how to use the new distribution methods to install an Ignite cluster.
Apache Ignite uses a new build system, Gradle. The assembly of new distributions are implemented only in Gradle, a more modern solution than Maven. Gradle simplifies the writing of clear, reusable code and to avoid the use of legacy code from Apache Ignite 2. With Ignite 2, to install a new node requires installation of the Ignite CLI and to execute some commas. “It can’t be used in a production environment because no one wants unneeded tools in a production environment,” said Pochatkin, listing several other drawbacks. The new distribution model is used from within the Ignite service for a much more streamlined distribution and installation experience.
Ignite SQL Workshop
Ian Ruffell, Senior Solutions Architect at GridGain, provided an overview of SQL development, tuning, and troubleshooting in Ignite. He included a look at related diagnostic tools, analysis examples, a SQL typical traps and optimizations troubleshooting checklist, and a look at what's coming in SQL in Apache Ignite 3.0. “This should improve the way queries are executed in Ignite and GridGain,” said Ruffell, who included more information on SQL Performance Tuning.
Portfolio Management Gets Ignite Retrofit at Bloomberg
Kinjal Patel, Senior Software Engineer at Bloomberg, works on an asset investment and management (AIM) system that helps institutional clients who want to maximize their returns to manage their trading workflows (e.g., comparing returns to global indexes). Patel’s team created an application to provide traders with data from the trading workflows. They used Apache Ignite and the SQL engine to store persistent data.
Effective portfolio management requires high data availability and consistency, live updates, valid data sets, data persistence in a transactional form, and the ability to analyze data across Bloomberg's accounts. The prior architecture had several drawbacks. Scalability was expensive because it had a high memory footprint (15-20 Gigs). Through the use of an in-memory cache with data persisted on disk, data could be unavailable upon application restart or cache refresh. And there was no way to recover the state if the system went down; the in-memory store had to be built from scratch.
Apache Ignite provides the AIM system with a low memory footprint when Ignite is used as a secondary cache, and data is stored on disk and off disk. The system stores transactional data and main relationships across persisted entities, with data allocation in an isolated physical memory region separate from main process memory (e.g., in JVM).
Ignite is run as a secondary application instead of a startup process, with most data on disk. In the future, the Bloomberg team plans to run Ignite as a standalone process to more easily scale the number of clusters and nodes and tune the architecture as needed. They will also parallelize queries to extract data faster from SQL storage and increase the number of public and query thread pools to further minimize latency and increase batch size to manage I/O operations seamlessly.
Ignite Handles 500 Million Order Events Per Month in Latin America
Senior software developer Henrique Arroyo of iFood, a Brazil-based online food ordering and food delivery platform operating in Brazil and Colombia, shared what his team learned working with Apache Ignite over the past few years. iFood is the largest food delivery company in Latin America, with 300,000 stores in 1700 cities, 65 million food orders per month, and more than 500 million order events per month.
Food orders are received as events delivered using a polling API, with devices polled every 30 seconds. iFood chose an Ignite in-memory data grid over a PostgreSQL database for the polling solution for two reasons: performance and scalability. "Ignite also provided us with distributed SQL queries so it was a very good choice," said Arroyo, along with many other features that greatly enhance productivity. The polling system has sub-10 millisecond latency.
All of the Ignite Summit Europe 2022 sessions are available for viewing on demand.