Comparing Apache Ignite / GridGain and Apache Cassandra / DataStax as the Power Behind the Moment
If you’re in the process of a digital transformation and trying to improve the customer experience to compete against the Amazon, PayPal, Uber, Expedia, Netflix in your industry, you need to understand something before you fix your next performance and scalability bottleneck. There is only one way to deliver a personalized customer experience in real-time, one that has the speed and scalability to perform analytics in real-time during a transaction or interaction. This is what some people call in-process hybrid transactional/analytical processing, or HTAP.
The only way is have to have all the data about the customer available, and all the analytics, in the same place as the transaction or interaction is happening, in memory. It means having a single in-memory version of the truth for the customer and the business that supports real-time transactions and analytics. Companies including these new competitors have already implemented this approach.
If you’re not implementing this architecture, you’re eventually in big trouble. Read on and I’ll explain. Also, make sure you learn more about some of the technologies used for HTAP. This webinar comparing Apache® Ignite™ and Apache Cassandra™ for HTAP is a good starting point.
Sometimes, in the quest to deliver a better customer experience at each moment, and handle the ever-growing amount of information about the customer and the experience, companies forget to look at the big picture.
Over the last decade, transaction and query volumes have grown 10x or more, and data has grown roughly 50x. Many companies tackle this growth at each bottleneck in their existing systems and processes. They try to scale up an existing database that’s taking new data like clickstream or machine data by adding expensive hardware, or choosing new software like GridGain (built on Apache Ignite) or DataStax (built on Apache Cassandra) to increase write performance. Then they have a separate project for improving Web responsiveness, or improving API response times by using GridGain (Ignite) as an in-memory data grid (IMDG). Then there’s yet another project for using this new data for personalization, analytics, compliance or any other type of calculation.
The problem is that solving each bottleneck separately is making the problem worse. Technically it’s continuing to make more copies of data or keep data separate. By the time the data is processed, enriched and moved for a new that copy of the data is out of date. Organizationally, keeping the projects separate makes it harder for teams to work together to solve the end goal: to improve the customer experience in real-time as they interact with the company through their channel of choice. These projects are all part of that. They all need to be looked at together to see if they’re fast enough end-to-end, and if they take the right actions at the right moment.
Think of it this way. If you’re going to respond to a customer within 1 second at any time to improve the experience, how much time do your applications and “middleware” have to get all the data it needs, to make all the calculations and then respond? How much time is left after you subtract network, server and other delays? A lot of the time I’ve seen the need for an end-to-end middleware and application response time of 100-200 milliseconds.
So here’s my claim. There is only one way to add the speed and scalability required to deliver this real-time responsiveness, to make decisions during a transaction or interaction, to perform real-time in-process HTAP. You have to have all the data about the customer available, all the analytics, in the same place as the transaction or interaction, usually in memory.
Eventually the data needed has to be in memory, in place, for speed. At some point, you’ll probably be supporting 1,000 customers at any given time. Let’s assume you need 1MB of information about each customer to support any interaction, that’s 1GB. It takes at least one second to move that data over a 10GigE network if you’re lucky, so the data needs to be on the same place you plan to run any processing. The data needs to be in RAM because hard disks (HDD), even SSD/Flash take a second or more to move 1GB into RAM for computing.
You may not be facing this today, but your architecture needs to support where you’re going. GridGain/Ignite is used as an in-memory computing platform for this reason. They all have to keep this much information about customers, product, pricing or other data to personalize the experience and their customers in real-time. The second you start to perform in-process HTAP and need data about the customer to automation decisions or personalization, you’ll need this architecture.
These companies often started by accelerating their existing architectures to handle greater query volumes by using GridGain/Ignite as an in-memory data grid in between existing applications and databases. This helped offload from the existing databases. Over time they added more and more existing and new applications to GridGain, and started to store data using GridGain/Ignite native persistence to achieve greater write throughput. But the processing all happens in a memory-centric architecture for speed. All the analytics happen in the same place as the transactions, in memory, without network or disk delays, using massively parallel processing (MPP) style collocated computing techniques to get near real-time results and take action during a transaction or interaction.
This isn’t possible when you use Cassandra or other databases. While Cassandra has great write scalability, it doesn’t support transactions with immediate consistency. What that means is that the transaction will not be accurate right away, or it could eventually “fail” and have to be undone. Cassandra also doesn’t provide great read performance or flexibility. So queries can take longer, and might not even be possible. It means you’re not able to deliver results fast enough from Cassandra during each and every moment in the customer experience.
So remember, focus on what the customer needs first, not what the technology needs. Whenever you look at any bottleneck in the customer experience, look at the end-to-end experience to determine if fixing the bottleneck will help in the long run or make the situation worse. See if you can put in place the right technology now that will fix the short term bottleneck and give you a path towards the right long term architecture. If you want to learn more about using in-memory computing for in-process HTAP, watch an expert explain how Ignite and Cassandra compare.