The GridGain® Connector for Apache® Kafka® makes it easy to leverage Apache Kafka and Confluent® in order to create high performance streaming analytics and applications. Certified by Confluent, the GridGain Connector for Apache Kafka native integration which allows applications using the connector to ingest massive data sets from or publish to Kafka and Confluent, as well as process and analyze the data in-memory with unlimited horizontal scalability. The Confluent certified connector is included in the GridGain Enterprise and Ultimate Editions.
The GridGain in-memory computing platform, Confluent and Apache Kafka are widely used together for mission-critical streaming analytics and Hybrid Transactional/Analytical Processing (HTAP) to support a host of use cases ranging from retail digital business, to large-scale transactions and payments processing, to enabling the Internet of Things (IoT).
GridGain Connector for Apache Kafka Features
With the GridGain Connector for Apache Kafka, even the most complex integration use cases can be done entirely through configuration within GridGain. It is based on years of best practices developed for using Confluent and Kafka with GridGain and Ignite at scale.
The connector increases developer productivity by leveraging the full power of Kafka, such as preserving source data schema, supporting initial data loads, and allowing both source and sink filters. It also leverages the additional capabilities only present in Confluent. For example, you can use the Confluent Schema Registry for accessing Avro schemas when receiving Kafka messages.
The GridGain Connector for Apache Kafka enables end-to-end horizontal scalability. As a messaging source, it scales by assigning multiple Kafka partitions and topics to tasks within each deployed connector. As a destination, GridGain scales with Kafka, allowing connectors to receive in parallel across nodes in a GridGain cluster.
High Availability with Automatic Rebalancing
The GridGain Connector for Apache Kafka delivers reliability and high availability with true elastic scalability. Within a cluster, GridGain can automatically add or remove nodes and connectors to help distribute loads and ensure SLAs. It automatically rebalances connections and processing tasks as workers or caches are added or removed from a cluster. GridGain also supports multi-datacenter active-active deployment across locations to help ensure high availability even in the case of an individual datacenter failure.
The connector guarantees delivery during node failures or rebalancing across GridGain, Kafka and Confluent by leveraging GridGain as a fully transactional distributed in-memory store. GridGain keeps track across the cluster of data transfers so that even if an individual connector fails, any data delivery can be taken over and completed by another connector upon rebalancing or failover.