Stream Processing with the GridGain In-Memory Data Fabric

Event Workflow and Complex Event Processing Capabilities

GridGain In-Memory Streaming, available as part of the GridGain In-Memory Data Fabric, combines both event workflow and Complex Event Processing (CEP) capabilities integrated in one product. GridGain In-Memory Streaming allows you to use either standard SQL or programmatic APIs to find, index, aggregate, and analyze real-time streaming events. GridGain also provides comprehensive support for customizable event workflows. As events come into the system, they can go through different execution chains, supporting branching and joining of execution paths, with every stage possibly producing new types of events.

Streaming in the GridGain In-Memory Data Fabric comes with a comprehensive feature set supporting underlying data store types including RDBMS, NoSQL and Hadoop, as well as full ACID transactions, horizontal and vertical scaling options, and data center replication. The GridGain In-Memory Data Fabric offers first-class support through the Unified API for Java/.NET/C++ applications, and provides industry-leading security, management and monitoring functionality.

Uses for the GridGain Streaming Engine

Real-time streaming processing fits a large family of applications for which traditional processing methods and disk-based storages, like databases or file systems, fall short. Processing market feeds, electronic trading, security and fraud detection, military data analysis — all these applications produce large amounts of data at very fast rates and require infrastructure capable of processing data in real-time without bottlenecking.

One of the most common use cases for stream processing is the ability to control and properly pipeline distributed events workflows. As events are coming into the system at a high rate, event processing is split into multiple stages and each stage has to be properly routed within a cluster for processing. Another important feature of stream processing is the support for Complex Event Processing (CEP) to control the scope of operations on streamed data. As streaming data never ends, an application must be able to provide a size limit or a time boundary on how far back each request or each query should go.

In the past few years, several technologies have emerged specifically to address the challenges of processing high-volume, real-time streaming data. Generally such technologies evolve around specific use cases such as event workflow management or streaming data querying. Customers looking for a real-time streaming solution usually require both — rich event workflow combined with CEP data querying — and as a result are left with the difficult task of integrating different streaming technologies together.