Real Time Big Data Processing with GridGain

This book is about how to use GridGain software to develop innovative distributed compute and data intensive applications that run on any managed infrastructure - from a simple laptop on which this book was written to a large grids and all types of clouds. When developing with GridGain and reading this book you can use both Java, Scala or Groovy programming languages. At the time of this writing - GridGain was the only distributed computing middleware with native Scala support.

This book does not replace API manual references, and you would still need to consult them from time to time for up to date method signatures, parameter description, etc. As we are always saying in our project - documentation is the code too and we pay great deal of attention to API References (Javadoc, Scaladoc and Groovydoc) - in fact we have one of the best organized and maintained code level documentation among any relevant projects.

This book is short considering its subject - and it is on purpose. We strongly believe that one of the main reason for slow adoption of grid and cloud computing in the last decade was over-complication and unnecessary "dramatization" of entire subject. In fact, the original idea behind GridGain came from rather unproductive experience developing application with Globus toolkit - innovative piece of software for early 90s but awfully over-engineered and out of place by the turn of the century with rapid advancements in server-side JVM-based programming.

As you discover in this book the distributed computing is not necessarily complex or unwieldy as it may seem from the outside - in fact, most of its concepts are readily familiar for the most of you. Where it was usually getting complicated in the past is the tooling and framework support. That was a sore state of the affairs for a long time - most of engineering community just accepted that the distributed programming (grid and cloud computing included) just has to be "involved" because…. it is so, right?

In reality - it’s largely inaccurate. With the right tooling and framework support the distributed computing can be relatively simple and very productive. One of the main ideas in this book is to try to convince you that it is so. Every month or so we at GridGain receive email or a forum post where someone relates his or her experience of downloading GridGain during the day and by the midnight having the first MapReduce application running on Amazon EC2. By the time you finish this book it won’t seem like an exaggeration and you will have your first application running on EC2 much, much quicker.

This book covers GridGain starting with version 3.0 and provides in-depth manual for all three main technologies that are tightly integrated into GridGain cloud application platform (as of October 2010 GridGain is the only distribute middleware that provides all three technologies in the same platform - let alone for Java, Scala and Groovy languages):

Each of these main topics is covered in depth with plenty of examples in both Java, Scala and Groovy.

All in all, this book’s character perfectly reflects on what we think modern distributed programming should be - simple, effective and amazingly productive.

That is if you use GridGain…

This book is written primarily by Nikita Ivanov and Dmitriy Setrakyan. As always we encourage to contact us directly should you have any questions about this book or about GridGain. You can reach Nikita at nivanov@gridgain.com and Dmitriy at dsetrakyan@gridgain.com.

Both Nikita and Dmitriy have over 30 years of combined experience in distributed programming mostly in Java (some MPI C/C++ back in 90s - ouch) and now Scala. Nikita and Dmitriy were at the beginning of the project and to this day coordinate and lead most of the development work.

They both write code almost every day for GridGain despite heavy travel and frequent speaking engagements. And when they don’t write code - you’ll find them rooting for San Jose Sharks - the favorite hockey team of GridGain.

There are many way to stay in touch with GridGain project:

In a nutshell - GridGain is a JVM-based middleware software that enables the development of compute and data intensive High Performance Distributed Applications. Applications developed with GridGain can scale up on any infrastructure - from a single Android device to a large cloud.

GridGain provides two major areas of functionality:

On top of that it provides the multitude of surrounding technologies many of which are frequently used by our clients on their own.

With GridGain your applications can:

Compute and In-Memory Data Grid act as two main axiomatic technologies for a modern distributed programming. They are fundamental because they solve two underlying problems faced by any distributed system:

I always like to provide this analogy: every computing device - from Turing machine to the latest iPod - contains memory and a processing unit. Think about it… memory and the processing form the foundation of our computing capabilities. And so is the ability to distribute computations and data form the foundation of distributed programming.

And just like in late 1960s we’ve had first "system on the chip" where memory and processing units were finally integrated and combined on the same chip providing for cheaper, more energy efficient and much faster overall systems - GridGain has pioneered integrated middleware that combines compute and in-memory data grids in one cohesive and integrated distributed middleware software. This has resulted in similar benefits of simplified programming model, easier applicability and unified configuration and management.

Compute and in-memory data grids are the key topics in this book and we’ll talk a lot more about these two in the following chapters.

You noticed in the previous chapter that we call GridGain as a software middleware for developing High Performance Cloud Computing application. But why we focus on High Performance and what do we mean by that? And what does it have to do with Cloud Computing?

Let’s talk about Cloud Computing first.

To start off this sub-chapter I’ll tell you one story that happened to me few years ago.

I was presenting about GridGain at Java User Group at Dayton, Ohio. For those of you who don’t know - Dayton, OH is essentially a "sleeping quarters" for Wright-Patterson Air Force Base, one of the largest military research facilities in the world. It employees almost 30,000 people and headquarters massive Air Force Institute of Technology and Air Force Research Laboratory just to boot and it is known to have one of the largest super-computing centers in the world too. So, I’ve had few people from the base on my presentation…

After the presentation I’ve chatted with some folks and conversation drifted into general topic of grid computing and what we all understand by that (and by relatively new back then concept of cloud computing). It was a real surprise to me (to say the least) that I got three very different answers from three guys working on the base - essentially working in one big company.

One answer was that grid is really nothing new alluding to parallel Fortran of almost 50 years old, another one was more in line with common understanding of grid computing being just a new re-tooling of traditional parallel processing, and yet another answer was that the whole thing is just a hype and multi-core CPUs will displace it all together in 5-10 years max.

I remember on my flight back I was a bit perplexed by how far apart those guys were while working side by side in albeit large organization and not just on technicality - but on fundamental view of distributed programming (grid or cloud - doesn’t matter). Starting with my next presentation onward I’ve made a rule to always state what I believe about grids and clouds to at least make sure we have common frame of reference. Whether or not you agree with - is another question…

So, here’s my take. I don’t particularly like terms grid and cloud computing. There’s nothing that resembles a "grid" in grid computing and obviously there’s nothing that is performed in the "cloud" when it comes to cloud computing. Both marketing terms "grid computing" and "cloud computing" represent slight variations of traditional distributed programming.

As you can see the difference between grid computing and cloud computing is only in representation of computing resources which in most cases should be irrelevant. The lines between grids and clouds are getting blurrier by the day and we use both terms interchangeably throughout this book (as long as context is clear).

It is, however, important to know the difference and new challenges that cloud infrastructure brings to the table for you as a software developer.

Only at the pick of the hype around the cloud computing can you get a chapter named like that…

Nowadays these terms are thrown often without much regard or understanding and while IaaS and SaaS are somewhat well defined, the PaaS is something that poorly defined if at all. Let us try to define these terms and see where GridGain fits in the picture.

Picture below provide basic overview of how GridGain related to IaaS and PaaS:

GridGain is dual-licensed:

EULA (end-used license agreement) for Enterprise Edition is available at GridGain root installation folder after you install GridGain and it is also available on download page on the http://www.gridgain.com website.

GridGain System also provides OEM, Enterprise an Academic licenses with further details available upon request at sales@gridgain.com

Note that throughout the book we don’t directly distinguish between features available in Enterprise and Community Edition. In cases where it is important we make a note.

The best way to get a free support on GridGain software is to dip into our active community with wealth of information on our free support forum: http://jive.gridgain.org. This forum is closely monitored by GridGain System’s engineers and we try our best to provide free support there when applicable.

GridGain Systems, Inc., as a company behind GridGain project, also provides full spectrum of commercial services around GridGain software including:

When it comes to training and support - we at GridGain have a very simple philosophy: we do our own heavy lifting. We believe that we are the best people to support our own software and who is it better to learn about GridGain from but the people who develop it daily?

All information about services provided by GridGain Systems can be found at http://www.gridgain.com/services.html

Now that we have topology set we are going to switch to writing the actual code of our first application. Code examples below don’t have any dependencies on IDE - and you can follow up using any text editor of your choice.

The first app we are going to write would be computational MapReduce. It will calculate number of non-space characters in a given string by splitting the string into individual words, calculating word’s length on the remote nodes and aggregating results back.

We’ll use FP-based approach that GridGain natively support (even in Java):

Not that there are many ways you can write this particular program in GridGain:

FP-based approach above, however, yields probably the shortest program but can be initially confusing since Java isn’t really supporting FP natively. So let me explain step by step how this works.

Lines 1-4
We start by importing all necessary classes and constants into the scope.

Line 7
We define a main(…) method that will take input string.

Line 9
Method GridFactory.in(…) simply takes a closure and:

So, essentially, it allows for a quick execution of the piece of code within context of a running GridGain node.

Line 9, 10
As a parameter to GridFactory.in(…) method call we are passing a newly created closure of type GridInClosureX<Grid>. The body of this closure will be executed within context of a running GridGain node.

Line 11-15
GridInClosureX<Grid> has one method applyx(Grid) which passed a Grid interface instance. Inside of applyx(Grid) we call reduce(…) method that performs MapReduce operation. It accepts four parameters:

The logic of this computational MapReduce should be clear by now. We split input string by spaces into individual words, we then send every word to a remote node where a method length will be called on that word, and results of these calls will be returned back to reducer that will simply sum them up.

Now that we have a basic understanding of what is happening inside of this code let’s run this example. Depending on whether or not you are using IDE, Maven or Ant build you simply need to include gridgain.jar file that’s located in GRIDGAIN_HOME directory and all JARs under GRIDGAIN_HOME/libs folder to your classpath. Also, if you use IDEs - make sure that either environment variable GRIDGAIN_HOME is inherited by IDE process or system property with the same name GRIDGAIN_HOME is setup in your runtime configuration.

When it’s all set and done I’ve passed "GridGain is awesome" string my IDEA 10 runtime configuration and the got the following output in my IDEA output window:

As you can see the output is very similar to standalone nodes we’ve started few minutes ago. But in the end we have output of our MapReduce task which says:

correctly computing number of non-empty characters in input string "GridGain is awesome". Notice also we’ve had topology snapshot with three nodes (as expected). If you check other standalone nodes you will see the similar output to this:

indicating that when our application started we’ve had three nodes in the topology and when our application completed and stopped we were back to two nodes (i.e. two standalone nodes).

Now, let’s advance our example. As you probably noticed we don’t see any evidence on remote nodes that any processing is happening there. In fact, by default GridGain starts in QUIET mode and most of the output is suppressed (if you need to start in normal mode - use -v flag for ggstart.sh script).

Let’s modify our example so that we’ll see what is being process and where:

We’ve commented out the reflection-based closure and added direct closure creation that prints out what string it is working on and returns its length. If we re-run our application we’ll now get the following output on three nodes:

Remote Node 1

Remote Node 2

Local node in IDE

Note that local node (i.e. the node running in IDE) performs calculation as well as performing the final reduction step. If we don’t want it to participate in the actual calculation and only perform the final reduction - we can simply change this line:

to this

Pretty neat, right? And so just in about two dozens lines of code and 10 minutes we’ve got our first MapReduce application running.

Now - let’s move to In-Memory Data Grid application and we’ll use Scala for that, more specifically - Scalar, our Scala-based DSL for GridGain.

Since we are going to use In-Memory Data Grid in this example we need to restart our standalone nodes with enabled data grid. Note that by default there are no caches configured (for obvious performance reasons). GridGain comes with handy example configuration that comes with three caches configurated examples/config/spring-cache.xml.

You can stop existing nodes by simply Ctrl-C and then start them again using:

The output from the nodes is almost the same as in previous example with one notable change:

indicating that we now have three configured caches that are named based on their type.

Now that we have standalone nodes running with necessary configuration let’s turn to writing our application that will utilize the In-Memory Data Grid (we’ll use shorter term data grid going forward) side of GridGain. We are going to create an application that will populate data grid with a set of key-value pairs and then execute the set of closures where each closure will have an affinity with the specific key in the data grid - and therefore it will be co-located with the data for that key instead of just randomly be executed on some node in the grid.

Scalar-based (Scalar is GridGain’s DSL based on Scala) application looks pretty simple:

Even if you are not familiar with Scala - the code looks pretty self-explanatory. Let’s go line by line like we did for Java example above.

Line 1-3
Necessary imports including import for Scalar

Line 7
Defines number of keys we’ll be storing in the data grid and number of closures we’ll be executing later.

Line 10
Initializes Scalar with the same configuration file as we used for standalone nodes examples/config/spring-cache.xml. Note that initializing Scalar essentially means starting up the local node.

Line 11
We are getting an instance of the cache named partitioned (which is a partitioned cache named so for clarity). Cache is typed for Int keys and String values.

Line 13, 14
Calls functions populate() and colocate() that are defined later.

Line 18-20
Function populate() simply puts key/value pairs into data grid. Node that partitioned cache will store particular key/value pair on one of the nodes (potentially including the local one) as well as on one back up node. Since we have three nodes in the topology (remember - one local node and two standalone noes) - each key/value pair will be store on two nodes.

Line 22-26
Function colocate() executes number of closures where each closure gets affinity co-located with some key in the data grid. Note that the closure itself simply prints the trace message and uses function peek() that gets the value only if it’s locally available - which should be since we are co-locating closure with the node where data is stored (so called master node).

Let’s go ahead and start our application. Starting Scala application is no different than starting Java application (at least if you use IDEs). Local node running in IDEA prints out the following log (abbreviated):

and the remote nodes print:

Remote Node 1

Remote Node 2

As you can see the key/value pairs got distributed roughly equal (the more keys the better the distribution will be obviously). What’s also important to note is that we didn’t get any null as values proving the fact that co-location work (remember: we’ve used function peek() that only return locally stored value or null if value for given key is not stored locally).

All in all - these were two quick examples of Java and Scala based applications that demonstrate some of the basics of GridGain functionality. The following chapters in the book will explain why these are just a scratch on the surface…

There are three ways how you can get GridGain:

We highly recommend to use the first method and simply download ZIP file from http://www.gridgain.com website. To do so - simply open http://www.gridgain.com in your favorite browser and locate the download link that usually on the right side:

Once you clicked on the download link you’ll be on download page and you’ll need to enter your name and email:

Keep in mind several things:

There are six downloads (as of version 3.0.2):

All downloads are simple ZIP files. ZIP files are versioned and clearly named to indicate for what OS family they are intended to.

Once you download whatever ZIP file your have selected - the installation process is rather trivial:

Note that installation does not perform any new-line translations and text files may have wrong new-lines depending on what OS installation is performed.

Unix/Linux/Mac OSX ZIP file has all Shell scripts with executable flag set so that they can be called directly.

Uninstalling GridGain is even simple than installing - you simply remove the GRIDGAIN_HOME folder where GridGain was installed. If it was configured to use paths outside of GRIDGAIN_HOME you will need to delete them too (if necessary).

Due to complexity of GridGain (mostly due to its distributed nature) we have decided not to provide incremental upgrade (or patching) capabilities. We recommend upgrading GridGain by cleanly uninstalling and installing a new upgrade version.

GridConfiguration^doc interface defines grid runtime configuration. This configuration is passed to GridFactory.start(GridConfiguration)^doc method. It defines all configuration parameters required to start a grid instance. Usually, a special class called "loader" will create an instance of this interface and call GridFactory.start(GridConfiguration)^doc method to initialize GridGain instance.

Note, that absolutely every configuration property in GridConfiguration^doc is optional. You can simply create a new instance of GridConfigurationAdapter^doc, for example, and pass it to GridFactory.start(GridConfiguration)^doc as is to start grid with default configuration. See GridFactory^doc documentation for information about default configuration properties used and more information on how to start grid.

The following configuration parameters can be used to configure grid node with GridConfigurationAdapter:

Some of the most commonly used configuration properties are explained in more detail below.

Starting with GridGain 2.1 you can start multiple instances of Topology SPI, Load Balancing SPI, Failover SPI and Checkpoint SPI. If you do that, you need to tell a task which SPI to use (by default it will use the first SPI in the list).

Add @GridTaskSpis^doc annotation for your task to specify what SPIs it wants to use. If this annotation is omitted, then by default GridGain will pick the first corresponding SPI implementation from the array of SPIs provided in configuration.

This example shows how to configure different SPI’s for different tasks. Let’s assume that you have two worker nodes, Node1 and Node2. Let’s also assume that you configure Node1 to belong to SegmentA and Node2 to belong to SegmentB. Here is a sample configuration for Node1:

Node2 configuration looks similar to Node1 with segment attribute set to B:

Now, if you have Task1 and Task2 starting from some master node NodeM, you can easily configure Task1 to only run on SegmentA and Task2 to only run on SegmentB. Here is how configuration on master node NodeM would look like:

Then your Task1 and Task2 would look as follows (note the @GridTaskSpis annotation):

and

Grid Spring bean allows to bypass GridFactory^doc methods. In other words, this bean class allows to inject new grid instance from Spring configuration file directly without invoking static GridFactory^doc methods. This class can be wired directly from Spring and can be referenced from within other Spring beans. By virtue of implementing org.springframework.beans.factory.DisposableBean and org.springframework.beans.factory.InitializingBean interfaces, GridSpringBean automatically starts and stops underlying grid instance.

The following configuration parameters are optional:

GridConfiguration may be defined in code:

or from Spring configuration file (default Spring configuration file can be found in GRIDGAIN_HOME/config/default-spring.xml file):

In order to use annotation based @Gridify^doc AOP-based grid-enabling the following AOP configuration needs to be in place depending on which AOP implementation you choose to use. Note that you only need to pick one AOP implementation.

GridNode^doc interface defines a logical grid node in the network topology. Note that a physical node (like a computer on the network) can have multiple logical grid nodes running on it. In fact, a single JVM can run multiple logical grid nodes - note that GridGain is the only software in the world allowing this unique capability.

GridNode interface has very concise API and deals only with a notion of a logical network endpoint, a node, in the topology: it has globally unique ID, node metrics, set of static attributes provided by the user and few other parameters.

The unique characteristic of GridGain is that it uses Peer-To-Peer (P2P) topology meaning that all nodes in GridGain are equal. There is no master or server nodes, and there are no worker or client nodes either. All nodes are equal from GridGain’s point of view - yet all these and any other roles can be assigned logically to the nodes.

This unique design gives GridGain tremendous flexibility: not only you are not limited to the master-worker mold - you can define any application specific roles and assign them to the nodes dynamically. More over, since these roles are logical, they can change and "migrate" from node to node as topology changes or based on your application logic.

GridNode interface is used primarily by internal kernal code and by discovery and communication SPI implementations and rarely used directly. GridRichNode^doc interface, its rich counterpart, is what used instead for majority of GridGain operations. More on this below.

Local grid node is an instance of GridNode interface that is instantiated in the local JVM runtime for a specific grid. In general, JVM process that runs GridGain runtime can have zero, one or more local grid nodes, but only one local node per specific grid’s topology.

As the logical extension of the network endpoint - a grid node as defined above - we use the term topology throughout the GridGain documentation to reference a set of all logical grid nodes where each node "knows" every other nodes (in other words, topology is a fully connected graph of all grid nodes including the local node). We often refer to such topology as simply a grid.

Depending on configured discovery SPI implementation (discussed later) the topology can be guaranteed to be consistent on all nodes at any point of time or be eventually consistent only. The eventual consistency means that all nodes will eventually get into fully consistent view but there a short time window where nodes can have a different view on the global topology. The guaranteed consistency is expensive to implement and it is optional in GridGain.

Note, however, that for data grid, for example, the configured SPI must provide consistent topology (i.e. support guaranteed discovery discussed later).

It is important to note that a single GridGain runtime can support any number of topologies or grids in the same time. Nodes from one topology have no knowledge about the nodes in other topologies. In such cases, single GridGain runtime (JVM process) will have several local grid nodes where each local node would belong to a different grid. Again, there is an important distinction between GridGain runtime (a JVM process) and a logical grid node running inside of that runtime.

Note that we often refer to virtual sub-grid or sub-topology which is essentially just a subset of grid nodes from one topology. More on all that later.

One of the major addition in GridGain 3.0 was introduction of a grid projection (and corresponding GridProjection^doc interface. The important observation that led to this addition was the fact there is a large set of GridGain operations that can be defined uniformly on any arbitrary set of grid nodes.

Put it differently, each such operation can be performed on zero, one, two or any other number of grid nodes. For example, you can send a message to one, or more nodes in the grid. Just as you can listen for messages from one, two or any other number of nodes. And so on. To use functional programming terminology - these are the monadic operations defined on a set of grid nodes (which correspondingly makes GridProjection a monad).

To logically group such operations the GridProjection interface is introduced and it defines all major operations in the GridGain that can be performed on a arbitrary set of nodes. You can think about grid projection as a specific view on topology. Projection can be static or dynamic and there are many ways how a projection can be defined.

As you will see throughout this book the elements of functional programming or functional API design are central to GridGain. You’ll discover that even when working with Java APIs you are dealing with functional constructs most of the time - even though Java is not a functional language to being with! This is one the unique sides of GridGain and it leads to extremely elegant and simple to use APIs.

We are going to cover projections and functional programming framework in Java in much greater details in subsequent chapters.

Once we introduced grid projection it is pretty logical to extend definition of a grid node as a grid projection with just that one node it. Similarly, one can extend grid cloud definition as a grid projection that contains all nodes belonging to that specific physical cloud. And finally, it is only logical to provide a conveniently defined global projection that contains all the nodes in the topology.

This is exactly what GridRichNode^doc, and Grid^doc interfaces do. They all extend GridProjection interface and add all necessary additional operations that are specific to a grid node, grid cloud or a global topology.

The idea of rich interfaces is central in Scala and Ruby libraries, for example. By having both thin and rich interfaces (GridNode and GridRichNode) we can satisfy both types of interface usages:

Historically, Grid interface - being a global all-inclusive projection, has an additional special purpose. Grid interface acts as a main entry point for entire GridGain functionality. In fact, most of the operations you perform on GridGain originate on Grid interface. That is where you can obtain the instance of data grid, get an instance of rich cloud interface for a specific cloud, create and manage grid projections, manually deploy grid tasks and perform multitude of other operations.

To get an instance of Grid interface you need to use GridFactory.

GridFactory^doc class is a life-cycle factory for Grid instances. Its purpose is to provide various ways to start and stop instances of Grid interface. Note that Grid interface - being the main entry point for GridGain APIs - has a strict life-cycle and state machine that is controlled by GridFactory. As noted before, single GridGain runtime (JVM process) can have zero or more Grid instances each providing a local view on a different grid.

The usual way to work with GridGain is to use GridFactory class to start Grid instance using specific (or default) configuration file (usually at the beginning of your application). Starting Grid instance means, among other things, starting a local grid node and have it join the topology. Once you have started Grid instance you can use any APIs provided by GridGain. When GridGain is no longer needed, you use GridFactory to stop the Grid instance and its node will leave the topology (usually at the end of your application).

GridCache^doc interface represent the main API entry point for data grid functionality. GridCache instance always refers to a single named cache. You can configure as many named caches as you like. You receive the GridCache instance from Grid instance (as anything else in GridGain). GridCache is a rich interface and represents global cache projection (see below).

GridCacheProjection^doc interface is analogous to GridProjection but it defines a cache projection over specified set of key-value pairs. In fact, GridCache interface extends GridCacheProjection and simply represents a global cache projection, i.e. the projection over all key-value pairs in this cache.

Cache projections are extremely powerful technique in GridGain’s data grid. It provides a monadic set of operations that is defined on any arbitrary set of:

giving data grid a distinct functional flavor and providing consistent API design between compute and data grids.

MapReduce is a relatively new name for very old concept. In a strict terms the term MapReduce refers to patented algorithm introduced by Google in their internal distributed data processing systems and closely mirrored by Hadoop project developed by a competing Yahoo!.

We (as well as distributed programming community in general) tend to use term MapReduce in more wider sense since it was extensively publicized and we refer to any divide-and-concur design strategies or traditional parallel computing as MapReduce. In fact, if you have a long running task, you can split this task into multiple sub-tasks, execute these tasks in parallel, aggregate their results back and get you final result in a fraction of time. That’s a classic parallel programming, or compute grid - and to avoid myriads of names we call it MapReduce too.

Hadoop provides one implementation of MapReduce that is closely matching Google definition as an Apache project. Note that Google granted the license to Hadoop to use patented Google algorithm.

Streaming MapReduce is a less-defined term but often refers to a type of processing similar to MapReduce (i.e. tasks gets split and reduced) but with input data is not finite in general. Typical example is a search in a live video feed: the obvious problem is that you can’t load the entire feed first and then partition it into small parts to be processed in parallel (like you would do in a traditional MapReduce); you need to somehow map and reduce the incoming data as it comes and in the same time keep providing failover, topology resolution, collision resolution, back pressure control, and all other necessary services.

GridGain provide several unique ways of how this type of processing can be implemented.

GridGain is all about processing large data sets (a.k.a. BigData) in real-time. But what real-time are we talking about?

In GridGain - we are talking about perceptual real-time, or a software real-time (Java Virtual Machine doesn’t technically support hardware real-time processing). Perceptual real-time is defined by a maximum response time that a typical user will wait for the task he or she expects to be "instant" before cancelling the task. For example, when a typical user clicks "Add to Basket" button on the website anything beyond couple of seconds will probably make that user to click "Back" or otherwise cancel the task. For a online trading application the delay of few seconds on submitting the order will indicate something wrong with the system. Portfolio management application that takes 10 seconds to open last minute moving average chart is practically unusable. And so on…

We mention closures and predicates here only because we provide their full implementation on Java side. Unlike Scala or Groovy, where closures (functions) are part of the language, in Java they are not - and we had to develop an entire state of the art distributed functional framework for Java that is included with GridGain.

Closure is a block of code that encloses its body and any outside variables used inside of it as a function object. You can then pass such function object anywhere you can pass a variable and execute it. Predicate is a special type of closure that simply returns boolean value.

Scala as a hybrid OOP and FP language offers natural advantage over Java since it provides native in-language support for closures which enables much more concise and elegant APIs provided by GridGain’s Scalar DSL. However, with GridGain’s functional framework we brought Java functional usage as close to Scala as possible.

Below is a simple example that broadcasts and prints string on all nodes in the topology. Just compare how close Java, Groovy and Scala implementations are:

Closures and predicates used extensively in GridGain APIs and at the core of our design. You will discover plenty of examples later in the book of how closures and predicates - in Java, Groovy and Scala - used throughout the GridGain.

One of the unusual features of our Java side APIs is introduction of type aliases (also known as typedefs) in GridGain 3.0. With introduction of functional design in GridGain 3.0 we have quickly discovered that Java APIs are just way too "chatty" due to lack of any type inference by the Java compiler. The only way to combat that problem in Java is to introduce type alias - a subclass with a shorter name.

Obviously, it works best (or at all) for static, factory-type classes but it also works for object instantiations but unfortunately you can’t use type aliases in method signature since they are different types. Such as live in a Java world…

We’ve introduced aliases only for key GridGain types and there’s only about a dozen of aliases in GridGain APIs. It’s pretty easy to memorize the key ones that are used most frequently. Usage of aliases is, of course, optional as you can always use full (original) names of the types. But we are pretty sure that you’ll find aliases on Java side useful to make your code more readable.

Here’s good example demonstrating how aliases can Java code slightly more readable:

Grid task and job are the main abstractions in the compute grid. As you recall the compute grid is about parallelization the processing, i.e. splitting a long running task into multiple sub-tasks, executing those sub-tasks in parallel and aggregating their results into one final result.

GridTask^doc defines the descriptor for the overall task to be processed and grid job defines a sub-tasks, i.e. the piece of code that will travel to remote nodes for execution. Essentially a grid task defines mapping and reducing logic, while GridJob^doc is very similar to java.util.Callable interface by defining a simple executable body.

Note that anything that gets executed on the compute grid should be defined as a grid task. Closures and AOP-based executions get converted to grid task automatically when needed.

Service Provider Interface (SPI) is at core of GridGain architecture as it provide pluggable modularization to GridGain. SPI concept is simply a component interface with multiple pluggable implementations that all share unified life cycle management. There are two key benefits in this approach:

The example of SPI is communication subsystem. It has simple interface GridCommunicationSPI^doc and half a dozen implementations that GridGain provides out of the box. These implementations can be set in grid configuration GridConfiguration (and TCP/IP-based implementation is always set by default so that you don’t have to set anything to get GridGain working right away).

GridGain is "sliced" into dozen of different SPIs for all major subsystems and each such subsystem can be fully replaced by user’s specific implementation - an extremely powerful feature of GridGain.

Type aliases or typedefs used in programming languages to give shorter name to existing type name to make the code that is using it more readable and easer to understand.

Many languages provide built-in support for type aliases or typedefs.

C-based languages (C/C++/Objective-C) provide direct support for it:

Scala also provides excellent built-in support for type alias that goes beyond C-based capabilities. You can declare a type alias right during importing the original type:

And you can also declare the type alias in your code much the same as declaring method or a field (and similar to C/C++):

You often use structural types in Scala with type aliases:

declares T as an alias for any type that has method foo with return type Unit.

Java, unfortunately, does not provide any support for type aliases… Yet Java needs them more than any language above due to its syntactic bloat and lack of any reasonable type inference. In fact, look at this "typical" Java code:

Since there is no type inference you have to repeat bloated HashMap definition twice in this definition for no reason what-so-ever - it only makes code look more busy and less readable. And if this type of HashMap is used frequently - and there’s no way to shorthand it - you have to repeat this over and over again in every place where it is used.

Surprisingly enough, this shortcoming of Java is often sighted as one of the major reason the Java code "feels" bloated and unnecessary verbose. This gets even more pronounced when you start using Scala that, like Java, is fully statically typed but removes most, if not all, bloat and unnecessary repetition from the code.

With introduction of Functional Programming in GridGain 3.0 (explained in the following chapter) we were faced with this very problem in Java APIs: we had plenty of new interfaces and classes that were used literally everywhere in our code and it was becoming unwieldy in many places since many of them require parametrization and code was becoming simply ridiculous in some place… Needless to say that users of GridGain APIs would have been faced with the same problems.

To solve this problem (somewhat) we have introduced typedefs to our Java APIs (Scala APIs naturally use Scala language type aliases).

In package org.gridgain.grid.typedef^doc we have few dozens of typedefs defined as sub-classes with short one-two letter names for various frequently used types in GridGain. The following table shows all typedefs shipped with GridGain:

As you can see typedefs defined primarily for functional classes (tuples, closures, and predicates) as well as for a few factory classes like GridFactory^doc and GridFunc^doc. Here’s a short sub-list of the most frequently used typedefs in GridGain:

Here’s a code snipped from GridFunctionalCopyExample example that is shipped with GridGain. First version does not use typedefs and uses full names of the types:

I would argue that the second version gains more readability and easier to understand since we don’t have to repeat ad nauseum GridFunc and GridPredicate in every line.

Note also that we have couple of typedefs that shorten parameterized types that allows for greater brevity and more concise code:

GridGain provides pluggable logging capability by allowing the user to specify his own logging framework. This is especially convenient when GridGain runs inside of the hosting environment such as servlet container or application server.

In such case, GridGain can be easily configured to route all its logging through host’s logging framework eliminating the need to have multiple log file locations. This also dramatically simplifies the debugging since multiple log file don’t have to be line-by-line synchronized.

To provide this pluggability GridGain relies on its own interface GridLogger^doc that provides absolute minimal API for logging. While GridGain uses this interface throughout entire product it further provides out-of-the-box complete implementations for this interface using the following popular logging frameworks:

Users, of course, are free to provide their own implementations and many often do for integration with existing log analysis or health monitoring solutions.

Grid loaders are used to start grid in different environments. Loaders provide basic boilerplate code for starting GridGain in various environments. For example, when starting within application servers such as JBoss, Weblogic or Websphere, provided loaders will configure GridGain to use "native" logging, JMX facility, discovery and execution services (JSR-237) which makes GridGain basically blend into hosting environment. Loaders do not need to implement any interface and their sole responsibility is to configure and start grid.

GridLifecycleBean^doc reacts to grid lifecycle events defined in GridLifecycleEventType^doc. Use this bean whenever you need to plug some custom logic before or after grid startup and stopping routines.

There are four events you can react to:

TODO

Messaging - an exchange of the messages between grid nodes - is one of the main functional areas that often used standalone in GridGain (without using main Compute and Data Grid capabilities). Given GridGain’s sophisticated topology management and auto-discovery it just makes sense for many applications to simply piggy-back on this functionality and use GridGain as an intelligent message bus.

TODO

Grid.executor()^doc method creates ExecutorService which will execute all submitted Callable and Runnable tasks on the grid. User may run Callable and Runnable tasks just like normally with java.util.ExecutorService, but these tasks must implement Serializable interface.

The execution will happen either locally or remotely, depending on configuration of Load Balancing SPI and Topology SPI. Distributed ExecutorService delegates commands execution to already started Grid instance. Every submitted task will be serialized and transfered to any node in grid.

Here is an example of an ExecutorService to show how it can be used.

where simple FooCallable is:

Peer class loading (P2P) is turned on by default. To turn it off set GridConfiguration.isPeerClassLoadingEnabled()^doc property to false in Grid Configuration.

Although internals of peer class loading are rather complex, what it means in a nutshell is that when a JVM on the remote node needs to find a certain class as part of the grid task execution, it will check the local class loader first and if such class cannot be found, it will ask the node that originated grid task execution (one that should have this class by design) to provide it. In an essence, GridGain class loading becomes grid-aware.

This technique is invaluable during grid application development. It allows for absolutely grid-transparent development cycle: you write your application as you normally do (in a single node environment of Eclipse, IDEA, etc.), compile and run it - and it seamlessly runs on the grid without any extra deployment or build steps what-so-ever. More over, GridGain supports hot-redeployment so you don’t have to restart GridGain every time you change the grid task code - again, just modify the code, compile and run and all your changes will be picked up on the grid.

Peer class loading sequence works as follows:

Peer class loading should be used in most situations, especially during development with Java IDEs. It allows to dramatically reduce overhead of grid-enabled application development effectively making it as quick and productive as local application development. You simply change code and run - and your modified application seamlessly runs on the grid.

Deployment mode is specified at grid startup via GridConfiguration.getDeploymentMode()^doc configuration property (it can also be specified in Spring XML configuration file). The main difference between all deployment modes is how classes and user resources are loaded on remote nodes via peer-class-loading mechanism. User resources can be instances of caches, databased connections, or any other class specified by user with @GridUserResource^doc annotation.

The following deployment modes are supported:

When deploying grid tasks into JEE container, you can keep using standard JEE deployment artifacts. For example, if you are deploying a WAR file into JEE container, simply add your grid task classes to the WAR file and that’s it.

GAR deployment is a traditional deployment model, similar to JAR/WAR/EAR deployment in JEE, where you create the *G*rid *AR*chive file that contains all necessary classes for the grid task and deploy it. GridGain comes with URL-based GridDeploymentSpi^doc implementation so that you can deploy your GAR files on any URLs accessible via FTP, HTTP(S), POP3 or FILE protocols.

For example, when properly configured, you can just drop your GARs into certain folder on your web server and they will be deployed on the grid.

GridGain comes with 2 deployment SPIs:

Network segmentation (a.k.a. "split-brain" problem) does happen in production and must be accounted for. The cluster may become segmented because of temporary network problems when nodes (or groups of nodes) become isolated from the rest of topology. If not addressed, such segmentation can cause inconsistent clusters and, what’s worse, inconsistent data. GridGain addresses this issue by making sure that no inconsistent write is allowed into the system if segmentation occurred, while writes that are certain to be consistent are still allowed to proceed.

Each node checks network segment individually, using configured segmentation resolvers. Segmentation resolvers are pluggable and can be tailored to any environment. GridGain comes with several implementations out of the box. Segment check is performed in following cases:

Each segmentation resolver checks segment for validity. Typically, resolver should run a light-weight single check (i.e. one IP address or one shared folder). Compound segment checks may be performed using several resolvers. If segmentation resolver determines that local grid node belongs to incorrect segment, the node will act in accordance to configured segmentation policy. For details on available policies refer to documentation: GridSegmentationPolicy^doc.

Here is the list of GridConfiguration^doc properties intended to configure segmentation logic. All configuration parameters below are optional.

GridGain has the following built-in event types to notify on segmentation events:

GridGain has the following built-in segmentation policies.

GridGain has the following built-in segmentation resolvers. You can specify multiple resolvers, in which case all the specified resolvers will be checked. Use setAllSegmentationResolversPassRequired(boolean)^doc to make sure that all resolvers must pass segmentation check - otherwise segment is declared valid if it passes one of all segmentation resolver checks.