Free Download v4.5

Products










	
				
	
	

		
				
		

		
		
In-Memory Data Grid
		
		

		
		
In-Memory Data Grid is the core technology behind GridGain’s capability to process large data sets with low latency in Real Time context. Easily scaling from a single computer to terabytes of data and thousands of nodes GridGain In-Memory Data Grid technology provides capability to parallelize the data storage by storing partitioned data in in-process memory – the closest location the data can theoretically reside in relation to the application using it.


Having data as close to application as theoretically possible is what makes GridGain’s In-Memory Compute Grid able to process large data sets stored in In-Memory Data Grid with low latency in Real Time.




In-Memory Data Grids allow to treat grids and clouds as a single virtualized memory bank that smartly partitions data among the participating computers and providing various caching and accessing strategies.




Key Features


The goal of In-Memory Data Grids is to provide extremely high availability of data by keeping it in memory and in highly distributed (i.e. parallelized) fashion.
 GridGain In-Memory Data Grid subsystem is fully integrated into the core of GridGain and is built on top of the existing functionality such as pluggable auto-discovery, communication, and marshaling, peer-to-peer on demand class loading, and support for functional programming.


Among its key features are:







Distributed Java-based In-Memory Key-Value Store


GridGain’s in-memory data grid is a distributed object store similar in interface to a typical hash map. You store objects with keys. Unlike traditional systems where keys and values are often limited to byte arrays – in GridGain you can use any Java object (or Scala or Groovy) as either value or key. In fact, GridGain allows to keep multiple types of objects as keys or values in the same cache.


This gives tremendous flexibility by allowing to keep exactly the same object your business logic is dealing with in the data grid without an extra step of marshaling and demarshaling. It also simplified the usage of data grid as you can in most cases interface with distributed data store as with a simple hash map.






MVCC-Based Concurrency


GridGain’s in-memory data grid concurrency is based on advanced implementation of MVCC (multiversion concurrency control) – the same technology used by practically all database management systems. MVCC provides practically lock free concurrency management by maintaining multiple version of data instead of wide-scope locks. MVCC provides a backbone for high performance and overall system throughput for GridGain-based systems under the load.






Pluggable Segmentation Resolution


Network Segmentation (aka split-brain) is a condition of the network when due to connectivity issues grid splits into multiple independent subgrids which do not know about each other. This does not happen often, but it does happen, especially on larger networks joined by various hubs and switches along the way. The danger of this situation is that these segmented grids continue to operate as usual without even knowing that there are other independent segments doing the same thing. This leads to various data consistency issues, because the same piece of data is now going to be updated by several segmented grid independently.


Various products treat this issue differently, some allowing for multiple writes to happen and then attempting to merge the dirty inconsistent pieces of data together into a consistent state. The biggest disadvantage of this approach is that it actually does allow for dirty write or read to occur. At GridGain we took a path where dirty writes or reads are impossible regardless of how segmented your grid gets. To achieve it we introduced a notion of “good segment”. A good segment is a segment which has connectivity to all vital deployment components within your system, for example connectivity to Database, file system, web server, etc… User gets to define and configure what components are important for his system to continue operate. We call these configurable checks “segmentation resolvers”. GridGain comes with several segmentation resolvers out of the box and user can always add more if needed.


When segmentation resolvers are configured, only the segment for which all segmentation resolvers returned success is considered “good segment” and other segments are considered “bad segments”. Good Segments are allowed to continue operate even after network segmentation occurred, while bad segments will be taken offline and will rejoin only after network connectivity is restored to normal. This approach guarantees that data is always consistent and system continues to operate even in case of large network failures.






Cache Modes


You can have as many named caches as you like and each can be of LOCAL, REPLICATED, or PARTITIONED type.


LOCAL mode is the most light weight mode of cache operation, as no data is distributed to other cache nodes. It is ideal for scenarios where data is either read-only, or can be periodically refreshed at some expiration frequency. It also works very well with read-through behavior where data is loaded from persistent storage on misses. Other than distribution, local caches still have all the features of distributed cache, such as automatic data eviction, expiration, disk swapping, data querying, transactions, and more.


REPLICATED mode provides the utmost availability as data is available on every grid node. However, in this mode every data update must be propagated to all other nodes which can have an impact on performance and scalability. As the same data is stored on all grid nodes, the size of replicated cache is limited by the amount of memory available on a node. This mode is ideal for scenarios where updates are infrequent and data availability is most important.


PARTITIONED cache is the most scalable distributed cache mode. In this mode the overall data set is divided equally into partitions and all partitions are split equally between participating nodes, essentially creating one huge distributed memory for caching data. This approach allows for storing as much data as can be fit in the total memory available across all nodes, hence allowing for loading gigabytes and terabytes of data into cache memory. Partitioned cache is always fronted by a smaller local cache, also known as Near cache, which stores most recently or most frequently accessed data. Such combination provides for high availability of data that is accessed often together with high scalability of partitioned cache. This mode is ideal for scenarios where data volumes are large and updates are relatively frequent.






Pluggable Expiration Policies


GridGain provides LRU, LFU, FIFO, LIRS, time based and random expiration policies out of the box. What is unique about GridGain is that it provides you an ability to plug any other custom-made eviction policy.


Eviction policies are very important in distributed caches as they essentially provide the means of smart controlling the size of the caches on individual nodes. In some cases the built-in implementations like LRU and LFU fit the requirements – but in many others it’s more efficient to provide your own version that account for specific business-level requirements such as date and time, specific processing cycles, and various data affinities.






Synchronous and Asynchronous Cache Operations


For every synchronous computation (e.g. MapReduce task) on compute grid and for every synchronous cache operation there is an asynchronous counter part. In distributed systems I/O may become an overhead and ability not to wait for results synchronously adds abilities to execute multiple operations in parallel and react to responses as they begin to come in. All GridGain asynchronous operations allow attaching listeners to them just so result notifications are also asynchronous.






HyperLocking Technology


HyperLocking is one of main ways to improve performance of GridGain 2-phase-commit (2PC) transactions. It is enabled on per-transaction level. Transactions with HyperLocking can be started via standard GridCacheProjection.txStartAffinity(...) and GridCacheProjection.txStartPartition(...) methods. What HyperLocking allows you to do is to minimize number of locks acquired within transactions.


In standard 2PC approach a separate lock will be acquired, either optimistically or pessimistically, for every key within transaction. If keys are backed up on other nodes, then these locks will have to be propagated to backup nodes, sequentially, imposing significant latencies and network overhead.


With HyperLocking is used, only one lock per transaction is acquired even though transaction may be modifying 1000s of cache entries. In pessimistic mode the lock is acquired at the beginning of transaction, and in optimistic mode it is acquired at commit phase. The requirement is that all elements within transaction are grouped by the same affinity key or partition ID. In this case all backups are guaranteed to be on the same node and GridGain is capable to complete 2PC transaction with just one network trip to backup node(s) making it dramatically faster than with a default lock-per-key transaction semantic. Hyperlocking can provide up to 100x performance boost over standart 2PC approach. This is one of the reasons why GridGain 2PC transactional approach outperforms most of Eventually Consistent (EC) and NoSql products by an order of magnitude.






JTA/JCA Integration


For JEE environments like application servers GridGain provides automatic integration with JTA/XA. Essentially GridGain becomes an XA resource and will automatically check if there is an active JTA transaction present. If it is, then GridGain transaction will automatically join JTA transaction, if there is no active JTA transaction, then GridGain transaction will complete on its on. Such transparent integration with JTA/XA allows users to use the same GridGain APIs regardless of the environment GridGain is running in.






Pluggable Eviction Policies


Selecting proper cache eviction strategy is one of the main parts of cache configuration. Generally, eviction controls the maximum number of elements that can be stored in cache, just so cache does not grow indefinitely. However, every eviction strategy will evict elements in different order and selecting a wrong strategy can have a significant impact on cache hit ratio and performance.


It is also important to note that eviction policy is pluggable in GridGain, and users can plug their own eviction policy whenever none of the ones provided out of the box is adequate. Out of the box GridGain comes with LRU, FIFO, LIRS, and Random eviction policies.


Whenever cache swap space is enabled, evicted entries will be overflown to a disk-based swap storage can store significantly more data than can fit in the memory.






Pluggable Swap Storage


Swap storage provides a mechanism in grid by which grid cache or any other component can store data outside of JVM heap (i.e. on disk or off-heap memory) and retrieve it later. GridGain cache uses swap space to overflow data if it cannot fit it in memory. All data stored in swap space continues to be indexed (often in memory) so it can be efficiently retrieved for query execution.


By default GridGain ships with LevelDB-based swap space storage, but other implementations can be plugged in if desired.






Write-through and Read-through Behavior


Transparent integration with persistent data stores is extremely important and GridGain provides a simple API to allow automatic read-through and write-through behavior. Read-through means that data will be read from persistent store whenever it’s not available in cache, and write-through means that data will be automatically persisted whenever it is updated in cache.


Note that there is also refresh-ahead mode which allows for automatic preloading of values in the background before they get accessed. This feature ensures that entries are always automatically re-cached whenever they are nearing expiration.






Advanced partitioned cache modes


Near and Partitioned

This is default mode which represents a scenario where a bigger partitioned cache is fronted with local near cache (nothing needs to be configured to enable this mode). In this case if data is available locally, it will be provided from either local near or partitioned cache. However, if data is not present in near or partitioned caches, then it will be fetched from remote partitioned cache which owns the data and then cached locally in near cache. However, to avoid moving of data it is best to colocate computations with data based on data affinity supported by GridGain to make sure that all cache operations are local and there is no redundant data movement.


Near Disabled

This mode is configured via GridCacheConfiguration.isNearEnabled() configuration property which is true by default. In this mode a GridGain node only participates in partitioned caching and does not perform near caching. This means that if data is not available in local partitions, it will be fetched from remote node which has the data, but it will not be cached locally. However, this mode is mostly used with data affinity where computations that perform operations on data are sent exactly to the node where the data is. In this case, when computation arrives, all the data it works with is cached locally and there are no remote cache fetch operations. This mode is absolutely recommended when working with data affinity and colocation.


Near Only

This mode is configured via GridCacheConfiguration.isNearOnly() configuration property and is turned off by default. When enabled, GridGain node will not participate in main partitioned caching, but will only cache most recently accessed items in the Near Cache. Near cache is generally smaller in size and has a more eager eviction policy to make sure that only recent items are cached. Although sometimes useful, this mode goes against collocation of computations and data philosophy recommended in GridGain as in this case instead of moving computations to data we are actually moving data to the computations. In systems requiring high performance it is best to disable near cache and use data affinity.






Write-Behind Caching


In a simple write-through mode each cache put and remove operation will involve a corresponding request to the storage and therefore the overall duration of the cache update might be relatively long. Additionally, an intensive cache update rate can cause an extremely high storage load.


For such cases GridGain offers an option to perform asynchronous storage update also known as write-behind. The key concept of this approach is to add a persist request to the queue and postpone data persistence to a certain point in future. The actual data persistence can be triggered by time-based events (the maximum time that data entry can reside in the queue is limited), by queue-size events (the queue is flushed when it’s size reaches some particular point), or by using both of them in combination in which case either event will trigger the flush.


What benefits does write-behind cache store provide? In addition to obvious performance benefits, because cache writes simply become faster, this approach scales a lot better as long as your application can tolerate delayed persistence updates. When number of nodes in data grid grows and every node performs frequent updates, it is very easy to overload the underlying system of records, like database. Write-behind approach allows to maintain high throughput of writes in the system without bottlenecking at the persistence layer. Moreover, cache can continue operating even if your database crashes or goes down. In this case the persistence queue will keep storing all the updates until the database comes back up.


Another advantage is natural integration with warehouse data store systems, such as Hadoop HDFS. This configuration allows to cache most recent transacted data in memory to efficient querying and reporting, and then flush it to HDFS whenever it becomes less relevant for offline batch oriented analysis.






Concurrent & Transactional Cache Preloading


Preloading newly started cache nodes is important whenever it is necessary to have common data set in memory on all nodes. When preloading is enabled, distributed caches will attempt to preload all necessary values from other grid nodes. We have taken significant amount of care to make sure that cache preloading is absolutely concurrent and imposes no overhead over normal cache performance. GridGain supports synchronous and asynchronous preloading modes.


In synchronous mode distributed caches will not start until all necessary data is loaded from other available grid nodes. Essentially existing data grid nodes will keep operating as usual, but the new nodes will not be allowed to start until they have all the necessary data.


If asynchronous mode is turned on, which is default behavior, distributed caches will start immediately and will load all necessary data from other available grid nodes in the background. This mode is most efficient preloading mode and should be usually used unless application logic has strong requirements for synchronous mode.


Note that REPLICATED caches will try to load the full set of cache entries from other nodes, while PARTITIONED caches will only load the entries for which current node is primary or backup.






Datacenter Replication


When working with multiple data centers it is often important to make sure that if one data center goes down, another data center is fully capable of picking its load and data. Data center replication is meant to solve exactly this problem. When data center replication is turned on, GridGain data grid will automatically make sure that each data center is consistently backing up its data to other data centers (there can be one ore more).




GridGain supports both active-active and active-passive modes for replication. In active-active mode both data centers are fully operational online and act as a backup copy of each other. In active-passive node, only one data center is active and another data center serves only as a backup for the active data center.


Datacenter replication can be either transactional or eventually-consistent. In transactional mode, a data grid transaction will be considered complete only when all the data has be replicated to another datacenter. If the replication step failed, then the whole transaction will be rolled back on both datacenters. In eventually consistent mode transaction will usually complete before the replication finished. In this mode the data is usually concurrently buffered on one data center and then gets flushed to another data center either when buffer fills up or when certain time period elapses. Eventually consistent mode is generally a lot faster, but it also introduces a lag between updates on one data center and data being replicated to another.


If one of the datacenters goes offline, then another will immediately take responsibility for it. Whenever the crashed data center goes back online then it will receive all the updates it has missed from another data center.






JDBC Driver For In-Memory Object Data Store


One of the biggest additions to GridGain in 4.2 release is inclusion of JDBC driver that you can use to query your data in GridGain’s In-Memory Data Grid. Now, this is a pretty big deal. No custom languages, standard SQL and hunders of existing tools can be used to query & examine the data in your data grid.




Here’s the quick example. Notice how Java code looks 100% identical as if you talk to a standard SQL database – yet you are working in in-memory data platform:


// Register JDBC driver.
Class.forName("org.gridgain.jdbc.GridJdbcDriver");

// Open JDBC connection.
conn = DriverManager.getConnection(
    "jdbc:gridgain:/ / localhost/" + CACHE_NAME,
    configuration()
);

// Create prepared statement.
PreparedStatement stmt = conn.prepareStatement(
    "select name, age from Person where age >= ?"
);

// Configure prepared statement.
stmt.setInt(1, minAge);

// Get result set.
ResultSet rs = stmt.executeQuery();





Delayed Preloading


GridGain 4.2 introduced support for delayed preloading. Essentially, whenever a new node joins the grid or an existing node leaves the grid, cluster repartitioning happens. This basically means that, in case of new node, it has to take responsibility for some of the data cached on other nodes, and in case of node leaving the grid, other nodes have to take responsibility for the data cached on that node. Essentially this results in data movement between data grid nodes.


Now imagine that you need to bring multiple nodes up concurrently. The 1st node that comes up will take responsibility for some portion of the data cached on other nodes and will start loading that portion of the data from other nodes. When a 2nd node comes up, it will also take responsibility for some portion of the data, including some data from the 1st node that was just started, and now portion of the data that was moved to 1st node will have to be moved to the 2nd node. Wouldn’t it be more efficient to wait till 2nd node comes up to start data preloading? The same happens when nodes 3, 4, etc… come up. So the most efficient way to do preloading of keys and to avoid extra network traffic causes by moving data between newly started nodes is to delay preloading until the last node starts.


Delayed preloading allows for delayed or manual preloading start from API or from Visor DevOps Console.






Cache Transactions


GridGain has full support for distributed data grid transactions which span data cached on local and remote nodes. While automatic enlisting into JEE/JTA transactions is supported, GridGain data grid also allows users to create more light-weight cache transactions which are often more convenient to use. GridGain cache transactions support all ACID properties that you would expect from any transaction, including support for Optimistic and Pessimistic concurrency levels and Read-Committed, Repeatable-Read, and Serializable isolation levels.


Note that if a persistent data store is configured, then data grid transaction will also span the data store, i.e. the cache transaction will either succeed as as a whole if data store transaction succeeded, and will fail as a whole if data store transaction failed.






OOP and FP-based APIs for Java, Scala and Groovy


GridGain is first and formost based on Java which is an OOP language – so all the GridGain APIs work very well with standard OOP principles. However, you will quickly discover that even when working with Java APIs you are often dealing with functional constructs, like closures or predicates, even though Java is not a functional language. This is one of the unique sides of GridGain and it leads to extremely elegant and simple to use APIs.


The functional side of GridGain comes from well thought out functional library which we first built for our internal use and then added to our public APIs simply because it allowed us to express much more functionality with fewer lines of code. Our functional library has its own predicates, closures, monadic operations, and many other functional features which can be used on GridGain in distributed fashion and can be executed and auto-deployed across the grid.






Structured and unstructured data


In-memory data grid in GridGain provides object key-value store, i.e. both key and value can be any arbitrary Java objects. Moreover, a single cache can store heterogenous objects of any time. GridGain also provides pluggable indexing SPI that allows user to introduce any type of indexing into GridGain querying capability. All this functionality enabled GridGain to easily process both structures and unstructured (document-oriented) data.






Master/Master data replication/invalidation in synch/asynch modes


Regardless of which cache mode is used, REPLICATED or PARTITIONED, whenever cached data changes the new state needs to be propagated to some remote cached nodes. In case of REPLICATED cache, all other nodes need to be updated, and in case of PARTITIONED cache the nodes that serve as backups for changed data and the relevant Near caches on remote nodes need to be updated. By default the new state will be moved to remote node which requires sending the whole new state across. However, whenever new state is too large, or whenever it is OK to load state on demand on remote nodes, you can configure caches to work in Invalidation mode in which case only small invalidation messages will be sent across. Upon receiving invalidation message, the node will simply remove the stale reference from cache and it will have to be loaded on demand upon next access from either primary node or persistent data store.


Note that in case of PARTITIONED mode, only relevant Near caches on remote nodes will be invalidated – the back up state will always be copied to the backup nodes.






Tiered storage with on-heap, off-heap, swap space, SQL, and Hadoop


GridGain provides advanced tiered storage technology. You can store your data in:


    

  • on-heap JVM memory
    • 

  • off-heap JVM memory
    • 

  • pluggable swap-space
    • 

  • JDBC-compliant SQL store or Hadoop HDFS/HBase
    • 



This flexibility enables developers to choose the best strategy and the best balance between speed of access and capacity characteristics that fit their application best. With pluggable swap space (and default implementation based on Google’s LevelDB) and pluggable cache store (supporting our-of-the-box SQL and HBase stored) supporting seamless write-through and read-through behavior the developers enjoy simple and fully automated persistence for their data in memory.






Affinity Routing With Compute Grid


Affinity routing is one of the key concepts behind Compute and Data Grid technologies (whether they are in-memory or disk based). In general, affinity routing allows to co-locate a job and the data set this job needs to process.


The idea is pretty simple: if job and data are not co-located, then job will arrive on some remote node and will have to fetch necessary data from yet another node where data is stored. Once processed this data will most likely will have to be discarded (since it’s already stored and backed up elsewhere). This process induces expensive network trip plus all associated marshaling and demarshaling. At scale – this behavior can bring almost any system to a halt.


Affinity co-location solves this problem by co-locating job with its necessary data set. We say that there is an affinity between a processing (i.e. job) and the data that this processing requires – and therefore we can route the job based on this affinity to a node where data is stored to avoid unnecessary network trips and extra marshaling and demarshaling.


GridGain provides advanced capabilities for affinity co-location: from a simple single-method call to sophisticated APIs supporting complex affinity keys and non-trivial topologies.






Partitioned cache with active replicas


Partitioned cache support active replicas. This feature allows reads from cache to be actively distributed across replicas/backup nodes leading to much better scalability for read operations. Active replicas are extremely important in read-mostly partitioned caches under the load – where this design can achieve significant performance increase without any extra configuration or user coding.






Pluggable data indexing support


Indexing functionality in GridGain allows for creation of in-memory indexes of various pieces of cached data so it can be efficiently used in SQL queries. GridGain default indexing implementation is based on H2 SQL engine, which gives our users fully ANSI compliant SQL queries for in-memory data. For its indexing GridGain supports unique constraints, ordered and unordered indexes, compound indexes, and also Text indexing based on Apache Lucene or H2Text.


One of the biggest advantages that pluggable indexing provides is ability to plug other, non-SQL languages if need be, which essentially allows our users to create customer query DSLs for querying their data.






BigMemory Support


Traditionally JVM has been very good with Garbage Collection (GC). However, when running with large amounts of memory available, GC pauses can get very long. This generally happens because GC now has a lot more memory to manage and often cannot coop without stoping your application completely (a.k.a. lock-the-world pauses) and allowing itself to catch up. In our internal tests with Heap size set to 60G or 90G GC pauses some times were as long as 5 minutes. Traditionally this problem was solved by starting multiple JVMs on the same physical box, but that does not always work very well as some applications want to colocate large amounts of data in one JVM for faster processing.




To mitigate large GC pauses, GridGain introduce support for BigMemory where data is allocated off-heap instead of on-heap. By allocating data off-heap, JVM GC does not know about it and hence does not slow down. In fact you can start your Java application with a relatively small heap, e.g. below 512M, and then let GridGain utilize 100s of Gigabytes of memory as off-heap data cache. Whenever data is first accessed, it gets cached in on-heap memory. Then, after certain period of non-use, it gets placed into off-heap memory cache. If your off-heap memory gets full, the least used data will be optionally evicted to disk-overflow-store, also called swap store.


One of the distinguishing characteristics of GridGain Off-Heap memory is that the on-heap foot memory foot print is constant and does not grow with the size of the data. Also, the off-heap memory foot print of an off-heap cache entry has very little overhead, which means that you can fit more data in memory.






In-Memory Queries


What use would be from caching all the data in memory if you could not query it? GridGain supports a variety of different ways to query in-memory data, including standard SQL-based queries, Lucene-based text queries, or plain full-scan queries whenever needed.


SQL query type allows to execute distributed cache queries using standard SQL syntax. There are almost no restrictions as to which SQL syntax can be used. All inner, outer, or full joins are supported, as well as rich set of SQL grammar and functions. Ability to join different classes of objects stored in cache or across different caches make GridGain queries very powerful. On top of that, all indexes are usually kept in memory which result in very low latencies for query execution.


Full-scan queries as useful when it is known in advance that SQL query will cause a full data scan, or whenever data set is relatively small. With this query type GridGain will iterate over all cache entries, skipping over the entries that don’t pass the optionally provided key or value filters.


Text queries are useful when working with unstructured text data. GridGain will index such data using either Lucene engine or H2Text engine for efficient low latency querying of large text data volumes.


If there is no need to return result to the caller, you can save on network overhead by visiting all query results directly on remote nodes with GridGain Visiting query mode. With this method, all the logic is performed inside of query directly on the remotely queried nodes without sending any queried data to the caller.




Example


This is a full source of example that is putting 20 values into distributed partitioned cache (spread over any number of available nodes) and then executes 20 distributed closures that get properly co-located with the data in in-memory data grid.


In Java:


public class Demo {
    private static final int keyCnt = 20;

    private static Grid g;

    public static void main(String[] args) throws Exception {
        g = G.start("examples/config/spring-cache.xml");

        try {
            GridCache cache =
                g.cache("partitioned");

            populate(cache);
            visit(cache);

        }
        finally {
            G.stop(true);
        }
    }

    private static void visit(final GridCache c)
        throws GridException {
        for (int i = 0; i < keyCnt; i++) {
            final int key = i;

            g.run(GridClosureCallMode.BALANCE,
                new CA() {
                    @GridCacheName
                    private String cacheName = "partitioned";

                    @GridCacheAffinityMapped
                    public int affinityKey() {
                        return key;
                    }

                    @Override public void apply() {
                        System.out.println("Co-located [key= " + key +
                            ", value=" + c.peek(key) + ']');
                    }
                });
        }
    }

    private static void populate(GridCache c)
        throws GridException {
        for (int i = 0; i < keyCnt; i++)
            c.put(i, Integer.toString(i));
    }
}



In Scala:


object Demo {
    val range = 0 to 20 by 1

    lazy val c = cache$[Int, Int]("partitioned").get

    def main (args: Array[String]): Unit =
        scalar("examples/config/spring-cache.xml") {
            populate
            visit
        }

    def populate = range foreach {
        i => c.put(i, i)
    }

    def visit = grid$ run$ (
        BALANCE,
        range map (
            i => new CA {
                @GridCacheName
                def cacheName = "partitioned"

                @GridCacheAffinityMapped
                def affKey = i

                def apply = println("Co-located [key=" + i +
                    ", value=" + c.peek(i) + ']')
            }.scala
        )
    )
}



More Information