GridGain Developers Hub
GitHub logo GridGain iso GridGain.com
GridGain Software Documentation

Data Rebalancing

Overview

When a new node joins the cluster, some of the partitions will be relocated to the new node so that the data remains distributed equally in the cluster. This process is called data rebalancing.

If an existing node permanently leaves the cluster and backups are not configured, you will lose the partitions stored on this node. When backups are configured, one of the backup copies of the lost partitions will become a primary partition and the rebalancing process will be initiated.

When Native Persistence is enabled, data rebalancing does not happen automatically, but is triggered by the changes in the baseline topology. See the Baseline Topology and Cluster Activation section for more details.

Rebalancing is configured per cache.

Configuring Rebalancing Mode

GridGain supports both synchronous and asynchronous rebalancing. In the synchronous mode, any operation on the cache data is blocked until rebalancing is finished. In the asynchronous mode, the rebalancing process is done asynchronously. You can also disable rebalancing for a particular cache.

To change the rebalancing mode, set one of the following values in the cache configuration.

  • SYNC — Synchronous rebalancing mode. In this mode, any call to the cache public API will be blocked until rebalancing is finished.

  • ASYNC — Asynchronous rebalancing mode. Distributed caches will be available immediately and will load all necessary data from other available grid nodes in the background.

  • NONE — In this mode no rebalancing will take place, which means that caches will be either loaded on demand from the persistent storage whenever data is accessed, or populated explicitly.

<bean class="org.apache.ignite.configuration.IgniteConfiguration" id="ignite.cfg">
    <property name="cacheConfiguration">
        <list>
            <bean class="org.apache.ignite.configuration.CacheConfiguration">
                <property name="name" value="mycache"/>
                <!-- enable synchronous rebalance mode -->
                <property name="rebalanceMode" value="SYNC"/>
            </bean>
        </list>
    </property>
</bean>
IgniteConfiguration cfg = new IgniteConfiguration();

CacheConfiguration cacheCfg = new CacheConfiguration("mycache");

cacheCfg.setRebalanceMode(CacheRebalanceMode.SYNC);

cfg.setCacheConfiguration(cacheCfg);

// Start a node.
Ignite ignite = Ignition.start(cfg);
IgniteConfiguration cfg = new IgniteConfiguration
{
    CacheConfiguration = new[]
    {
        new CacheConfiguration
        {
            Name = "mycache",
            RebalanceMode = CacheRebalanceMode.Sync
        }
    }
};

// Start a node.
var ignite = Ignition.Start(cfg);

Configuring Rebalance Thread Pool

By default, rebalancing is performed in one thread on each node. It means that at each point in time only one thread is used to transfer batches from one node to another, or to process batches coming from the remote node.

You can increase the number of threads that will be taken from the system thread pool and used for rebalancing. A system thread is taken from the pool every time a node needs to send a batch of data to a remote node or needs to process a batch that came from a remote node. The thread is relinquished after the batch is processed.

<bean class="org.apache.ignite.configuration.IgniteConfiguration" id="ignite.cfg">

    <property name="rebalanceThreadPoolSize" value="4"/>

    <property name="cacheConfiguration">
        <list>
            <bean class="org.apache.ignite.configuration.CacheConfiguration">
                <property name="name" value="mycache"/>
            </bean>
        </list>
    </property>
</bean>
IgniteConfiguration cfg = new IgniteConfiguration();

cfg.setRebalanceThreadPoolSize(4);

CacheConfiguration cacheCfg = new CacheConfiguration("mycache");
cfg.setCacheConfiguration(cacheCfg);

// Start a node.
Ignite ignite = Ignition.start(cfg);

Rebalance Message Throttling

When rebalancing transfers data from one node to another, the whole data set is split into batches and each batch is sent in a separate message.

<bean class="org.apache.ignite.configuration.IgniteConfiguration" id="ignite.cfg">
    <property name="cacheConfiguration">
        <list>
            <bean class="org.apache.ignite.configuration.CacheConfiguration">
                <property name="name" value="mycache"/>
                <!-- Set batch size. -->
                <property name="rebalanceBatchSize" value="#{2 * 1024 * 1024}"/>
                <!-- Set throttle interval. -->
                <property name="rebalanceThrottle" value="100"/>
            </bean>
        </list>
    </property>
</bean>
IgniteConfiguration cfg = new IgniteConfiguration();

CacheConfiguration cacheCfg = new CacheConfiguration("mycache");

cacheCfg.setRebalanceBatchSize(2 * 1024 * 1024);
cacheCfg.setRebalanceThrottle(100);

cfg.setCacheConfiguration(cacheCfg);

// Start a node.
Ignite ignite = Ignition.start(cfg);
IgniteConfiguration cfg = new IgniteConfiguration
{
    CacheConfiguration = new[]
    {
        new CacheConfiguration
        {
            Name = "mycache",
            RebalanceBatchSize = 2 * 1024 * 1024,
            RebalanceThrottle = new TimeSpan(0, 0, 0, 0, 100)
        }
    }
};

// Start a node.
var ignite = Ignition.Start(cfg);

Other Properties

The following table lists the properties of CacheConfiguration related to rebalancing:

Property Description Default Value

rebalanceDelay

A delay in milliseconds before the rebalancing process starts after a node joins or leaves the topology. Rebalancing delay is useful if you plan to restart nodes or start multiple nodes at once or one after another and don’t want to repartition and rebalance the data until all nodes are started.

0 (no delay)

rebalanceBatchSize

The size in bytes of a single rebalance message. The rebalancing algorithm splits the data on every node into multiple batches prior to sending it to other nodes.

512KB

rebalanceThrottle

See Rebalance Message Throttling.

0 (throttling disabled)

rebalanceOrder

The order in which rebalancing should be done. Rebalance order can be set to a non-zero value for caches with SYNC or ASYNC rebalance modes only. Rebalancing for caches with smaller rebalance order will be completed first. By default, rebalancing is not ordered.

0

rebalanceTimeout

Timeout for pending rebalancing messages when they are exchanged between the nodes.

10 seconds

Monitoring Rebalancing Process