Gigaspaces XAP multi site replication

Vaibhav Hajela

10 min readMay 26, 2022

Multi-Region Replication for Disaster Recovery

Replicate state between different deployed Spaces, where each Space can be physically located in a different geographical location.

Why Multiple site replication?

Disaster recovery.
Maintaining Global data per site.

WAN Gateway Features

The GigaSpaces WAN Gateway features the following:

Total data consistency, ordering with no data loss
Conflict resolution support — Identifies data conflicts, enables user to decide whether override the local copy reject remote copy or merge.
Monitoring and Management — The WAN Gateway exposes statistics on every activity going through the replication to remote sites.
Indirect replication
Data filtering

Topologies

Two sample multi-site replication topologies. Well there are many other possible topologies too.

Multi-master with two sites, where each site is active and updates its subset of the data.
Master-slave, where only one site actually updates the data while the rest either serve as a backup or use it in read only mode.

For both of the above topologies, replication is done in in a similar way. Each Space replicates the relevant data to its target Space(s) via a local gateway, which routes the data to the gateway of the target Space(s) and from there to the target Space. The data is replicated asynchronously in a reliable mode, which means that even if a primary Space instance fails on the source site, the backup Space instance that replaces it will immediately take control and replicate the missing data along with new data that has been generated on the newly elected primary Space instance. This is very similar to the Mirror Service replication scheme. The gateway is discussed in full below.

Replication can use Hub & Spoke, Ring, Hierarchical or Pass-Through architecture:

Configuring a Space With Gateway Targets

A replication from one Space to another is basically a replication from a Space to a target gateway.
The source Space is decoupled from the target Space. Instead, it is configured to replicate to the target Space’s local gateway; the local gateway is in charge of dispatching the incoming replication packets to the relevant target Space.
Each Space needs to be made aware of the target gateways to which it replicates the data, by specifying the target gateways in the source Space configuration.
From the Space’s perspective, a replication from one Space to a target gateway is like any other replication channel (such as mirror or backup), and the backlog and confirmation state of the replication channel to the target gateway is kept in the source Space. As a result, the gateway is stateless as far as holding the replication state is concerned, and only the source Space is in charge of the replication progress and state. Each gateway has a unique logical name that usually corresponds to the physical site name (for example, “London” or “New York”). This name is used by remote Spaces that replicate to this Space via its local gateway.

The following snippet shows how to configure a Space that resides in New York to replicate to two other Spaces, one in London and one in Hong Kong:

<os-core:embedded-space id="space" space-name="myNYSpace" gateway-targets="gatewayTargets"/><os-gateway:targets id="gatewayTargets" local-gateway-name="NEWYORK">
  <os-gateway:target name="LONDON"/>
  <os-gateway:target name="HONGKONG"/>
</os-gateway:targets>

Each replication channel to the gateways can be configured with more parameters, and these parameters can applied to all gateways or specifically per gateway, for example:

<os-core:embedded-space id="space" space-name="myNYSpace" gateway-targets="gatewayTargets"/><os-gateway:targets id="gatewayTargets"
  local-gateway-name="NEWYORK" bulk-size="1000"
  max-redo-log-capacity="1000000">
  <os-gateway:target name="LONDON" />
  <os-gateway:target name="HONGKONG" bulk-size="100"/>
</os-gateway:targets>

Here, we specified a global bulk size of 1000 but have specifically overridden it in the replication channel to Hong Kong with 100, and have a global maximum redo log capacity for both targets of 1000000.

For more details about all the available configuration elements of the space gateway targets, refer to Configuring Targets.

Use the partitioned cluster schema

You should have the partitioned cluster schema used with the Space to enable replication to the gateway. If you are not interested in having backups running but have the replication to the gateway running, you should have ZERO as the number of backups. See the following example of an sla.xml configuration you can use in such a case:

<os-sla:sla cluster-schema="partitioned" number-of-instances="1" number-of-backups="0">
</os-sla:sla>

When there are no backups, running any failure of the primary might cause a loss of data.

Configuring and Deploying the Gateway

A gateway needs to be deployed locally as a Processing Unit in each site, and is composed of two different components; the delegator and the sink. The delegator is in charge of delegating outgoing replication from the local site to a remote gateway, and the sink is in charge of dispatching incoming replication from remote sites into the local Space.

Gateway Lookup

The different gateway components need to locate each other across the different sites. For example, a delegator needs to locate the sink of the target gateway to which it delegates the replication. This lookup is accomplished via dedicated lookup services that the gateways use to register and locate each other. This lookup process is usually done across the WAN, so the most reasonable way of locating the lookup services is using unicast (multicast is also supported). The following example demonstrates a unicast lookup discovery.

Gateway Basic Configuration

Following the above example, here we demonstrate how to configure the local gateway processing unit in New York, which needs to send replication to London and Hong Kong and also receive replication from these two sites.

<os-gateway:delegator id="delegator"
  local-gateway-name="NEWYORK" gateway-lookups="gatewayLookups">
  <os-gateway:delegations>
    <os-gateway:delegation target="LONDON"/>
    <os-gateway:delegation target="HONGKONG"/>
  </os-gateway:delegations>
</os-gateway:delegator><os-gateway:sink id="sink"
  local-gateway-name="NEWYORK"
  gateway-lookups="gatewayLookups"
  local-space-url="jini://*/*/myNYSpace">
  <os-gateway:sources>
    <os-gateway:source name="LONDON" />
    <os-gateway:source name="HONGKONG" />
  </os-gateway:sources>
</os-gateway:sink><os-gateway:lookups id="gatewayLookups">
  <os-gateway:lookup gateway-name="NEWYORK"
    host="ny-gateway-host-machine" discovery-port="10001"
    communication-port="7000" />
  <os-gateway:lookup gateway-name="LONDON"
    host="london-gateway-host-machine" discovery-port="10002"
    communication-port="8000" />
  <os-gateway:lookup gateway-name="HONGKONG"
    host="hk-gateway-host-machine" discovery-port="10003"
    communication-port="9000" />
</os-gateway:lookups>
<!--The above ports and host names are randomly selected and
have no meaning, all sites could designate the same ports as well-->

In the above example, we see that both the sink and the delegator need a reference to the gateway lookup configuration, because both components are using this configuration to locate the relevant component or to register themselves. They use their local gateway name to identify themselves to the lookup configuration, where they should be registered and where they should look for their targets.
The delegator and sink components are actually isolated and can even be deployed in separate Processing Units, but the most simple deployment is to bundle them two together. However, in some cases you may want to separate them across two or more machines due to system load or other reasons.

For full details and available configurations, refer to Gateway Components.

Gateway and Distributed Transactions

By default, the gateway preserves the atomicity of distributed transactions (distributed transaction consolidation). This can be disabled by adding the following property to the Space configuration:

<os-core:embedded-space id="space" space-name="localSpace" gateway-targets="gatewayTargets">
  <os-core:properties>
    <props>
      <prop key="cluster-config.groups.group.repl-policy.processing-type">
        global-order
      </prop>
    </props>
  </os-core:properties>
</os-core:embedded-space>

Distributed transaction consolidation is done by waiting for data from all the transaction participants before processing is done by the sink component. In some cases, the data from certain distributed transaction participants may be delayed due to network delay or disconnection, and this can cause delays in replication. In order to prevent this potential delay, you can set a timeout parameter that indicates how much time to wait for distributed transaction participant data before processing the data individually for each participant.

You can specify the behavior when processing is split into individual participants upon consolidation failure (timeout or other reasons); the unconsolidated transaction can be either aborted or committed.

While waiting for the pieces of a distributed transaction to arrive at the sink, replication isn’t waiting but is keeping the data flow and preventing conflicts from happening.

The following example shows how to set the timeout for waiting for distributed transaction data to arrive. You can also set the amount of new operations to perform before processing data individually for each participant.

<os-gateway:sink id="sink" local-gateway-name="NEWYORK"
  gateway-lookups="gatewayLookups"
  local-space-url="jini://*/*/myNYSpace">
  <os-gateway:sources>
    <os-gateway:source name="LONDON" />
    <os-gateway:source name="HONGKONG" />
  </os-gateway:sources>
  <os-gateway:tx-support
     dist-tx-wait-timeout-millis="10000"
     dist-tx-wait-for-opers="20"
     dist-tx-consolidation-failure-action="commit"/> <!--or "abort"-->
</os-gateway:sink>

Distributed transaction participant data will be processed individually if 10 seconds have passed and not all of the participant data has arrived. or if 20 new operations were executed after the distributed transaction.

AttributeDefault Valuedist-tx-wait-timeout-millis60000 millisecondsdist-tx-wait-for-opersunlimited (-1 = unlimited)dist-tx-consolidation-failure-actioncommit

If you set the cluster-config.groups.group.repl-policy.processing-type property to global-source, all reliable asynchronous replication targets for that Space will work in non-distributed transaction consolidation mode. For example, a Mirror will also work in non-distributed transaction consolidation mode.

Consolidation failure can occur under normal circumstances, if the target gateway is restarted or crashes during the consolidation process. For example, you may have a scenario where the transaction was successfully consolidated and executed on the target cluster, but the gateway was stopped while sending confirmation to the transaction participants in the source site and some of them received the confirmation while others did not. In this case, the transaction is actually successfully executed in the target site, and by default when the consolidation failure event occurs, the unconfirmed part reaches the conflict resolution handler will abort it (by default), and the state will remain consistent.

Setting both dist-tx-wait-timeout-millis and dist-tx-wait-for-opers to unlimited (or to very high value) is risky. The replication backlog may accumulate due to a packet that is unconsolidated, causing the replication process to wait for consolidation that may never occur.

Master-Slave Topology

With this architecture, we have a master-slave topology where all data is manipulated in one site, and two other sites read the data but don’t update it. In other words, the “other sites” — the slaves — should not replicate data to the other gateways.

In this case, New York’s site will be the active site while London and Hong Kong will be the passive sites. As explained before, being passive does not necessarily mean no work is done in these sites. However, in terms of replication over the WAN, these sites should not replicate to the other sites and usually should not alter data replicated from other sites, because it may cause conflicts.

Like all GigaSpaces Processing Units, the configuration details of each of the above components is placed in a pu.xml file. These are the contents of the file for each component:

New York Space
New York Gateway
London Space
London Gateway
Hong Kong Space
Hong Kong Gateway

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:os-core="http://www.openspaces.org/schema/core"
  xmlns:os-events="http://www.openspaces.org/schema/events"
  xmlns:os-remoting="http://www.openspaces.org/schema/remoting"
  xmlns:os-sla="http://www.openspaces.org/schema/sla"
  xmlns:os-gateway="http://www.openspaces.org/schema/core/gateway"
  xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
    http://www.openspaces.org/schema/core http://www.openspaces.org/schema/16.1.1/core/openspaces-core.xsd
    http://www.openspaces.org/schema/events http://www.openspaces.org/schema/16.1.1/events/openspaces-events.xsd
    http://www.openspaces.org/schema/remoting http://www.openspaces.org/schema/16.1.1/remoting/openspaces-remoting.xsd
    http://www.openspaces.org/schema/sla http://www.openspaces.org/schema/16.1.1/sla/openspaces-sla.xsd
    http://www.openspaces.org/schema/core/gateway http://www.openspaces.org/schema/16.1.1/core/gateway/openspaces-gateway.xsd">  <os-core:embedded-space id="space" space-name="myNYSpace" gateway-targets="gatewayTargets"/>  <os-gateway:targets id="gatewayTargets" local-gateway-name="NEWYORK">
    <os-gateway:target name="LONDON"/>
    <os-gateway:target name="HONGKONG"/>
  </os-gateway:targets>
</beans>

Multi-Master Topology

With this architecture, we have a multi-master topology where data is generated and manipulated in all sites.

The example below describes two sites, but any number of sites is supported in the same manner. In a master-slave topology, each site should try to modify different subsets of the data as much as possible, because many conflicts can occur if multiple sites change the same Space entries at the same time. These conflicts can be resolved using a conflict resolver, which is described in detail in the Multi-Site Conflict Resolution topic.

In the example below, New York and London are the two active sites.

Here are the contents of the file for each component:

New York Space
New York Gateway
London Space
London Gateway
Symmetric Gateway Config

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns:os-core="http://www.openspaces.org/schema/core"
    xmlns:os-events="http://www.openspaces.org/schema/events"
    xmlns:os-remoting="http://www.openspaces.org/schema/remoting"
    xmlns:os-sla="http://www.openspaces.org/schema/sla"
        xmlns:os-gateway="http://www.openspaces.org/schema/core/gateway"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.openspaces.org/schema/core http://www.openspaces.org/schema/16.1.1/core/openspaces-core.xsd
       http://www.openspaces.org/schema/events http://www.openspaces.org/schema/16.1.1/events/openspaces-events.xsd
       http://www.openspaces.org/schema/remoting http://www.openspaces.org/schema/16.1.1/remoting/openspaces-remoting.xsd
       http://www.openspaces.org/schema/sla http://www.openspaces.org/schema/16.1.1/sla/openspaces-sla.xsd
       http://www.openspaces.org/schema/core/gateway http://www.openspaces.org/schema/16.1.1/core/gateway/openspaces-gateway.xsd">        <os-core:embedded-space id="space" space-name="myNYSpace" gateway-targets="gatewayTargets"/>        <os-gateway:targets id="gatewayTargets" local-gateway-name="NEWYORK">
          <os-gateway:target name="LONDON"/>
        </os-gateway:targets></beans>

The Multi-Master running example page in the Service & Best Practices section includes a three-way setup that replicates data between three sites.

Filtering Replication Between Gateways

On occasion, there may be data that should not be replicated between sites but should still be replicated locally to the backup or a mirror service. If this is the case, it isn’t suitable to specify that the object should not be replicated. A replication channel to a gateway is like any other replication channel, so a custom Replication Filter at the source Space can be used to filter the relevant data from being sent to the target gateway. This filtering should be based on the replication target name, in order to identify that the replication filter is called for the correct outgoing replication to the gateway.

For full details and an example, refer to Filtering Data.

Bootstrapping a Site From Another Site

To bootstrap a site from another site is to restart a site Space from scratch, and to populate it with data from another site Space. This can be useful after a very long disconnection where the replication redo-log in the source Spaces that replicate to this site was dropped due to breaching capacity limitations, and the disconnected site should therefore start fresh. Other reasons include an explicit planned downtime due to site maintenance that leads to a complete system bootstrap upon restart.

For more informaion on how to enable the bootstrap mechanism, refer to Bootstrapping Process.

Changing the Site Topology During Runtime

You may have to modify your site topology during runtime. For example, a new site may be added and the existing sites may have to start replicating to it and receive replication from it. Alternatively, a site may be removed and the existing sites should stop holding replication backlogs for the removed site, and delete it from their list of gateway targets.

For more information on how to add and remove sites during runtime, refer to Changing Deployment during Runtime.

Additional Resources

The following pages in the Services & Best Practices section provide more details on the Multi-Site Replication module: