Intro
Dotmetrics is one of the leading audience measurement platforms in the world, it operates in 7 countries, and the company works hard on its global expansion.
We have been working with Dotmetrics for more than eight years. Recently, we successfully migrated their Adria measurement platform to AWS and then scaled the platform to meet the needs of UK audience measurement.
Recently, they approached us with a new problem. In 2021, the company won the tender for an official measurement system for the Australian media. They needed a solution to solve visitor data consolidation across two AWS geographic regions (Ireland, Sydney).
We used Couchbase’s “Cross Data Center Replication” (XDCR) feature, achieved near real-time transcontinental replication, and successfully solved consolidation issues.
The Problem
Dotmetrics’s big data audience measurement solution for the Adria region (Slovenia, Serbia, Bosnia and Herzegovina, North Macedonia) and the UK is hosted in AWS Ireland region.
In 2021. the company won the tender for official audience measurement of Australian media. To support Dotmetrics’s expansion, we deployed their solution on a cluster in the AWS Sidney region. The problem that we faced was the lack of user unification across different geographic regions. If the same webpage visitor visits the page in UK and Australia, the system detects two visitors, not one, although we are talking about the same visitor.
We needed to find a solution to replicate data across continents to identify visitors globally.
Why Couchbase?
Couchbase offers a “Cross Data Center Replication” (XDCR) feature. XDCR allows for unidirectional and bidirectional replication between clusters.
In our case, bidirectional replication was the key to solving Dotmetrics’s problem.
Why AWS CLoud
Dotmetrics’s existing solutions are already hosted in AWS Ireland region. The simplicity of deploying a new instance to a remote geographic region like Australia was the primary reason why we chose AWS once again.
By deploying the solution in the AWS Sidney region, we achieved low latencies and short response times.
Solution Description
We used Couchbase’s “Cross Data Center Replication” (XDCR) feature for bidirectional data replication between two AWS regions (Ireland, Sydney).
An example of bidirectional replication between two data centers
Independent Regions
The existing Adria and UK audience measurement solution is hosted in AWS Ireland. The new instance is deployed to the AWS Sydney region. These two instances are entirely independent of each other; they both have their own data processing and CI/CD procedures. The only thing these two instances have in common is the Couchbase data.
Bidirectional Asynchronous Replication
We used Couchbase’s “Cross Data Center Replication” (XDCR) feature for data replication. With bidirectional replication, data can be created and changed in any of the connected clusters - it does not matter where the data was created (master/master replication). The replication process is near real-time; it is asynchronous and takes a couple of seconds.
Conflict Resolution
The bidirectional replication process can cause data conflicts. A conflict is caused when the source and target copies of an XDCR-replicated document are updated independently, each by a local application. The Couchbase system uses automated conflict resolution to resolve conflicts.
We used “conflict resolution based on a sequence number” to resolve the conflicts. Each document has a sequence number, which is incremented on each document update. When conflicted documents are compared, the one with the higher sequence number prevails.
An example of ring topology (replication between multiple data centers)
