Replicate Existing Objects Using AWS S3 Batch Replication

With new updates, you can replicate existing AWS S3 objects and synchronize AWS S3 buckets using new replication features.

Until today, S3 replication did not support replicating existing objects but now you can do it using AWS S3 batch replication. This is different from live replication which continuously and automatically replicates new objects across different S3 buckets located in different AWS accounts or AWS regions.

Amazon Web Service S3 Replication is a low cost, fully managed feature that automatically replicates S3 objects between buckets in same AWS region using S3 Same-Region-Replication (SRR) or across different AWS Region by using S3 Cross-Region-Replication (CRR). S3 Batch Replication refills newly created buckets with existing objects, can migrate data across different accounts, retry objects that got failed or unable to replicate in existing replication run.

You can configure S3 Batch replication using AWS SDK’s, AWS S3Console or AWS Command Line Interface (CLI).

Here is the replication process diagram from AWS site,

AWS S3 Batch Replication can help to do,

• Replicate Existing Objects - S3 Batch Replication can be used to replicate objects that were added to buckets before configuring any replication rules.
• Replicate Objects that failed to replicate previously - You can retry to replicate objects that were previously failed to replicate due to any reasons.
• Replicate Objects that were already replicated - You might need to store multiple copies of your data to separate AWS accounts or different AWS Regions. S3 Batch can replicate existing objects to newly added destinations.

How to get started

I have already created two AWS S3 buckets (replication-bucket1 and replication-bucket2) in the region us-east1 for this demo.

Step 1

Before configuring the replication rule, if Bucket Versioning is not enabled then you need to enable Bucket Versioning on source and target bucket else you will receive error message like this on replication configuration page,

Step 2

Go to the Properties page and enable Bucket Versioning.

Step 3

Go to S3 bucket list and select a source bucket (replication-bucket1) that contains objects for replication.

Step 4

Go to Management page and choose Create Replication Rule option.

Step 5

Enter Replication rule name

Step 6

After filling required details and creating rule, you will get a prompt asking if you want to replicate existing objects.

Step 7

If you click option Yes, then you will get redirected to a Create Batch operations job. Choose the default option to Automatically run the job when it's ready.

Step 8

You are recommended to choose the option Generate completion report which will contain results of replication job. If you don't have a required IAM role for this then keep the default setting and AWS S3 will create a new IAM role with sufficient permission to run this Batch operation.

Step 9

Once you Save this job, you can check status on the Batch Operations page. Job Status will keep changing from configuring -> in progress -> completion during this process.

Step 10

You can go to your destination bucket and confirm new objects has been replicated.

Conclusion

Using new replication feature it is easy to replicate existing S3 objects between different S3 buckets in same AWS region or different AWS Regions or account.