AWS Elastic Disaster Recovery Service. Is it Safe?

Elastic Disaster Recovery Service (DRS) is a disaster recovery service provided within AWS that allows you to “recover” from disasters both on-premiss and in the cloud. But, how does it actually work and is it safe to use within your account?

I recently came across DRS, and I was surprised by how it actually works. The basic premise is you install an agent on the server you want to backup in the event of a disaster. This can be both on premiss or any cloud estate. The agent backs the server up to AWS (in the form of EBS volumes), and then from the console, you can hit an “initiate recovery” button, and your server will appear on an EC2 instance. In principle, this sounds like quite a useful service, but if set up badly, it could give an attacker access to lots of sensitive data… lets’ review to see why.

The Agent (installed on our target server)

In our case, we will be backing up a “web server” located on our AWS estate, we first need to download the Linux agent. We can use the below command replace the region sections respectively.

wget -O ./aws-replication-installer-init.py https://aws-elastic-disaster-recovery-<REGION>.s3.<REGION>.amazonaws.com/latest/linux/aws-replication-installer-init.py

After downloading the agent, we need to run it as root.

The agent will ask for the following information:

• AWS region (for backup)
• AWS Access Key ID
• AWS Secret Access Key
• AWS Session Token
• Volumes for replication (enter for all or specify specific)

When you first start using DRS, it will create the required IAM roles in your account. You should not use longed-lived credentials for the agent. Instead, request a set of temporary keys.

Once the agent has finished installing, it will begin backing up the selected disks to the AWS console.

The Backup In AWS

Now that we’ve created the backup, you should be able to view it in the DRS console (in the region you selected earlier):

Okay, so our backup is there, excellent. But what’s actually happened here? Well, the agent has replicated our entire HDD and sent it to AWS, where it has been transformed into an EBS volume and attached to a “recovery server”. Which we can now also see has been spawned in our account (if this was your first time using DRS, you would be able to specify what size you want this server to be – it’s the t3.small instance in the below picture).

And if we check the EBS volumes it has got attached, we will find an extra 8 GB volume which is our backup.

Now, I personally find this a bit strange. The whole service is basically underpinned by an EC2 instance that just sits in your account and behaves no differently than any other instance. Anyone with enough permissions can still come along and shut it down etc. (which, from my own testing, can mess up the backup process on instances that are currently in sync).

In addition to this, AWS has stopped anyone from SSH’ing into the instance or using SSM to gain access (via the instance user-data), and although I agree this is a good thing. It is possible to reverse this by making the user data run again. Details on how to do that can be found here.

#!/bin/bash
rpm -e amazon-ssm-agent.x86_64
systemctl mask sshd.service
systemctl stop sshd.service

sed -i -e 's/.*open-vm-tools.*//' /etc/crontab

If someone were able to access this instance over SSH or any other method. They would instantly have access to any backup that was stored in this region. Keeping in mind that this is potentially entire servers, it could contain very sensitive information.

Restoring A Server

Now you want to restore a server? This part is relatively straightforward, we go to the console, pick the server we need restoring and hit the “initiate recovery” button.

We now wait a few minutes and then check our EC2 instances. You can also monitor the process of the recovery job by clicking the instance in the DRS screen and viewing the active recovery job.

Our instance has now been recovered and is available in our AWS account.

Okay, DRS has restored our instance, and we’ve averted disaster, awesome! DRS can be useful, especially if you’ve got stuff lying around on old servers it can actually be used as an easy way to move it to AWS. However, there are some security pitfalls you can fall into if you’re not careful…

The Risks of DRS:

Back to the main question, this article is trying to answer… is DRS really safe for use in your AWS workload? Yes, it can, but you need to take extra steps and can’t just assume it’s okay out of the box. A separate AWS account would be a good idea:

Ideally, you’ll want to create a separate account whose only purpose is to run DRS. This will keep it (by default) as safe as we can make it from the start. Additionally, all the security points raised below will be easier to manage in a separate account.

EBS volumes and EC2 snapshots:

Everything that DRS backs up is going to be available as an EBS volume and EC2 snapshot. Take care with roles in the account so no one can just peep into all the backups DRS has made by attaching them to EC2 instances or spawning new ones with the snapshot.

Additionally, EBS volumes and EC2 snapshots are not deleted when the source server is removed from DRS. Although this makes sense from an operational perspective – care needs to be taken that some ancient servers with keys to on-prem/other cloud systems are just left hanging in AWS indefinitely.

SSH and SSM access to the recovery server:

As the SSH/SSM prevention put in place by AWS is reversible – care must be taken on the security group assigned to the recovery instances. If an attacker can communicate with any of the ports/IP’s in the outbound list, they can use this to put a reverse shell into the metadata. Then reenable SSH or SSM for longer-term access.

If this were an attacker, they would now have access to any machine backed up by DRS (or the contents of those machines, to be more specific)

Granting access to multiple accounts?

If you’re adding additional accounts to redeploy the backup instances in the event of a disaster – be careful which accounts are granted access. As these accounts will be able to list any backups you’ve made (and if they have the correct KMS permissions) restore those instances in their account. Although, the attack path for this would require a lot of permissions for the attacker to have. It does, however, open up a valid data exfiltration path out of your account.

EC2 Security?

DRS is basically underpinned by EC2 instances (and its features) the same considerations in regards to security need to be taken here. Such as instance profiles, security groups and access controls.

Need More Info?

I’ve also created a page on secwiki.cloud for DRS. Check it out here!

Want to read the documentation on DRS? That can be found here!

Check out some of my other blogs here!