Lately I’ve seen a several instances in the news talking about ransomware targeting ESXi servers.  While it doesn’t surprise me much at all, it has been gaining a lot of attention.  With the amount of environments that are virtualized today, the focus seems to be on recovery of virtual machines versus the recovery of the ESXi infrastructure as well.

Don’t get me wrong, recovery of your virtual machines is one of the most important aspects of recovering from ransomware, but if you don’t have any place to recover them to, you’re going to have a bad time.

I want to take a closer look at this topic that has gained so much attention in the last several months.  We’re going to break it all down when it comes to VMware ransomware: why your vSphere environments are being targeted, what happens with an attack, and things to think about when it comes to recovering from ESXi ransomware.

Why are ransomware groups targeting ESXi?

Ransomware groups are targeting ESXi because it just makes sense.  If I can get access to an ESXi host, I can also impact the virtual machines running on the host, as well as other virtual machines in the cluster if I’m going after storage volumes that are shared.  This gives hackers more bang for their buck, and allows them to do damage faster.

It only takes one host to bring a whole ESXi cluster to a grinding halt – after all, every host in the cluster is connected to the same datastores.  If it is a smaller environment with one cluster, hackers have destroyed everything within minutes.

I’ve been working with VMware technologies for longer than I would like to admit at this point, and something I’ve noticed is that the VMware teams aren’t as in tune with security sometimes as they should be.  After all, ESXI doesn’t get viruses, and the VMs are someone else’s problem, aren’t they?

When you’re responsible for critical infrastructure like VMware, it really is everyone’s problem and everyone is a stakeholder.  If you’re VMware inclined and don’t have a security incident response plan for your environment, now is the time to create one in line with the security polices of your organization

But let’s get back to the current threat – ransomware.

How can I recover from ESXi ransomware?

Let me say one thing before I start talking about the recovery aspect: I do not believe paying the ransom to get the decryption key is valid recovery strategy.  At that point, the integrity of the data you have recovered is long gone.  I wouldn’t trust a decrypted VM as far as I could throw a hard drive.

What this means, to me, is you need to plan before ransomware strikes to be able to recover from ransomware.  Once you’ve been encrypted, it is obviously too late.

Recovering from ESXi based ransomware adds an additional layer of complexity, since you must recover your VMware infrastructure before you can begin to recover virtual machines.

ESXi ransomware recovery and IT security incident response

For my VMware inclined friends, you’ll be working with the security incident response team on this one.  Depending on what happens during the initial phases of response, you’ll then be able to decide how to proceed.

The security incident response is important, this will allow everyone to begin to determine exactly what happened, and more importantly, what is impacted.  Once you know the scope and impact, you will then be able to start recovery, or not, you may not find out.

Recovering ESXi hosts from ransomware

The first step is to kiss your ESXi hosts goodbye, you need to start over.

If you’re not used to deploying ESXi and configuring it, and do not have any automation in place, this can take extra time.

The first step is of course to install a fresh copy of ESXi.  Consider taking a look at scripted installations of ESXi if you never have before.

Here are some simple ways you can recover after you install a fresh copy of ESXi:

• Use host profiles for configuration, which should be exported as a backup on a regular basis
• You can also export a copy of your distributed virtual switch config as a backup on a regular basis

These are both enterprise plus features for VMware.  No enterprise plus licensing?

There are two simple and quick ways to create an easily repeatable host configuration

• PowerCLI script(s)
• Ansible

Not confident in your abilities here?  Spin up a nested ESXi environment to practice on and test your configuration with before you test it on real hardware.  This will allow you to get your configuration straightened out without any impact to your production systems.

It will also let you practice recovery, though I also suggest testing on bare metal servers as well to get as close as possible as you can to a real world scenario.

There are many, many ways to rapidly configure or deploy ESXi, the trick is finding the one that works for your organization the best.

What about vCenter?

If vCenter is determined to be uncompromised, you can of course continue with your current vCenter.  If it was encrypted, of course it is trash and you need start over.  Even if you just want to start fresh for piece of mind, deploying vCenter is not a complicated task and does not take a tremendous amount of time. 

You should also deploy a fresh vCenter to test the process on a regular basis.  No matter the case, vCenter remains a critical part of your VMware infrastructure.

Restoring virtual machines from ransomware

Remember, ransomware is a disaster.  A disaster recovery plan the best starting point for ransomware recovery if it is kept up to date and recently and thoroughly tested.  I guess it is still a good starting point if it isn’t updated or tested, because really, what else do you have at this point?

After ESXi is back online, the next task is to restore your virtual machines.  Again, this depends on many things, like do you know when the ransomware entered the environment, or do you need to scan VMs as you are restoring them?

Another option is to have a pre-defined isolated environment set up from a networking perspective.  Restore VMs to the isolated environment, then scan them.  You will likely also be guided on this by the security incident response team.

You also need to know where you’re going to restore to.  If there’s law enforcement involved, they may have told you cannot touch the ransomware impacted environment until they’ve finished their investigation, in this case you may need to recover to a DR site versus in place.

You have a DR site, and a disaster recovery plan, right? 

What if my backups are encrypted?

Then you’re in trouble.  Everyone should have an immutable copy of their backups so this doesn’t happen.  Immutable copies are something you can start doing RIGHT NOW and may be the single most impactful thing you can do now to save yourself later.

Now is the time to take a good hard look at your data protection strategy, and make sure your VMs are protected at a bare minimum.  I’m not just saying this because I work for a data protection company either.

vMiss’ random ransomware protection tips

Now for vMiss’ seemingly paranoid and extreme threat level tips.  I’ve been staying up to date with the latest miserable ransomware trends for a while now, and have even had the pleasure of having one of my environments ransomware as well.  You don’t deal with this every day and not come up with a few crazy ideas.

Always follow the VMware security guide

Look, there’s some stories going around meant to scare people about ESXi ransomware, and yes, we should all be appropriately scared and prepared.  The fact of the matter is that in some cases, security was so bad that the hackers basically opened an unlocked car door and the keys were already in the ignition.

Please take a look at VMware’s ransomware resources.

This is a newer site from VMware that has everything you need to know to get up to speed on ransomware and VMware.

Have immutable backups yesterday

You need to have an immutable copy of your VMs yesterday.  Immutable means it cannot be deleted by modified – by things like hackers or ransomware.  This is the single most important thing to protect your environment that you can do today.

You should be able to laugh in the face of ransomware, not be destroyed by it.

If you’re really concerned about VMware based ransomware and the time it takes to recover critical apps when you have to recover VMware vSphere too, there’s a couple of things you can do to speed things up.

The air gapped vSphere cluster

Have a powered off vSphere cluster ready and waiting with critical components you’ll need to recover.  Of course, it will need to be brought up to be patched, but it should remain isolated and powered off when it is not.

VMware Cloud based recovery

Have a small environment ready and waiting for you in your favorite VMware cloud solution along with a copy of your backup data.  You can expand this cluster within just a few hours and begin restoring VMs almost immediately.  This is especially useful if law enforcement is involved in your ransomware case and you are not allowed to modify the current environment to restore VMs in place, and do not have sufficient capacity at any of your other sites.

The threat of ransomware, and ransomware targeting ESXi is very real.  Unfortunately, if you aren’t prepared to recover today, you have a good chance of a bad day tomorrow.  The barrier to entry for ransomware is just so low – anyone who can get into your network can then consume ransomware as a service, and before you know it everything is encrypted.

Stay tuned for more random VMware ransomware thoughts as they come to mind…