Category Archives: Backup and Disaster Recovery

Improving the Nation’s Cybersecurity

Leave a reply

Due to the rampant cyber attacks on private and public companies as well as government institutions, the President of the United States issued an Executive Order 14028 on May 12, 2021 to improve the nation’s cybersecurity.

To comply, companies must implement the five best practices from the Executive Order of the President:

  • Back up data, system images, and configurations; regularly test them, and keep the backups offline.
  • Update and patch systems promptly.
  • Test the incident response plan.
  • Check the work of the security team.
  • Segment the networks.

More information can be found at the Cybersecurity and Infrastructure Security Agency.

Disaster Recovery Site vs Cyber Recovery Site

Leave a reply

While the ultimate goal of both Disaster Recovery (DR) and Cyber Recovery(CR) is the same, to bring your IT services back online after an event, DR and CR differ in many ways. DR protects against physical disasters (such as flooding, earthquake, fire, terrorist attacks), user errors, and hardware malfunctions, while CR protects against cyber attacks.

Because of these differences, a DR site may not be able to provide the necessary function for restoring systems after a cyber attack. A Cyber Recovery site is needed to recover your systems from a cyber attack. A CR site is a vaulted site, containing replicated data of critical systems. It is isolated from the network and only connects during replication. A CR site doesn’t have to be in a geographically separate location (unlike a DR site), but it should be physically secured. In fact, it will be beneficial if it’s close to the primary Data Center to take advantage of a fast network connection.

DR usually supports most of the business operations (since disasters are usually bigger in magnitude), whereas CR supports only the most critical systems for operational recovery.

While IT admins have access to a DR site, a CR site should be restricted to a few select security folks.

Usually a DR site is also dormant, but a CR site will always have monitoring and analytics tool – checking for security breaches.

Many companies with mature IT infrastructure typically have a DR site. They also conduct regular DR tests. But most do not have a Cyber Recovery site yet. With the prevalence of cyber security attacks, it is high time for companies to install Cyber Recovery sites and conduct regular cyber recovery tests.

Checklist for a Secure Backup Infrastructure

Leave a reply

Ramsomware attacks are running rampant these days. The ability to restore your servers, data and applications after an attack is critical to minimize costly disruption and bring your business back in service.

A secure backup infrastructure is key to this. Here’s a checklist on how to keep your backup systems robust and secure:

  1. Backup and retain data for several weeks. For critical systems, you will know right away if they have been compromised. But some may take days or weeks before you know you have been attacked or get a demand for payment from attackers. It’s better to have retained good copies of backup to restore.
  2. Replicate backup to an offsite location.
  3. Conduct regular file recovery and bare metal recovery tests.
  4. Perform regular patching and upgrade of backup devices. You also need to work with your device vendor to discuss vulnerabilities, monitor security advisories and apply critical security patches as soon as possible.
  5. Harden your backup infrastructure, for instance by replacing default passwords and turning off unneeded services.
  6. Implement data-at-rest and data-in-flight encryption.
  7. Monitor and send alerts for unusual activities such as privileged user multiple login attempts as well as backup deletion attempts. You can integrate with Splunk, CyberSense or other security analytics applications that have advance AI technology to easily monitor, detect, and analyze security breaches.
  8. Recertify user accounts on your backup devices periodically.
  9. Use security tool to store and encrypt local and service accounts passwords on the devices.
  10. Use two factor authentication.
  11. Implement Retention Lock.
  12. Implement backup vault by isolating the second copies of the backup. This can be done by using tapes or air gapping the backup storage.

Improving Security of Backup Data

Leave a reply

One of the best defense against ransomware is to backup data and verify its integrity regularly.  If your data has been breached by a ransomware, you can always restore the data from backup.  However, hackers using ransomware are increasingly targeting primary backups. Adding an air gap to the secondary copy of the backup can mitigate this, 

An air gap is a security measure that protects backup data from intrusion, malicious software and direct cyber attacks  The idea is to place a secondary copy of backups behind a private network that is not physically connected to the wider network (i.e. behind air gaps). These secondary air-gapped backups will provide preserved backup copies and will be capable of restoring data that have been attacked by ransomware.

One example of air gap implementation is by DellEMC.  In the figure below, the Data Domain primary backup storage (Source) is replicated to a Data Domain secondary backup storage (Target) inside a vault.  The vault is self-contained and self-secured.  It is air-gapped except for replication in cycles.  It also has encryption and data protection controls including mutual authentication of source and target, data-at-rest encryption, data-in-motion encryption, replication channel encryption, Data Domain hardening, and immutable data (using retention lock). In addition, it also contains applications that scans for security issues and tests for critical apps.

DellEMC Cyber Recovery


VMWare Instant Recovery

Leave a reply

When a virtual machine crashes, there are two ways to quickly recover it – first is by using the VMware snapshot copy and second is by restoring an image-level backup.  Most VMware environment though do not usually perform snapshots on the virtual machines (VMs) due to increased usage on the primary storage, which can be costly.   On the other hand, using traditional method to restore image-level backup can take longer since it has to be copied back from the protection storage to the primary storage. 

However, most backup solutions nowadays – including Netbackup, Avamar/Data Domain, and Veeam – support VMware instant recovery where you can immediately restore VMs by running them directly from backup files.  The way it works is that the virtual machine image backup is staged to a temporary NFS share on the protection storage system (e.g. Data Domain).   You can then use the vSphere Client to power on the virtual machine (which is NFS mounted on the ESXi host), then initiate a vMotion of the virtual machine to the primary datastore within the vCenter. 

Since there is no need to extract the virtual machine from the backup file and copy it to production storage, you can perform restore from any restore point in a matter of minutes. VMware instant recovery helps improve recovery time objectives (RTO), and minimizes disruption and downtime of critical workloads.

There are also other uses for instant recovery. You can use it to verify the backup image, verify an application, test a patch on a restored virtual machine before you apply the patch to production systems, and perform granular restore of individual files and folders.

Unlike the primary storage, protection storage such as Data Domains are usually slow.  However, the new releases of Data Domains have improved random I/O (due to additional flash SSD), higher IOPS and better latency, enabling faster instant access and restore of VMs. 

Encrypting In-flight Oracle RMAN Database Backup via DD Boost

3 Replies

To secure Oracle database backup from a DB server to a Data Domain system, DD Boost for RMAN encryption can be enabled so that RMAN backup data can be encrypted after deduplication at the Oracle server and before transmitting across the network. Since the encryption happens after deduplication and before the segment leaves the Oracle server (in-flight encryption), deduplication ratios will not suffer on the Data Domain system. In contrast, if Oracle RMAN encryption is used, data will not be deduplicated because they will be encrypted first, thus deduplication ratio will suffer.

In-flight encryption enables applications to encrypt in-flight backup or restore data over the network from the Data Domain system. When configured, the client is able to use TLS to encrypt the session between the client and the Data Domain system.

To enable in-flight encryption for backup and restore operations over a LAN, run the following command on the Data Domain:

# ddboost clients add client-list [encryption-strength {medium | high} authentication-mode {one-way | two-way | anonymous}]

This command can enable encryption for a single client or for a set of clients.

The specific cipher suite used is either ADH-AES256-SHA, if the HIGH encryption option is selected, or ADH-AES128-SHA, if the MEDIUM encryption option is selected.

The authentication-mode option is used to configure the minimum authentication requirement. A client trying to connect by using a weaker authentication setting will be blocked. Both one-way and two-way authentication require the client to be knowledgeable about certificates.

For example:

# ddboost clients add db1.domain.com db2.domain.com encryption-strength high authentication-mode anonymous

To verify:

# ddboost clients show config
Client          Encryption Strength  Authentication Mode
*               none                 none
db1.domain.com  high                 anonymous
db2.domain.com  high                 anonymous

Using BoostFS to Backup Databases

Leave a reply

If your company is using DellEMC Data Domain appliance to backup your databases, you are probably familiar with DD Boost technology. DD Boost increases backup speed while decreasing network bandwidth utilization.  In the case of Oracle, it has a plugin that integrates directly into RMAN. RMAN backs up via the DD Boost plugin to the Data Domain. It is the fastest and most efficient method to backup Oracle databases. 

However, some database administrators are still more comfortable with performing cold backups.  These backups are usually dumped to the Data Domain via NFS mount.   This is not the most efficient way to backup large databases as they are not deduplicated before sending to the network, thus consuming a lot of bandwidth.

Luckily, DellEMC created the product BoostFS (Data Domain Boost Filesystem) which provides a general file-system interface to the DD Boost library, allowing standard backup applications to take advantage of DD Boost features.   In the case of database cold backup, instead of using NFS to mount the Data Domain, you can use BoostFS to stream the cold backups to the Data Domain, thus increasing backup speed and decreasing network bandwidth utilization. In addition, you can also take advantage of its load-balancing feature as well as in-flight encryption.

To implement BoostFS, follow these steps:

1. DDBoostFS is dependent on FUSE.  So before installing DDBoostFS, install fuse and fuse-libs first.

2. Edit the configuration file /opt/emc/boostfs/etc/boostfs.conf, specifying the Data Domain hostname, DD storage-unit, username, security option, and if you want to allow users other than the owner of the mount to access the mount.  This is useful if you are using the same storage-unit for multiple machines.

3. Create the lockbox file, if you specified lockbox as the security option.  This is the most popular choice.

4. Verify host has access to storage using command /opt/emc/boostfs/bin/boostfs lockbox show-hosts

5. Mount the new boostfs storage unit using command /opt/emc/boostfs/bin/boostfs mount

6. To retain the mount after reboots, add the boostfs entry on /etc/fstab

For more information, visit the DellEMC support site.

Using the Cloud for Disaster Recovery

Leave a reply

One of the common use cases for using the cloud, especially for companies with large on-prem data centers, is Disaster Recovery (DR).  Instead of building or continuing to maintain an expensive on-prem DR site, the cloud can provide a cheaper alternative for replicating and protecting your data.

There are many products and services out there for DR in the cloud.  If your company is using EMC devices – specifically Avamar and Data Domain (DD) – for data protection, you can replicate your virtual machines (VM) backup to AWS and be able to perform disaster recovery of your servers in AWS.  This solution is called Data Domain Cloud DR (DDCDR) and  it enables DD to backup to AWS S3 object storage. Data is sent securely and efficiently, requiring minimal compute cycles and footprint within AWS. In the event of a disaster, VM images can be restored and run from within AWS. Since neither Data Protection Suite nor DD are required in the cloud, compute cycles are only required in the event of a restore.

Backup Process

  • DDCDR requires that a customer with Avamar backup and Data Domain (DD) storage install an OVA which deploys an “add-on” to their on-prem Avamar/DD system and install a lightweight VM (Cloud DR server) utility in their AWS domain.
  • Once the OVA is installed, it will read the changed data and will segment, encrypt, and compress the backup data and then send this and the backup metadata to AWS S3 object storage.
  • Avamar/DD policies can be established to control how many daily backup copies are to be saved to S3 object storage. There’s no need for Data Domain or Avamar to run in AWS.

Restore Process

  • When there’s a problem at the primary data center, an admin can click on a Avamar GUI button and have the Cloud DR server uncompress, decrypt, rehydrate and restore the backup data into EBS volumes, translate the VMware VM image to an AMI image, and then restarts the AMI on an AWS virtual server (EC2) with its data on EBS volume storage.
  • The Cloud DR server will use the backup metadata to select the AWS EC2 instance with the proper CPU and RAM needed to run the application. Once this completes, the VM is running standalone, in an AWS EC2 instance. Presumably, you have to have EC2 and EBS storage volumes resources available under your AWS domain to be able to install the application and restore its data.

Source: https://www.dellemc.com/

Upgrading Avamar Proxies from Version 7.2 to 7.4

Leave a reply

Avamar Proxies cannot be upgraded anymore from version 7.2 to 7.4 using the old method (i.e. mounting the ISO file and rebooting the proxy), due to incompatibility with the new version.

In general, you have to delete the old proxies, and deploy new proxies using the new tool Proxy Deployment Manager.   To preserve the settings of the old proxies, perform the following steps when there are no backup jobs running:

  1. Collect all the details from the old proxies including:
    • Hostname, Domain Name, IP address, Netmask, Gateway, DNS Server
    • VM host, VM datastore, VM network
  2. Delete proxies on the Avamar Console:
    • First, on the POLICY window, edit all the backup policies that are using the proxies, and uncheck them.
    • Once removed from policy, go to ADMINISTRATION, and delete the proxies.
  3. Go to vCenter to power down the proxies, then “Delete from Disk”
  4. Once all the proxies are gone, you are now ready to deploy the new proxies. Go to Avamar Console, click VMware > Proxy Deployment Manager.
  5. Click “Create Recommendation” button.
  6. Once you see the proxy recommendation, enter the proxy detail one by one for all proxies (including hostname, IP, gateway, VM network, etc.) on their respective VMware hosts.
  7. Remove all other “New Proxies” and hit “Apply”
  8. Once the proxies are deployed, they need to be registered to the Avamar server, one by one.
  9. Using vmware console or ssh, connect to the proxy, and logon as root.
  10. Enter the command: /usr/local/avamarclient/etc/initproxyappliance.sh start
  11. Register the proxy to the appropriate Avamar server (use the Avamar server FQDN).
  12. Once registered, go to the Avamar Console and configure the proxies:
    • On ADMINISTRATION window, edit the proxy, then select the appropriate “Datastores” and “Groups”
    • On POLICY window, edit the image-level backup policies, then add back (or check) the Proxies
  13. Perform test backup.

Data Protection in AWS

Leave a reply

Data protection along with security used to be an afterthought in many in-house IT projects. In the cloud, data protection has became the forefront for many IT implementations. Business users spinning up servers or EC2 instances in AWS clamor for the best protection for their servers and data.

Luckily, AWS provides a highly effective snapshot mechanism on EBS volumes that are stored on a highly durable S3 storage. Snapshots are storage efficient and use copy-on-write and restore-before-read which allow for both consistency and immediate recovery. Storing snapshot in S3 which is a separate infrastructure from EBS, has the added benefit of data resiliency – failure in the production data will not affect the snapshot data.

However, this backup and restore mechanism provided by AWS lacks many of the features found in the traditional backup solutions such as cataloging, ease of management, automation, and replication. In response, third party vendors are now offering products and services that will make backup and recovery easy and efficient in AWS. Some vendors provide services to manage and automate this. Other vendors provide products that mimics the ease of management of the traditional backup. For instance, Dell EMC provides Avamar and Data Domain virtual editions that you can use on AWS.