Category Archives: Data Storage

IT Infrastructure for Remote Offices

When designing the IT infrastructure (servers, storage, and network) of small remote offices, infrastructure architects of large enterprises are often faced with the question, what is the best IT infrastructure solution for remote sites? Low-cost, simple, secure, and easy to support solution always come to mind, but positive end-user experience in terms of network and application performance, and user friendliness should also be in the top priorities when building the infrastructure.

Most small sites just need access to enterprise applications and to file and print services. Network infrastructure definitely needs to be built – such as the site’s local area network (LAN), wireless access points, wide area network (WAN) to connect to the enterprise data center, and access to the Internet. The bigger question though is: should servers and storage be installed on the site?

There are a lot of technologies such as WAN accelerators and “sync and share” applications that will forgo installing servers and storage on the remote sites without sacrificing positive end-user experience. For instance, Riverbed WAN accelerator products tremendously improve performance access to files and applications from the remote sites to the enterprise data center.  These products can even serve up remote datastore for VMware farms. “Sync and share” applications are dropbox-like applications (such as EMC Syncplicity). Enterprises can build a storage as a service solution in their internal infrastructure. This will eliminate the need to install file servers or storage appliances on the remote sites.

The decision to “install servers” or “go serverless” at the remote sites still depends on many factors. They should be dealt with on a case by case basis and should not rely on a cookie cutter solution. Some of the criteria to consider are: the number of people at the sites and the growth projection; the storage size requirement, available WAN bandwidth, the presence or absence of local IT support, office politics, and country/region specific regulation for data to remain local. If these issues are factored in, a better solution can be designed for remote offices.

Migrating Files to EMC VNX

There are several methodologies in migrating files to EMC VNX. One method I used recently to migrate Windows files (CIFS) was to copy files between the source CIFS server to the target VNX server using the emcopy migration suite of tools from EMC. EMC provides these free tools including emcopy, lgdup, and sharedup to accomplish the migration task. There are several steps you need to follow for a successful migration. In general, I used the following procedure:

1. Create the necessary VDM (Virtual Data Mover), File System, and CIFS server on the target VNX machine.
2. Create CIFS share. Copy the share permissions (or ACL) and the NTFS root folder ACLs from the old share to new share. You can also use the sharedup.exe utility.
3. Use lgdup.exe utility to copy local groups from the source CIFS server to the target CIFS server.
4. Run emcopy.exe to perform baseline copy.
5. Create an emcopy script to sync the files every night. This will temendously cut the time needed to update the files on the final day of migration.
6. Analyze emcopy log to make sure files are being copied successfully. You may also spot check the ACLs and/or run tools to compare files and directories between the source and target.
7. On the day of the cutover:
a. Disconnect users from the source CIFS server and make the file system read-only.
b. Run the final emcopy script.
c. Follow EMC156835 to rename the CIFS server so that the new CIFS server will have the name of its old CIFS server. This procedure entails unjoining the source and target CIFS server from Active Directory (AD), renaming NetBIOS name on the new CIFS server, joining back the CIFS server, etc. Update the DNS record too if necessary.
8. Check the new CIFS shares and make sure the users are able to read/write on the share.

To migrate UNIX/Linux files, use the UNIX rsync utility to copy files between source and target VNX.

Moving a Qtree Snapmirror Source in NetApp Protection Manager

A couple of weeks ago, one of the volumes in our NetApp Filer storage almost ran out of space. I cannot expand the volume since its aggregate is also low in space. I have to move it to a different volume contained in an aggregate with plenty of space on the same filer. The problem is, this volume contained qtrees that are snapmirrored to our Disaster Recovery (DR) site and it is being managed by NetApp Protection Manager. How do I move the qtree snapmirror sources without re-baselining the snapmirror relationship using Protection Manager? Unfortunately, there is no way to do this using Protection Manager, and re-baselining is not an option – it has terabytes of data that may take couple of weeks to complete.

Like any sane IT professional, I googled how to do this. I did not find a straight forward solution, but I found bits and pieces of information. I consolidated this information and generated the steps below. Generally, the steps below are the combination of the snapmirror CLI commands and Protection Manager configuration tasks.

1. On the CLI of the original source filer, copy the original qtree to a new qtree on a new volume by using the following command:

sourcefiler> snapmirror initialize -S sourcefiler:/vol/oldsourcevol/qtree sourcefiler:/vol/newsourcevol/qtree

This took some time, and I also updated snapmirror.conf so that snapmirror updates daily.

2. On the day of the cutover, perform a final snapmirror update on the new volume. Before doing this, make sure that nobody is accessing the data by removing the share.

sourcefiler> snapmirror update sourcefiler:/vol/newsourcevol/qtree

3. Login to the Operations Manager server, and run the following on the command prompt:

c:\> dfm option set dpReaperCleanupMode=Never

This prevents Protection Manager’s reaper cleaning up any relationship.

4. Issue the following command to relinquish the primary and secondary member:

c:\> dfpm dataset relinquish destination-qtree-name

This will mark the snapmirror relationship as external and Protection Manager will no longer manage the relationship.

5. Using NetApp Management Console GUI interface, remove the primary member from the dataset. Then remove the secondary member.

6. On the CLI of the source filer, create a manual snapshot copy by using the following command:

sourcefiler> snap create oldsourcevol common_Snapshot

6. Update the destinations by using the following commands:

sourcefiler> snapmirror update -c common_Snapshot -s common_Snapshot -S sourcefiler:/vol/oldsourcevol/qtree sourcefiler:/vol/newsourcevol/qtree

destinationfiler> snapmirror update -c common_Snapshot -s common_Snapshot -S sourcefiler:/vol/oldsourcevol/qtree destinationfiler:/vol/destinationvol/qtree

7. Quiesce and break the SnapMirror relationship between source filer and destination filer, and oldsource and newsource volumes, using the following commands:

destinationfiler> snapmirror quiesce /vol/destinationvol/qtree
destinationfiler> snapmirror break /vol/destinationvol/qtree
sourcefiler> snapmirror quiesce /vol/volnewsourcevol/qtree
sourcefiler> snapmirror break /vol/volnewsourcevol/qtree

8. Establish the new snapmirror relationship using the following command on the destination system:

destinationfiler> snapmirror resync -S sourcefiler:/vol/newsourcevol/qtree destinationfiler:/vol/destinationvol/qtree

The new SnapMirror relationship automatically picks the newest common Snapshot copy for replication. This is the common Snapshot copy.

9. Verify that the SnapMirror relationship is resynchronizing by using the following command:

destinationfiler> snapmirror status

10. Recreate the shares on the new source volume.

11. At this point, on the Protection Manager GUI console, you will see the snapmirror relationship in the External Relationship tab.

12. Create a new dataset with required policy and schedule. Use the import wizard to import the snapmirror relationship to the new dataset.

13. On the Operations Manager server command prompt, set back the reaper cleanup mode back to orphans.

c:\> dfm options set dpReaperCleanupMode=orphans.

Please send me a note if you need more information.

Sources

https://library.netapp.com/ecmdocs/ECMP1196991/html/GUID-301E424F-62C5-4C89-9435-F13202A1E4B6.html
https://communities.netapp.com/message/44365

Deduplication and Disk-based Backup

Disk-based backup has been around for a long time now.  I remember when I first deployed a Data Domain DD560 appliance eight years ago. I was impressed by its deduplication technology.  Data domain appliances are disk storage arrays used primarily for backup.  We used it as a disk backup destination for Symantec Netbackup via NFS.  Data Domain sets itself apart because of its deduplication technology.  For instance, in our environment, we are getting a total compression (reduction) of 20x; which means that our 124TB data only uses 6TB of space – a very large space savings indeed.

In addition, disk-based backup makes it very fast and easy to restore data. The need to find the tapes, retrieve them, and mount them have been eliminated.

In fact, the need for tapes can be totally eliminated.  A lot of companies are still using tapes to store them off site for disaster recovery.  If you have a remote office site, a Data Domain appliance can be replicated to your remote site (or disaster recovery site).  What is nice about Data Domain replication is that only the deduped data is sent over the wire.  This means less bandwidth consumption on your WAN (Wide Area Network).

Deduplication technology is getting better. Storage vendors and backup software vendors are offering it in different flavors.  With the cost of disk going down, there is really no need for tapes. Even long term retention and archiving of data can now be stored on a low cost, deduped disk storage array.

Best Practices for Using NFS Datastore on VMware

More companies are now deploying VMware with IP based shared storage (NAS). NAS storage is cheaper than Fiber Channel (SAN) storage because there is no separate Fiber Channel (FC) based network to maintain. More importantly, IP based storage performance and stability are now comparable with FC based storage.

Other advantages of using IP based storage, specifically NFS, are thin provisioning, de-duplication, and the ease-of-backup-and-restore of virtual machines and files on a virtual disk via array based snapshots. In addition, IP based storage is easier to maintain.

VMware published a whitepaper on the best practices for running VMware vSphere on Network Attached Storage (NAS) using NFS. Following the best practices in deploying an NFS based storage is very important to obtain a stable and optimized VMware environment. Here are the important things to consider:

On the network side, the local area network (LAN) on which the NFS traffic will run needs to be designed with availability, downtime-avoidance, isolation, and failover:

1. NFS traffic should be on a separate physical LAN, or at least on a separate VLAN.
2. Use private (non-routable) IP addresses. This will also address a security concern since NFS traffic is not encrypted and NFS is mounted with root privileges on the VMware host.
3. Use redundancy by teaming the NICs on the VMware host, configuring LACP protocol, and using two LAN switches.
4. Use jumbo frames.
5. Use 10GB Ethernet.

On the storage array side, the storage controller must be redundant, in case the primary one fails. In addition,

1. Configure the NFS exports to be persistent. (e.g. exportfs –p)
2. Install the VAAI and other plug-in tools from the storage vendor. For instance, NetApp has the Virtual Storage Console (VSC) plug-in that can be installed on the vCenter.
3. Refer to the storage vendor best practices guide. For instance, NetApp and EMC published their own best practice whitepapers for using NFS on VMware.

On the VMware hosts, the following configuration should be implemented:

1. Use the same datastore name across all hosts.
2. Select “No” for NIC Teaming failback option. If there is some intermittent behavior in the network, this will prevent the flip-flopping of NIC cards being used.
3. If you increase the maximum number of concurrent mount points (from the default of 8), also increase Net.TcpipHeapSize as well. For instance, if 32 mount points are used, increase tcpip.Heapsize to 30MB.
4. Set the following VMware High Availability options: (NFS heartbeats are used to determine if an NFS volume is still available.)
NFS. Hearbeat.Frequency=12
NFS.Hearbeat.Timeout=5
NFS.Hearbeat.MaxFailure=10

When configured properly, IP based storage, specifically NFS, provides a very solid storage platform for VMware.

NetApp Virtual Storage Console for VMware vSphere

One of the best tools for managing NetApp storage and VMware is a plug-in called NetApp Virtual Storage Console (VSC) for VMware vSphere. VSC provides administrators the ability to manage NetApp storage from vCenter client. It can configure, monitor, provision, and migrate NetApp datastores with fewer clicks. In addition, it can perform backup and recovery of LUNs and volumes from the vCenter client.

VSC can automatically discover your NetApp storage controllers and ESXi hosts. This task can take a lot of time if not using VSC. VSC can also automatically apply “best practices” settings on the ESXi host to optimize its configuration. It can rapidly provision datastores without going through the NetApp management interface. You can get backup (snapshots) of the datastore in a consistent state, and perform recovery in minutes.

NetApp implementation of its vStorage API for Array Integration (VAAI) offloads significant processing tasks to the storage array, freeing ESXi resources for other tasks. If you are using NFS though, you still need to download and install the NetApp NFS Plug-in for VMware VAAI.

For now, the VSC plug-in is only available for the traditional vCenter client. VMware is moving towards replacing the traditional vCenter client with the vSphere Web Client. I hope that NetApp releases the plug-in version for the web client pretty soon.

Hot Adding NetApp Shelf

One of the great features of NetApp FAS 3200 series is the ability to add shelves without any downtime. As our need for storage space exponentially increases, we need the ability for our storage system to be expanded without any interruption to our business users. I recently added a DS4243 shelf into an existing stack, and followed the steps below:

1. Change the disk shelf ID. Make sure the shelf ID is unique among the stack. On the DS4243 shelf, the ID can be changed by pressing the U-shaped button located near the shelf LEDs. The shelf needs to be power cycled for the new ID to take effect.

2. Cable the SAS connection. It is very important to unplug/connect the cable one at time.

a. Unplug the cable from the I/O module A (IOM A)circle port from the last shelf in the stack.

b. Connect the cable from the new shelf IOM A square port to the IOM A circle port that was disconnected in step a.

c. Reconnect the cable that was disconnected in step a to the IOM A circle port of the new shelf.

d. Repeat the same procedure for IOM B.

3. Check connectivity by running the following commands on the console:

sasadmin expander_map
sasadmin shelf
storage show disk -p

4. Assign disk to the filer. If auto assign is turned on, the disks will be auto assigned to the filer. I disabled autoassign disk, since in a cluster, I want to control where the disks go. I usually go to the console of the filer where I want the disk assigned, check to see all unassigned disk drive using the command disk show -n, and finally issue the command disk assign all to assign the disks.

For a complete step by step instructions, consult your NetApp manuals.

Creating LUN in NetApp Using CLI

If you want to create a LUN (Logical Unit Number) on a vfiler in NetApp, you will be forced to use CLI commands.  There is no wizard GUI tool for vfilers at least for now.

To carve up a storage space in NetApp to be presented to a SPARC Solaris machine using iSCSI HBA, I used the following steps:

1. Configure iSCSI HBA on Solaris (i.e., configure IP address, netmask, gateway, vlan tagging [it if its on a separate vlan], etc)

2. Login through NetApp console or remote session.

3. Go to the vfiler

nas3240> vfiler context vfiler-iscsi

4. Determine which volume to create the LUN. Make sure it has enough space.

nas3240@vfiler-iscsi> vol status

nas3240@vfiler-iscsi> df -h

5. Create qtree. I usually create the LUN on qtree level instead of volume. This makes my structure cleaner.

nas3240@vfiler-iscsi> qtree create /vol/iscsi_apps/solaris

6. Create the LUN using this syntax: lun create -s size -t ostype lun_path

nas3240@vfiler-iscsi> lun create -s 200g -t solaris /vol/iscsi_apps/solaris/lun0

Successful execution of this command wil create LUN “/vol/iscsi_apps/solaris/lun0” with a size of 200GB, and space-reserved.  For LUN, the best practice is to thick provision (space-reserved), so you won’t have problems when the storage runs out of space.

7. Create initiator group or igroup which contain the IQN for the solaris host. Initiate a iscsi login command from solaris host, and NetApp will see the IQN number.  This IQN number will appear on the console and you can cut and paste that number. Use this syntax to create igroup: igroup create -i -t ostype initiator_group iqn_from_host

nas3240@vfiler-iscsi> igroup create -i -t solaris solaris_group iqn.2000-04.com.qlogic.gs2342345.2

8. Map the LUN to the host using igroup you created. Use this syntax: lun map lun_path initiator_group [lun_id] where: lun_path is the path name of the LUN you created, initiator_group is the name of the igroup you created, and lun_id is the identification number that the initiator uses when the LUN is mapped to it. If you do not enter a number, Data ONTAP generates the next available LUN ID number.

nas3240@vfiler-iscsi> lun map /vol/iscsi_apps/solaris/lun0 solaris_group

9. Verify LUN list and their mapping.

nas3240@vfiler-iscsi> lun show -m

LUN path                                    Mapped to              LUN ID    Protocol
——————————————————————————————
vol/iscsi_apps/solaris/lun0      solaris_group        2                   iSCSI

10.  Go to solaris box, and do iSCSI refresh.  Check that it can see the LUN disk that has been provisioned.

NetApp Fpolicy Tool to Block W32/XDocCrypt.a Malware

There is a virus going around called W32/XDocCrypt.a that is causing havoc to Excel and Word files located on the network.  This virus renames files to .scr.

 

If you do not have the latest cure and your files are stored on NetApp filers, you can prevent the virus from infecting your files by using the fpolicy tool on NetApp.  The McAfee vscan for NetApp storage does not work very well.

 

On the NetApp filers,verify that fpolicy is enabled by issuing this command:

 

options fpolicy

 

If it’s not enabled, enable it:

 

options fpolicy.enable on

 

Then run the following commands:

 

fpolicy create scrblocker screen
fpolicy ext inc set scrblocker scr
fpolicy monitor set scrblocker -p cifs create,rename
fpolicy options scrblocker rquired on
fpolicy enable scrblocker -f

 

If you are using vfiler, create the above commands on the vfiler.  Also, do not specify any volume, because it does not work.

 

The fpolicy tool can also be a great tool in blocking unwanted files on your filers such as mp3. For more information on fpolicy, go to this website.

 

Disaster Recovery using NetApp Protection Manager

In our effort to reduce tape media for backup, we have relied on disks for our backup and disaster recovery solution. Disks are getting cheaper and de-duplication technology keeps on improving. We still use tapes for archiving purposes.

One very useful tool for managing our backup and disaster recovery infrastructure is NetApp Protection Manager. It has replaced the management of local snapshots, snapmirror to Disaster Recovery (DR) site, and snapvault. In fact, it doesn’t use these terms anymore. Instead of “snapshot,” it uses “backup.” Instead of “snapmirror,” it uses the phrase “backup to disaster recovery secondary.” Instead of “snapvault,” it uses “DR backup or secondary backup.”

NetApp Protection Manager is policy-based (e.g. backup primary data every day @ 6pm, and retain backups for 12 weeks; backup primary data to DR site every day @ 12am; backup the secondary data every day @ 8am and retain for 1 year). As an administrator, one does not have to deal with the nitty-gritty technical details of snapshots, snapmirror, and snapvault.

There is a learning curve in understanding and using Protection Manager. I have been managing NetApp storage for several years and I am more familiar with snapshots, snapmirror, and snapvault. But as soon as I understood the philosophy behind the tool, it gets easier to use it. NetApp is positioning it for the cloud. The tool also has dashboards intended for managers and executives.