Author Archives: admin

Bare Metal Restore

Leave a reply

One important capability of a disaster recovery plan is the ability to do bare metal restore (BMR). A BMR is a restore of your entire server to new or original hardware after a catastrophic failure. A BMR can either be done manually – by reformatting the computer from scratch, reinstalling the operating system, reinstalling software applications, and restoring data and settings; or automatically – by using BMR tools to facilitate the bare metal restore process. The manual process, however, takes time and can be error prone, while BMR tools can be fast and easy.

With the majority of servers being virtualized, what’s the use of BMR? With virtualization, especially when using image level backup, there is no need to use specialized BMR tools. However, there are still servers that cannot be virtualized (such as applications requiring dongle, systems requiring extreme performance, applications/databases with license agreements that do not permit virtualization, etc.). With these systems requiring physical servers, BMR is critical to their recovery.

Backup vendors usually have bare metal solution integrated in their package, but usually not enough. There are software vendors that specialize in bare metal recovery.

Typically, a bare metal restore process involves:
1. Generating an ISO recovery image
2. Using the ISO image to boot the system to be recovered
3. Once in the restore environment, setting up the network connection (IP address, netmask, etc.), so it can connect to the backup server to restore the image.
4. Verifying the disk partitions and mapping.
5. Stepping through the restore wizard – such as choosing the image file you want to restore (point in time), and the partition (or unallocated space) to which you want to restore.
6. Performing any post recovery tasks – such as checking the original IP address, checking that the application services are running, etc.

Bare metal restore is essential to a server disaster recovery plan.

Enterprise Search Using Google Search Appliance

Leave a reply

One of the pain points for companies these days is how difficult it is to find relevant information inside their corporate network. I often hear people complain that it is easier to find any information on the Internet using Google or Bing rather than inside the enterprise.

Well, Google has been selling their Google Search Appliance (GSA) for many years. GSA brings Google superior search technology to a business corporate network. It even has the familiar look and feel that people have been accustomed to when doing a search on the Internet.

GSA can index and serve content located on the internal websites, documents located on file servers, and Microsoft Sharepoint repositories.

I recently replaced an old GSA, and quickly remembered how easy and fast it is to deploy. The hardware of the GSA is a souped up Dell server with a bright yellow casing. Racking the hardware is a snap. It comes with instructions on where to plug the network interfaces. The initial setup is done via a back-to-back network connection to a laptop, where network settings such as the IP address, netmask, gateway, time server, mail server, etc are configured.

Once the GSA is accessible on the network, the only other thing to do is to configure the initial crawl of the web servers and/or file systems, which may take a couple of hours. Once the documents are indexed, the appliance is ready to answer user search requests.

The search appliance has many advanced features and can be customized to your needs. For instance, you can customize the behavior and appearance of the search page. You can turn on or off the auto-completion feature. You can configure security settings, so that content is only available to certain people that are properly authenticated, and many other features.

Internal search engines such as the Google Search Appliance will increase the productivity of corporate employees by helping them save time looking for information.

Leading Several Toastmasters Clubs as Area Governor

Leave a reply

There are many leadership opportunities in Toastmasters. One of those opportunities is serving as Area Governor – where one leads and oversees several clubs in a geographical area. I have been honored and privileged to serve as Area Governor of Area 53, District 31 (Eastern Massachusetts and Rhode Island) in the past Toastmaster year (July 2013 to June 2014). I am proud that during my tenure, my area has earned the President’s Distinguished Area award.

Area Governors help clubs succeed. They visit the clubs several times a year, determine the clubs’ strengths and weaknesses, create club success plans with club officers, encourage members to finish their speech and leadership projects, and help the club obtain new members. In addition, area governors facilitate area speech contests – one of the most important tradition in Toastmasters.

As Area Governor, I have encountered many challenges but the time and effort I spent has been well worth it. I met and worked with various people, learned to work with different personalities, and nurtured relationships. I guided a struggling club to become a great club. I have strengthened my leadership skills in the process, and learned how to truly motivate and inspire.

There are so many opportunities in Toastmasters to learn and lead. You just need to step up.

Installing High Performance Computing Cluster

Leave a reply

A high performance computing (HPC) cluster is usually needed to analyze data from scientific instruments. For instance, I recently setup an HPC cluster using Red Hat Enterprise Linux 6.5 consisting of several nodes which will be used to analyze data generated from a gene sequencer machine.

Basically, to build the cluster, you need several machines with high speed processors and multiple cores, lots of memory, a high speed network to connect the nodes, and a huge and fast data storage. You also need to install an operating system – such as the Red Hat or CentOS Linux, and configure tools and utilities such as kickstart, ssh, NFS, and NIS. Finally, a cluster software or queueing system is needed to manage jobs to fully utilize the compute resources. One of the commonly used open source cluster software is Son of Grid Engine (SGE)  – an offshoot of the popular Sun Grid Engine.

An excellent write up for setting up an HPC cluster can be found at this Admin article.

The latest Son of Grid Engine version (as of this writing) is 8.1.7 and can be downloaded from the Son of Grid Engine Project Site.

Since the environment I setup is running Red Hat Enterprise Linux 6.5, I downloaded and installed the following rpms:

gridengine-8.1.7-1.el6.x86_64.rpm
gridengine-execd-8.1.7-1.el6.x86_64.rpm
gridengine-qmaster-8.1.7-1.el6.x86_64.rpm
gridengine-qmon-8.1.7-1.el6.x86_64.rpm

After the installation of the rpms, I installed and configured the qmaster, then installed sge (execd) on all the nodes. I also ran a simple test to verify that the cluster is working by issuing the following commands:

$ qsub /opt/sge/examples/jobs/simple.sh
$ qstat

MIT Sloan CIO Symposium

Leave a reply

I recently attended the MIT Sloan CIO Symposium held in Cambridge, MA on May 21st, 2014. The event was well attended by Chief Information Officers (CIOs), VPs, business executives, academics, entrepreneurs, and professionals from companies all over the world. The speaker lineup was superb, the symposium was content rich, and the #mitcio twitter hashtag was trending during the event.

I enjoyed the symposium because of the combination of academic and business perspective coming from a diverse set of speakers. Hot topics such as security, big data, robotics and self-driving cars and its implications to society, and the evolving role of CIOs were big topics of conversation.

The key takeaways for me are the following:

1. The future role of CIOs and IT professionals in general will be service brokers. They will increasingly serve as in-house IT service providers, and as brokers for business managers and external cloud service providers.

2. On the issue of “build vs buy, and when does it make sense to build your own system”, the answer is — when it is a source for your competitive advantage, or when what you build will differentiate your business from anyone else.

3. CIOs increasingly have to work closely with the business to deliver on technology promises rather than focusing on the technology alone. They should have a seat at the executive table. CIOs need to stay in front of their organizations and should talk to boards regularly. They should be communicating the risks of IT investments and demonstrate its benefit to the business.

4. To maximize and communicate the business value of IT, use the following sentence when explaining the benefits of IT to business: “We are going to do ___, to make ___ better, as measured by ___, and it is worth ____.” Also, consider “you” and “your” as the most important words when translating the business value of IT.

5. In terms of the future of technology – everything is becoming data-fied. Brynjolfsson, the author of the book, “The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies” said that “We are at the cusp of a 10 year period where we go from machines not really understanding us to being able to.” Thus we are seeing tremendous technological advancements in robotics and self-driving cars. With all these technological progress, we also have to think about how our culture, laws, ethics, and economics will be affected. For instance, how will employment be affected by robots that can generally do repetitive tasks? The advice from the panel is that “creative lifelong learners will always be in demand.”

Redefining Data Center In A Box

Leave a reply

Data center in a box is traditionally defined as a “type of data center in which portable, mobile, and modular information nodes are self-contained within a cargo container. It is designed and packaged for quick deployment and acquisition of data center solutions in organizations or facilities, including remote off-site locations.” Data center in a box usually contains equipment from large storage, compute, and network vendors such as EMC, NetApp, Dell, and Cisco. They are pieced together to form the IT infrastructure. Virtual Computing Alliance (VCE) for instance, offers Vblock, a bundled product containing EMC storage, Cisco servers, and VMware. NetApp has a similar offering called Flexpod.

But new innovative companies such as Simplivity, Nutanix, and Scale Computing are changing the definition of data center in a box. They are creating a purpose-built product from the ground up that incorporates not just compute, storage, and network, but additional services such as data deduplication, wan optimization, and backup in a box.

For instance, Simplivity’s product called OmniCube is “a powerful data center building block that assimilates the core functions of server, storage and networking in addition to a wide range of advanced functionality including: native VM-level backup, WAN optimization, bandwidth efficient replication for DR, cache accelerated performance, and cloud integration.”

These products will further simplify the design, implementation, and operation of IT infrastructure. With these boxes, there is no more storage area network (SAN) to manage, nor additional appliances such as WAN accelerator to deploy. A few virtual machine (VM) administrators can manage all the boxes in a cluster from the VMware server virtualization management user interface.

Data center in a box will continue to evolve and will change how we view and manage IT infrastructure for years to come.

Data Migration Using PowerPath Migration Enabler

Leave a reply

One project I recently led is the migration of data from an old EMC Clariion to the new EMC VNX. There are a couple of strategies for migrating block data on a storage area network (SAN) – either use storage-based migration (migration is between the two storage arrays) or use host-based migration (migration is done on the host). EMC provides several tools for accomplishing these tasks. SAN Copy for instance is an excellent storage-based migration tool.

There are many factors to consider when choosing a migration strategy – size of data, cost, SAN bandwidth, complexity of the setup, application downtime, among many others. One strategy that is relatively simple and requires no downtime is to use the host-based migration tool PowerPath Migration Enabler Hostcopy.

This tool is part of PowerPath when you install the full software. In version 5.7 SP2, as long as the PowerPath is licensed, there is no additional license needed for Hostcopy (unlike the older version).

The migration process is non disruptive. It does not require shutting down the application. The host is still operational while migration is going on. In general, the steps for migrating data are:

1. On Windows or Linux host, make sure Powerpath 5.7 SP2 is installed and licensed.

powermt check_registration

2. Check source disk and record the disk pseudo name.

powermt display dev=all

3. On new storage, present the target LUN to host.

4. On host, rescan and initialize the target disk.

5. Check that the target disk is present and record the pseudo name.

powermt display dev=all

6. Setup the PowerPath Migration Enabler session

powermig setup -src harddiskXX -tgt harddiskYY -techType hostcopy

7. Perform initial synchronization

powermig sync -handle 1

8. Monitor status of the session

powermig query -handle 1

9. The data transfer rate can also be throttled

powermig throttle -throttleValue 0 -handle 1

10. When ready to switch over to the new storage, enter the following command:

powermig selectTarget -handle 1

11. Commit the changes

powermig commit -handle 1

12.Cleanup/delete the session

powermig cleanup -handle 1

13. Remove the old storage by removing lun from the old storage group

14. On host, rescan HBA for hardware changes, then remove old LUNs from PowerPath

powermt display dev=all
powermt remove dev=all
powermt display dev=all

For more information about PowerPath Migration Enabler, visit EMC website.

EMC VNX2 Storage Array Review

Leave a reply

VNX is EMC’s unified enterprise storage solution for block and file. The latest release called VNX2, uses the advanced Intel Sandy Bridge processor with more cores. It also has more memory (RAM).

It’s Fast VP technology which dynamically moves data between SSD (flash drives), SAS drives and NL-SAS tiers, is now improved by decreasing the data “chunk” from 1GB to 256MB, which allows greater efficiency of data placement. Also, using SSD as the top tier is new in VNX2.

It’s Fast Cache technology has also been improved. Per EMC, “the warm up time has been improved by changing the behavior that when the capacity of FAST Cache is less than 80% utilized, any read or write will promote the data to FAST Cache.”

VNX2 boasts of its active/active LUNs configuration. However, active/active LUNs only work when the LUN is provisioned using RAID Groups. It does not work with Storage Pools. Hopefully, active/active LUNs will be available for Storage Pools in the future because more and more LUNs are being configured using Storage Pools instead of RAID Groups.

Another improvement is that in Unisphere, storage administrators do not need to set the storage processors (SP) cache settings – read and write cache settings and high and low water marks. It needs only to be turned on or off. The system now adjusts the cache settings automatically.

There are also no hot spare drives now. You simply don’t provision all the drives, and a blank drive becomes a hot spare. You can set the hot spare policy for each type of drive. The recommended is 1 per 30 drives.

I noticed a couple of shortcomings in this release. I do not like the fact that when creating a LUN in a pool, the “thin” is checked by default now. I believe that thick LUNs should be the default because of performance considerations. In addition, if storage administrators are not careful, they may end up over-provisioning the pool with thin LUNs.

On the file side, there is really no major improvement. I believe there is no updates on the data movers. Data movers still function in active/passive mode. One change though is that you can now use VDM (Virtual Data Mover) for NFS, although to configure this, you need to use the CLI.

Overall, VNX2 is one of the best enterprise storage array in terms of its performance and functionality.

Avamar Backup Solution Review

Leave a reply

I recently completed a hands-on training on Avamar management and was impressed by its deduplication technology. Deduplication happens at the client side which means less bandwidth consumed on the network and less space used for storage. Backup jobs run very fast, especially the subsequent backups after the initial baseline backup.

The Avamar disk-based backup appliance is based on Linux operating system, thus its Command Line Interface (CLI) commands are excellent. Its Redundant Array of Independent Nodes or RAIN architecture provides failover and fault tolerance across its storage nodes. It can also integrate with Data Domain as its backend storage nodes. It has an intensive support for VMware and NAS appliances via the NDMP accelerator device. Avamar servers can be replicated to other Avamar servers located at a disaster recovery site. The management GUI is intuitive for the most part and it’s very easy to do backup and restore.

However, I also found several shortcomings that I think could improve the product. First, the management GUI does not have an integrated tool to push the agent software to the clients. If you have hundreds of clients, you need to rely on third party tools such as Microsoft SMS to push the agent software. Second, there is no integrated management GUI. You have to run several tools to perform management tasks – the Avamar Administrator Console, Avamar Client Manager, Enterprise Manager, and Backup Recovery Manager. Third, there is no extensive support for Bare Metal Restore (BMR). Only Windows 2008 and later are supported for BMR. Finally, the system requires a daily maintenance window to perform its HFS checks and other tasks, during which very few backup jobs are allowed to run. This should not be a big deal though since a short backup window is usually enough to finish backup jobs because as I mentioned earlier, backups run very fast.

Overall, I consider Avamar coupled with the Data Domain appliance as the leading backup solution out there.

IT Infrastructure Qualification and Compliance

Leave a reply

One of the requirements of building and operating an IT infrastructure in a highly regulated industry (such as the pharmaceutical industry, which is regulated by the FDA) is to qualify, or validate the servers, network, and storage when they are being built. Once built, any changes to the infrastructure should undergo a change control procedure.

Building the infrastructure and making changes to it should undergo verification. They should also be documented so that they can be easily managed and traced. These activities are really not that different from the best practices guide in operating an IT infrastructure, or even from the ITIL processes.

FDA does not dictate how to perform IT infrastructure qualification or validation, as long as you have documented reasonable procedures.

The problem is that some companies overdo validation and change control processes. The common problems I’ve seen are: (1) too many signatures required to make a change, (2) no automated procedure to perform the documentation – many still use papers to route documents (3) and finally, the people who perform the checks and balances sometimes do not understand the technology.

The result is that IT personnel get overwhelmed with paperwork and bureaucracy. This discourages them to make critical changes to the infrastructure such as performing security patches on time. This also leads to the relunctance of IT personnel to implement newer or leading edge technologies into their infrastructure.

Fortunately, the International Society for Pharmaceutical Engineering (ISPE) has published a Good Automated Manufacturing Practice (GAMP) guidance on IT Infrastructure Control and Compliance. Companies can create their own IT infrastructure qualification program and procedures based on the GAMP guidance document. They should be simple but comprehensive enough to cover all the bases. It is also important that these procedures be periodically reviewed and streamlined to achieve an optimized procedure.