Category Archives: Virtualization

Redefining Data Center In A Box

Data center in a box is traditionally defined as a “type of data center in which portable, mobile, and modular information nodes are self-contained within a cargo container. It is designed and packaged for quick deployment and acquisition of data center solutions in organizations or facilities, including remote off-site locations.” Data center in a box usually contains equipment from large storage, compute, and network vendors such as EMC, NetApp, Dell, and Cisco. They are pieced together to form the IT infrastructure. Virtual Computing Alliance (VCE) for instance, offers Vblock, a bundled product containing EMC storage, Cisco servers, and VMware. NetApp has a similar offering called Flexpod.

But new innovative companies such as Simplivity, Nutanix, and Scale Computing are changing the definition of data center in a box. They are creating a purpose-built product from the ground up that incorporates not just compute, storage, and network, but additional services such as data deduplication, wan optimization, and backup in a box.

For instance, Simplivity’s product called OmniCube is “a powerful data center building block that assimilates the core functions of server, storage and networking in addition to a wide range of advanced functionality including: native VM-level backup, WAN optimization, bandwidth efficient replication for DR, cache accelerated performance, and cloud integration.”

These products will further simplify the design, implementation, and operation of IT infrastructure. With these boxes, there is no more storage area network (SAN) to manage, nor additional appliances such as WAN accelerator to deploy. A few virtual machine (VM) administrators can manage all the boxes in a cluster from the VMware server virtualization management user interface.

Data center in a box will continue to evolve and will change how we view and manage IT infrastructure for years to come.

Best Practices for Using NFS Datastore on VMware

More companies are now deploying VMware with IP based shared storage (NAS). NAS storage is cheaper than Fiber Channel (SAN) storage because there is no separate Fiber Channel (FC) based network to maintain. More importantly, IP based storage performance and stability are now comparable with FC based storage.

Other advantages of using IP based storage, specifically NFS, are thin provisioning, de-duplication, and the ease-of-backup-and-restore of virtual machines and files on a virtual disk via array based snapshots. In addition, IP based storage is easier to maintain.

VMware published a whitepaper on the best practices for running VMware vSphere on Network Attached Storage (NAS) using NFS. Following the best practices in deploying an NFS based storage is very important to obtain a stable and optimized VMware environment. Here are the important things to consider:

On the network side, the local area network (LAN) on which the NFS traffic will run needs to be designed with availability, downtime-avoidance, isolation, and failover:

1. NFS traffic should be on a separate physical LAN, or at least on a separate VLAN.
2. Use private (non-routable) IP addresses. This will also address a security concern since NFS traffic is not encrypted and NFS is mounted with root privileges on the VMware host.
3. Use redundancy by teaming the NICs on the VMware host, configuring LACP protocol, and using two LAN switches.
4. Use jumbo frames.
5. Use 10GB Ethernet.

On the storage array side, the storage controller must be redundant, in case the primary one fails. In addition,

1. Configure the NFS exports to be persistent. (e.g. exportfs –p)
2. Install the VAAI and other plug-in tools from the storage vendor. For instance, NetApp has the Virtual Storage Console (VSC) plug-in that can be installed on the vCenter.
3. Refer to the storage vendor best practices guide. For instance, NetApp and EMC published their own best practice whitepapers for using NFS on VMware.

On the VMware hosts, the following configuration should be implemented:

1. Use the same datastore name across all hosts.
2. Select “No” for NIC Teaming failback option. If there is some intermittent behavior in the network, this will prevent the flip-flopping of NIC cards being used.
3. If you increase the maximum number of concurrent mount points (from the default of 8), also increase Net.TcpipHeapSize as well. For instance, if 32 mount points are used, increase tcpip.Heapsize to 30MB.
4. Set the following VMware High Availability options: (NFS heartbeats are used to determine if an NFS volume is still available.)
NFS. Hearbeat.Frequency=12
NFS.Hearbeat.Timeout=5
NFS.Hearbeat.MaxFailure=10

When configured properly, IP based storage, specifically NFS, provides a very solid storage platform for VMware.

NetApp Virtual Storage Console for VMware vSphere

One of the best tools for managing NetApp storage and VMware is a plug-in called NetApp Virtual Storage Console (VSC) for VMware vSphere. VSC provides administrators the ability to manage NetApp storage from vCenter client. It can configure, monitor, provision, and migrate NetApp datastores with fewer clicks. In addition, it can perform backup and recovery of LUNs and volumes from the vCenter client.

VSC can automatically discover your NetApp storage controllers and ESXi hosts. This task can take a lot of time if not using VSC. VSC can also automatically apply “best practices” settings on the ESXi host to optimize its configuration. It can rapidly provision datastores without going through the NetApp management interface. You can get backup (snapshots) of the datastore in a consistent state, and perform recovery in minutes.

NetApp implementation of its vStorage API for Array Integration (VAAI) offloads significant processing tasks to the storage array, freeing ESXi resources for other tasks. If you are using NFS though, you still need to download and install the NetApp NFS Plug-in for VMware VAAI.

For now, the VSC plug-in is only available for the traditional vCenter client. VMware is moving towards replacing the traditional vCenter client with the vSphere Web Client. I hope that NetApp releases the plug-in version for the web client pretty soon.

The Value of IT Certifications

I recently passed the VMware Certified Professional 5 – Data Center Virtualization exam. The last VMware certification I took was in 2007 when I passed the VMware Certified Professional 3 exam. It’s nice to have the latest VMware certification under my belt.

VMware certification is a little bit unique, because it requires one-week training and hands-on experience. You will find it difficult to pass the test without hands-on experience. Most of the questions in the test are real life scenarios and you can only understand the questions if you have encountered them in real life.

Some people argue the value of certifications. They say that certifications are useless because most of those people who have them are inexperienced. I agree that experience is the best way to learn in the IT field. I can attest to this after almost 20 years in the field. But IT certifications are valuable for the following reasons:

1. Not all IT certifications are created equal. While some certifications are easier to pass just by reading books, most IT certifications such as VCP (VMware Certified Professional), CISSP (Certified Information Systems Security Professional), and RHCE (Red Hat Certified Engineer) certifications need a high degree of experience to pass the tests.

2. Not all people are lucky enough to have access to expensive hardware to gain hands-on experience nor lucky enough to be assigned to IT projects to get the maximum exposure. Some people take the certification route to get knowledge and experience.

3. Not all IT knowledge is learned via experience since not all scenarios can be encountered in real life. Some are learned via reading books and magazines, taking the training, and passing certification tests. For instance, if your company’s standard is Fiber Channel for VMware datastore, the only way to learn about iSCSI datastore is to read or get trained on it.

4. IT certifications are solid evidence of your career. It will be very useful, for instance, when looking for a job. Prospective employers do not have a concrete evidence of your accomplishments, but a solid and trusted IT certification can prove your worth.

5. And finally, seasoned IT professionals, just like me, take certification tests to validate our knowledge.

Important Features and Capabilities in the New vSphere 5.1

vSphere 5.1 has been released several months ago, and among its new features and capabilities, I think the important ones are the following:

1. Larger virtual machines. Virtual machines can now have up to 64 virtual CPUs (vCPUs) and 1TB of virtual RAM (vRAM). This means that enterprise apps such as SAP, large databases, email, and other high demand apps can now be virtualized without worrying about CPU and memory resources. The goal of attaining 100% virtualization in the Data Centers can now be realized.

2. vSphere Data Protection and vSphere Replication. vSphere Data Protection is used to backup and recover virtual machines. vSphere Replication is used to replicate virtual machines to remote Data Center for disaster recovery. No need to integrate third party tools such as Veeam for backup or Zerto for replication to remote DR site.

3. vSphere 5.1 eliminates the need to reboot virtual machines for subsequent VMware Tools upgrades on Windows.

For a complete list of the new features and capabilities, go to this website:

http://www.vmware.com/files/pdf/products/vsphere/vmware-what-is-new-vsphere51.pdf

However, the vSphere Web Client is now the core administrative interface for vSphere. The vSphere client is still available but I’m afraid it will not be supported in future releases. I still like the vSphere client because it’s more roubust, stable, and faster. In addition, there are a lot of plug-ins that are still not available in the Web Client such as the NetApp Virtual Storage Console for VMware vSphere. We use NetApp for our datastore and this plug-in is very important to us.

Easy and Cheap vCenter Server

If your VMware infrastructure contains no more than 5 hosts and 50 virtual machines, you can save some effort and Windows license fee by using the VMware vCenter Server Appliance instead of the vCenter Server on a Windows machine. The vCenter Server Appliance is a preconfigured Suse Linux-based virtual machine, with PostgreSQL for the embedded database.

The vCenter appliance is easy to deploy and configure, and it will save you time and maintenance effort, because unlike Windows, you do not have to install anti-virus and monthly patches. It can join Active Directory for user authentication. It will save you Windows license fee, but you still need to purchase vCenter license.

The vCenter appliance can be downloaded from the VMware site as an ova or an ovf plus vmdk files. You do not need to download the ovf and the vmdk files if you downloaded the ova file. Ova file is merely a single file distribution of ovf and vmdk, stored in tar format.

To deploy the appliance, use the vSphere Client and deploy the downloaded ova file as an ovf template. You can deploy it as a thin provisioned format if you do not want to commit 80GB space right away. Once deployed and powered on, you can continue with the rest of the configuration using the GUI browser based interface at https://vCenterserver:5400/. The vCenter Server Appliance has the default user name root and password vmware.

The wizard will guide you through the rest of the configuration. There are really very few configuration items. The common ones are static IP address (if you don’t want dhcp), and the Active Directory settings. And the best thing is, you do not have to manage/configure the Suse-Linux-based appliance via CLI. Everything can be managed via the GUI browser-based interface.

Cloning Linux on VMware

When you clone or ‘deploy from template’ a linux virtual machine on Vmware, specifically Red Hat based linux such as CentOS, you need additional steps on the cloned machine to make it work. The obvious settings you need to change are the IP address and hostname. But changing those settings is not enough. You also need to change other parameters.

When you clone a linux machine, the hardware address (or MAC address) of the NIC changes, which is correct — the cloned machine should never have the same MAC address as the source. However, the new MAC address is assigned to eth1, not eth0. The eth0 is still assigned the MAC address of the source, although it is commented out in udev’s network persistent file, so it’s not active.

When you cloned a linux machine and noticed that the network does not work, it is probably because you assigned the new IP address to eth0 (which is not active). You can use eth1 and assign the new IP address on that interface. However, I usually want to use eth0 to make it clean and simple. You can easily switch back to eth0 by editing the file /etc/udev/rules.d/70-persistent-net.rules. Edit the string that starts with SUBSYSTEM, remove or comment out the line for eth1, uncomment the line for eth0, and replace the ATTR(address) for eth0 to get the MAC address from eth1. Here’s a sample edited file:

# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:60:66:88:00:02",
ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x100f (e1000)
#SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:60:66:88:00:02",
ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

Now edit the /etc/sysconfig/network-scripts/ifcfg-eth0 file to make sure that the DEVICE is eth0, the BOOTPROTO is static, and the HWADDR matches the ATTR{address} for eth0 in the 70-persistent-net.rules file.

Restart the network by issuing the command “service network restart” or you can reboot the system.

Vmware Datastore via NFS

One of the objectives of our recently concluded massive storage upgrade project, was to replace our vmware datastore from iSCSI to NFS. I have been hearing the advantages of using NFS versus block-level storage (ie, iSCSI or Fiber Channel), and true enough NFS did not disappoint.

We have been using iSCSI on NetApp as datastore on vmware for a long time, and it has been running pretty well. But when we perform maintenance on the NetApp storage, the virtual machines were often times affected. In addition, restore procedures can be a pain.

While Fiber Channel (FC) is still the standard storage for most vmware implementation because of its proven technology, in my experience here are the advantages of using NFS over iSCSI or FC:

1. Robust, as long as you follow the best practices guidelines. For instance, separate the NFS network from the general use network. Vmware and NetApp released white papers discussing the NFS datastore best practices. In our environment, we did several failover on the NetApp storage to upgrade the Data ONTAP version, and the virtual machines were never affected.

2. Easier to configure both on the vmware side and the NetApp side.

3. Easier to backup, via NDMP on the NetApp side.

4. Easier to restore vmdk files using the snapshots on the NetApp side, since there is no need to mount LUNs.

5. Vmware and NetApp built great tools for seamless maintenance and operations.

Book Review: The Big Switch – Rewiring the World from Edison to Google

The Big Switch: Rewiring the World from Edison to Google. Nicholas Carr. New York: W. W. Norton and Company, 2008. 278 pp.

The future of computing, the book argues, is utility computing. Information Technology (IT) will reside “in the cloud” in a centralized fashion, and will be controlled by a few service providers who have built massive data centers. Just like electricity, IT will be delivered as a service to home users and to small and big companies. The IT departments of these companies may become irrelevant. There will be no need for them because “individuals and business units will be able to control the processing of information directly.”

High bandwidth availability makes utility computing possible. Soon, companies will outsource all of their IT functions from storage to applications to programming, to service providers. As a service provider, Google has started this trend with their Google Apps. Similarly, Amazon has offered software and hardware as a service. For instance, if a company needs an application, all they have to do is tell one of these service providers and the application will be available in no time. They don’t have to go through the hassle of procuring equipment, hiring programmers, and developing the application.

This next big thing has many names – cloud computing, utility computing, grid computing, and software/hardware as a service (SAAS) – but the book called it the World Wide Computer.

The premise of the switch from internal IT to the World Wide Computer is that too many resources are wasted on IT – labor, hardware, software, redundant systems, and overbuilt IT assets. The book contends that IT costs too much for what it delivers. There is just an excess in servers and computing capacity. Ultimately, it’s not the technology but the economics of it that will prevail. The cloud will make efficient use of IT resources.

Because everything is wired, physical location will not matter anymore. The same is true with software licensing. The model will be much like the electricity – the client pays for usage, not the costly software license that have made companies like Microsoft very rich. The new model, the book argues is very much like the Google Apps model. Users will be empowered when tapping the World Wide Computer – the possibilities are endless with its infinite information and computing power.

For people who have been following the computing revolution, Carr’s concept of utility computing is old news. IBM and other IT visionaries have been talking about utility computing for years. However, his book has successfully articulated the concept by drawing the parallelism of the evolution of electrification and the evolution of computing.

The history of electrification was well researched from the first waterwheels to windmills to the current centralized power generators. Similarly, the history of computing was well researched too, from Hollerith’s machine to IBM mainframe to personal computing, to client-server computing, and web computing. Along the way, Carr infused the business and economic forces that shaped their current form. He likewise talked about the social impacts of these – how it has changed societies and consequently changed people’s lives for the better. He discussed in great length the economic and social impact of the World Wide Computer – how the world will become more increasingly multi-polar instead of being united, the weaknesses of free flowing information, and the loss of human privacy.

Inasmuch as I agree with Carr’s position of utility computing, I do not believe that everything will go to the “cloud”. In my opinion, the future will be hybrid computing. There is so much computing power in every personal computer, laptop and mobile device that not utilizing them is a waste.
The IT department of large corporations will not disappear. The book missed the point that for some companies, the IT system is strategic, and they cannot simply outsource all of their IT functions. For instance, financial companies rely heavily on their IT system. Take it away from the stock market, for example, and trading will halt. The point is that: IT has varying degrees of importance for each company. But for electricity, there is none. Everybody needs electricity since it’s a commodity and can easily be sourced from other sources (such as using internal generators). IT cannot simply be commoditized – companies need specialized applications.

Another issue is data security and privacy. In the cloud, we don’t know where the data is stored. Intellectual property and company knowledge are just too important for the company to be hosted somewhere where security and privacy laws are not well defined. Unless there is a global law on data security and privacy, companies will hesitate to put their precious information in the cloud.

Finally, there is the law of unintended consequences. We cannot simply have a complete picture of the future. It is ironic for instance that because of the current concern for the environment, companies and homes alike may be generating their own power using solar, windmill or other means, thus decentralizing the electricity generation once again. The use of electrification as a metaphor for the World Wide Computer may not be accurate after all.

Performing maintenance tasks on vmware hosts

There are times when you need to perform hardware maintenance (such as adding a new Network Interface Card [NIC]) on VMware hosts, or the host simply disconnects from vCenter.  The only way to perform maintenance is to shutdown or reboot the hosts.  To minimize damage, here’s the procedure I use:

  1. Run vSphere client on the workstation.  Do not use the vSphere client on the servers. The reason being – a server might be a virtual machine (VM) which will go down.
  2. Using vSphere client, connect to VMware host, *not* the vCenter server.
  3. Login as user root.
  4. Shutdown all the VM’s, by right clicking the VM, selecting Power, Shutdown Guest.  This is faster than logging in to each machine using RDP and shutting it down.  The vmtools though have to be up to date, or else the Shutdown Guest option will be grayed out. If Shutdown Guest is grayed out, you need to login to the VM to shut it down.  Performing “Power Off” on the VM should be the last resort.
  5. Once all the VM’s are powered down, right click on the VMware host and select Enter Maintenance Mode.
  6. Go to the console of the VMware host, and press Alt-F11 to get the login prompt.
  7. Login as root.
  8. Issue the command “shutdown -h now” to power down the host.  If you just want to reboot, issue the command “shutdown -r now”.
  9. Wait until the machine is powered off.
  10. Perform maintenance.
  11. Power on the VMware host.  Look for any problems on the screen.  The equivalent of blue screen in VMware is purple screen.  When there’s a purple screen, that means there is something very wrong.
  12. When the VMware host is all booted up, go back to your workstation, and connect using vSphere client to the VMware host.
  13. Right click on the Vmware host first, and select “Exit Maintenance Mode”
  14. Power On all the VM’s.

If there are multiple VMware hosts, and Vmotion is licensed and enabled (i.e. Enterprise License), you can vmotion VMs to the other hosts, and perform maintenance.  When the host gets back, you can vmotion back the VM’s to the host, and do the same maintenance on the other.