Category Archives: IT Strategy

The Importance of Disaster Recovery (DR) Testing

Leave a reply

Recently, we conducted disaster recovery (DR) testing on one of our crucial applications. The server was running Windows 2008 on an HP physical box. We performed bare metal restore (BMR) using Symantec Netbackup 7.1. However, after Symantec BMR completed the restore, the server will not boot up. We troubleshoot the problem and tried several configurations. It took a couple of days before we figured out the issue. The issue, by the way, was that the boot sector got misaligned after the restore and we have to use Windows installation disk to repair it.

What if it was a real server disaster? The business cannot wait for a couple of days to restore the server. We defined an RTO (Recovery Time Objective) for that server to be 8 hours. And we did not meet it during our testing. This is the reason why DR testing is very important.

During DR testing, we have to test the restore technology and the restore procedures. In addition, we need to test if we can restore it on time (RTO) and if we can restore the data at a point in time (or RPO – Recovery Point Objective) (e.g. from a day before, or from a week ago).

With a lot of companies outsourcing their DR to third parties or to the cloud, DR testing becomes even more important. How do you know if the restore works? How do you know if their DR solution meets your RPO and RTO? Companies assume that because backups are being done, then restore will automatically work.

We perform DR testing once a year. But for crucial applications and data, I recommend DR testing twice a year. Also, perform a test every time you make significant changes on your backup infrastructure, such as software updates.

Security Done Right

Leave a reply

During my job-related trip to Israel a couple of months ago, I was subjected to a thorough security check at the airport. I learned later on that everybody goes through the same process. It was a little inconvenient, but in the end, I felt safe.

With all the advance technologies in security, nothing beats the old way of conducting security – thorough checks on individuals. I also noticed the defense in depth strategy at the Israel airport – the several layers of security people have to pass to get to their destinations. No wonder some of the greatest IT security companies come from Israel (e.g. Checkpoint Firewall).

As an IT security professional (I’m a CISSP certified), I can totally relate to the security measures Israel has to implement. And companies need to learn from them. Not a day goes by that we learn companies being hacked, shamed, and extorted by hackers around the world.

Sadly, some companies only take security seriously when it’s too late – when their data has been stolen, their systems have been compromised, and their twitter account has been taken over. It will be a never ending battle with hackers, but it’s a great idea to start securing your systems now.

Getting Promoted in IT

One of the perks of serving at an Harvard alumni club (I am currently the Secretary of the Harvard-Radcliffe Club of Worcester), was attending a 2-day Alumni Leadership Conference in Cambridge, MA. It was a nice break from work. I met alumni leaders from all over the world, talked to accomplished people (I met the writer of one of my daughter’s favorite movies – Kung Fu Panda), learned what’s new in the Harvard world, and learned leadership skills from great speakers.

One of those speakers is David Ager, a faculty member at the Harvard Business School. He totally engaged the audience while delivering his opening address – “Leadership of High Performing Talent: A Case Study.” We discussed a case study about Rob Parson, a superstar performer in the financial industry. In a nutshell, Rob Parson delivered significant revenue to the company but his abrasive character and non-teamwork attitude didn’t fit well into the culture of the company. He was due for performance review and the question was – Should Rob be promoted?

The setting of the case study was in the financial industry, but the lesson holds true as well in the Information Techology (IT) industry. There are a lot of Rob Parson in IT – software developers, architects, analysts, programmers – who are high performers, but they rub other people the wrong way. They are intelligent, smart, and they develop very sophisticated software — the bread and butter of IT companies. Some of these IT superstars aspire for promotion for managerial role. Should they be promoted? Too often we hear stories about a great software architect who went to manage people, but faltered as a result.

IT professionals who would really like to manage people should be carefully evaluated for their potential. They should learn people and business skills in order to succeed. Before giving them any managerial position, they should undergo a development program and they should be under a guidance of a mentor (or a coach) for at least a year. Most IT professionals should not take on the managerial role. They should remain on their technical role to be productive, but they should be given other incentives that motivate and make them happy – such as complete authority of their work, flex time, an environment that foster creativity and so on.

BYOD

Leave a reply

Recently, I attended a security seminar on the newest buzzword in the IT industry – BYOD, or Bring Your Own Device – to complete my CISSP CPE (Continuing Professional Education) requirement for the year. The seminar was sponsored by ISC2 and the speaker, Brandon Dunlap, is a seasoned, insightful, and very entertaining speaker.  I highly recommend the seminar.

BYOD came about because of the popularity of mobile devices – iPhone, iPad, Android, Blackberry, etc.- , the consumerization of IT, and employees getting more flexible schedules.    Companies are starting to allow their employees to use their own devices – to improve productivity, mobility, and supposedly save the company money.  The millennials, in particular, are more apt to use their own devices.  Owning these devices for them signifies status symbol or a fashion statement.

However,  does it make sense to allow these devices into the company’s network?  What are the security implications of the BYOD phenomenon?

From a technology standpoint, there are a lot of innovations to secure both the mobile devices and the company’s applications and data, for instance, using containers, to separate personal apps and company’s apps.  Security companies are creating products and services that will improve the security of BYOD.  But from a policy and legal standpoint, very little is being done.  Companies who jumped into this BYOD buzz are getting stung by BYOD pitfalls as exemplified by one of the greatest IT companies in the world – IBM.   In addition, recent studies showed that BYOD does not really save company money.

Companies need to thoroughly understand BYOD before adopting it.  It is a totally new way of working.

The seminar highlighted the many problems of BYOD, and the immense work that needs to be done to make it successful.  No wonder the organizer entitled it “Bring Your Own Disaster” instead of “Bring Your Own Device.”

 

Internal Web Analytics

Leave a reply

There are a lot of tools out there that can analyze web traffic for your site. Leading the pack is Google Analytics. But what if you want statistics of your internal website, and you don’t necessarily want to send this information to an external provider such as Google? Here comes Piwik.  Piwik is very much like Google Analytics but can be installed on your internal network. The best part is that it’s free.

Since Piwik is a downloadable tool, you need to have a machine running web server and mysql. You can install it on your existing web server or on a separate web server. I installed it on a separate CentOS machine. I found the installation very easy. In fact, you just unzip a file and put those files in a web directory. The rest of the installation is via the browser. If there is a tool missing on your server, (in my case, I need the PDO extension) it will tell you how to install it. Pretty neat.

After installing the server, you just need to put a small javascript code on the pages you want to track. That’s it. Piwik will start gathering statistics for your site.

I also evaluated Splunk and it’s companion app – Splunk App for Web Intelligence, but I found that it is not ready for prime time. There are still bugs. No wonder it is still in beta. When I was evaluating, it wasn’t even able to get usable information from apache logs.

I’ve been using Awstats to extract statistics for internal websites for years. It has been very reliable but sometimes it provides inaccurate results. The open source Piwik web analytic tool provides accurate statistics and is the best tool I’ve used so far.

Security Strategy

Leave a reply

Amidst the highly publicized security breaches, such as the LinkedIn hacked passwords, hacktivists defacing high profile websites, or online thieves stealing credit card information, one of the under-reported security breaches are nation states or unknown groups stealing Intellectual Property information from companies such as building designs, manufacturing secret formulas, business processes, financial information, etc. This could be the most damaging security breach in terms of its effect on the economy.

Companies do not even know they are being hacked, or are reluctant to report such breaches. And the sad truth is that companies do not even bother beefing up their security until they become victims.

In this day and age, all companies should have a comprehensive security program to protect their assets. It starts with an excellent security strategy, a user awareness program (a lot of security breaches are done via social engineering), and a sound technical solution. A multi-layered security is always the best defense – a firewall that monitors traffic, blocks IP addresses that launches attacks, and limits the network point of entry; an IDS/IPS that identifies attacks and gives signal; a good Security Information and Event Management (SIEM) system; and good patch management system to patch servers and applications immediately once vulnerabilities are identified, to name a few.

Cost is always the deciding factor in implementing technologies. Due diligence is needed in creating cost analysis and threat model. As with any security implementation, you do not buy a security solution that costs more than the system you are protecting.

Disaster Recovery using NetApp Protection Manager

Leave a reply

In our effort to reduce tape media for backup, we have relied on disks for our backup and disaster recovery solution. Disks are getting cheaper and de-duplication technology keeps on improving. We still use tapes for archiving purposes.

One very useful tool for managing our backup and disaster recovery infrastructure is NetApp Protection Manager. It has replaced the management of local snapshots, snapmirror to Disaster Recovery (DR) site, and snapvault. In fact, it doesn’t use these terms anymore. Instead of “snapshot,” it uses “backup.” Instead of “snapmirror,” it uses the phrase “backup to disaster recovery secondary.” Instead of “snapvault,” it uses “DR backup or secondary backup.”

NetApp Protection Manager is policy-based (e.g. backup primary data every day @ 6pm, and retain backups for 12 weeks; backup primary data to DR site every day @ 12am; backup the secondary data every day @ 8am and retain for 1 year). As an administrator, one does not have to deal with the nitty-gritty technical details of snapshots, snapmirror, and snapvault.

There is a learning curve in understanding and using Protection Manager. I have been managing NetApp storage for several years and I am more familiar with snapshots, snapmirror, and snapvault. But as soon as I understood the philosophy behind the tool, it gets easier to use it. NetApp is positioning it for the cloud. The tool also has dashboards intended for managers and executives.

Vmware Datastore via NFS

Leave a reply

One of the objectives of our recently concluded massive storage upgrade project, was to replace our vmware datastore from iSCSI to NFS. I have been hearing the advantages of using NFS versus block-level storage (ie, iSCSI or Fiber Channel), and true enough NFS did not disappoint.

We have been using iSCSI on NetApp as datastore on vmware for a long time, and it has been running pretty well. But when we perform maintenance on the NetApp storage, the virtual machines were often times affected. In addition, restore procedures can be a pain.

While Fiber Channel (FC) is still the standard storage for most vmware implementation because of its proven technology, in my experience here are the advantages of using NFS over iSCSI or FC:

1. Robust, as long as you follow the best practices guidelines. For instance, separate the NFS network from the general use network. Vmware and NetApp released white papers discussing the NFS datastore best practices. In our environment, we did several failover on the NetApp storage to upgrade the Data ONTAP version, and the virtual machines were never affected.

2. Easier to configure both on the vmware side and the NetApp side.

3. Easier to backup, via NDMP on the NetApp side.

4. Easier to restore vmdk files using the snapshots on the NetApp side, since there is no need to mount LUNs.

5. Vmware and NetApp built great tools for seamless maintenance and operations.

Thoughts on Information Security

Leave a reply

I cannot stress enough the importance of information security. Almost everyday we hear stories about security breaches – hacker groups defacing websites for political purposes, countries stealing proprietary information from other countries and companies, organized crime stealing credit card information and selling those in the black market.

Cloud computing and mobile devices have exacerbated the problem.

The thing with security is that it is at odds with convenience. We want to get things done quickly, but security slows us down. For instance, we are required to enter hard to guess passwords to access our bank account online or access our company’s applications. Why not just let us in right away? Remembering passwords (and lots of them) and being required to change them every three months take some time and effort.

But if we want ourselves and our companies we work for to be secure, we should give up a little convenience. There is no other way.

A lot of technical solutions and innovations have been devised to improve information security. But no amount of technical innovation can solve the weakest link in security – social engineering. Remember the “I Love You” virus several years ago? It was a virus that was spread when you open an email with the subject line “I Love You.” Who wouldn’t want to open an email with that subject line?

User awareness is the key. Companies and individuals should at least invest in training on security and privacy.

The sad thing is that many companies and individuals do not take security very seriously, until they become victims. True, we should not spend significant amount of time and money for security. The resources we spend on security should be proportional to the assets we are protecting. You should not buy a 1 million dollar vault to protect your 100K painting.

When I obtained my CISSP certification several years ago, I didn’t plan on specializing on information security. I have, however, incorporated good security practices in system and network design and implementation, virtualization, storage, and almost all aspect of IT. But with the tremendous need for IT security professionals these days, I might consider specializing in information security.

Book Review: The Big Switch – Rewiring the World from Edison to Google

Leave a reply

The Big Switch: Rewiring the World from Edison to Google. Nicholas Carr. New York: W. W. Norton and Company, 2008. 278 pp.

The future of computing, the book argues, is utility computing. Information Technology (IT) will reside “in the cloud” in a centralized fashion, and will be controlled by a few service providers who have built massive data centers. Just like electricity, IT will be delivered as a service to home users and to small and big companies. The IT departments of these companies may become irrelevant. There will be no need for them because “individuals and business units will be able to control the processing of information directly.”

High bandwidth availability makes utility computing possible. Soon, companies will outsource all of their IT functions from storage to applications to programming, to service providers. As a service provider, Google has started this trend with their Google Apps. Similarly, Amazon has offered software and hardware as a service. For instance, if a company needs an application, all they have to do is tell one of these service providers and the application will be available in no time. They don’t have to go through the hassle of procuring equipment, hiring programmers, and developing the application.

This next big thing has many names – cloud computing, utility computing, grid computing, and software/hardware as a service (SAAS) – but the book called it the World Wide Computer.

The premise of the switch from internal IT to the World Wide Computer is that too many resources are wasted on IT – labor, hardware, software, redundant systems, and overbuilt IT assets. The book contends that IT costs too much for what it delivers. There is just an excess in servers and computing capacity. Ultimately, it’s not the technology but the economics of it that will prevail. The cloud will make efficient use of IT resources.

Because everything is wired, physical location will not matter anymore. The same is true with software licensing. The model will be much like the electricity – the client pays for usage, not the costly software license that have made companies like Microsoft very rich. The new model, the book argues is very much like the Google Apps model. Users will be empowered when tapping the World Wide Computer – the possibilities are endless with its infinite information and computing power.

For people who have been following the computing revolution, Carr’s concept of utility computing is old news. IBM and other IT visionaries have been talking about utility computing for years. However, his book has successfully articulated the concept by drawing the parallelism of the evolution of electrification and the evolution of computing.

The history of electrification was well researched from the first waterwheels to windmills to the current centralized power generators. Similarly, the history of computing was well researched too, from Hollerith’s machine to IBM mainframe to personal computing, to client-server computing, and web computing. Along the way, Carr infused the business and economic forces that shaped their current form. He likewise talked about the social impacts of these – how it has changed societies and consequently changed people’s lives for the better. He discussed in great length the economic and social impact of the World Wide Computer – how the world will become more increasingly multi-polar instead of being united, the weaknesses of free flowing information, and the loss of human privacy.

Inasmuch as I agree with Carr’s position of utility computing, I do not believe that everything will go to the “cloud”. In my opinion, the future will be hybrid computing. There is so much computing power in every personal computer, laptop and mobile device that not utilizing them is a waste.
The IT department of large corporations will not disappear. The book missed the point that for some companies, the IT system is strategic, and they cannot simply outsource all of their IT functions. For instance, financial companies rely heavily on their IT system. Take it away from the stock market, for example, and trading will halt. The point is that: IT has varying degrees of importance for each company. But for electricity, there is none. Everybody needs electricity since it’s a commodity and can easily be sourced from other sources (such as using internal generators). IT cannot simply be commoditized – companies need specialized applications.

Another issue is data security and privacy. In the cloud, we don’t know where the data is stored. Intellectual property and company knowledge are just too important for the company to be hosted somewhere where security and privacy laws are not well defined. Unless there is a global law on data security and privacy, companies will hesitate to put their precious information in the cloud.

Finally, there is the law of unintended consequences. We cannot simply have a complete picture of the future. It is ironic for instance that because of the current concern for the environment, companies and homes alike may be generating their own power using solar, windmill or other means, thus decentralizing the electricity generation once again. The use of electrification as a metaphor for the World Wide Computer may not be accurate after all.