A high performance computing (HPC) cluster is usually needed to analyze data from scientific instruments. For instance, I recently setup an HPC cluster using Red Hat Enterprise Linux 6.5 consisting of several nodes which will be used to analyze data generated from a gene sequencer machine.

Basically, to build the cluster, you need several machines with high speed processors and multiple cores, lots of memory, a high speed network to connect the nodes, and a huge and fast data storage. You also need to install an operating system – such as the Red Hat or CentOS Linux, and configure tools and utilities such as kickstart, ssh, NFS, and NIS. Finally, a cluster software or queueing system is needed to manage jobs to fully utilize the compute resources. One of the commonly used open source cluster software is Son of Grid Engine (SGE)  – an offshoot of the popular Sun Grid Engine.

An excellent write up for setting up an HPC cluster can be found at this Admin article.

The latest Son of Grid Engine version (as of this writing) is 8.1.7 and can be downloaded from the Son of Grid Engine Project Site.

Since the environment I setup is running Red Hat Enterprise Linux 6.5, I downloaded and installed the following rpms:

gridengine-8.1.7-1.el6.x86_64.rpm
gridengine-execd-8.1.7-1.el6.x86_64.rpm
gridengine-qmaster-8.1.7-1.el6.x86_64.rpm
gridengine-qmon-8.1.7-1.el6.x86_64.rpm

After the installation of the rpms, I installed and configured the qmaster, then installed sge (execd) on all the nodes. I also ran a simple test to verify that the cluster is working by issuing the following commands:

$ qsub /opt/sge/examples/jobs/simple.sh
$ qstat