Archives for August 2009

HowTo monitor RedHat Cluster using snmp and python

Aug 282009

Now that I am done with the implementation of RHE Cluster with GFS2, I now need to setup monitoring. As you all know, monitoring is a vital part of any environment. Even though we have a cluster of nodes setup, we still need to be aware of what is happening. Currently here are 2 very important tools, for checking the cluster status.

clustat ( which is installed by the rgmanager rpm ) The clustat command will give you a quick status about all the nodes in the cluster and of the services running.
cman_tool ( which is installed by the cman rpm ) The cman_tool command is for managing a node in the cluster ( leaving, joining, votes, and status ).

Here is an example of clustat…

clustat Cluster Status for MyCluster @ Sat Aug 29 18:37:27 2009Member Status: Quorate Member Name ID Status ------ ---- ---- ------ gfs1 1 Online, Local, rgmanager gfs2 2 Online, rgmanager gfs3 3 Online, rgmanager /dev/disk/by-path/pci-0000:00:11.0-scsi-0:0:1:0-part1 0 Online, Quorum Disk Service Name Owner (Last) State ------- ---- ----- ------ ----- service:CIM gfs1 started service:Pirahna (gfs2) disabled

Continue reading “HowTo monitor RedHat Cluster using snmp and python” »

HowTo setup a Quorum Disk

Clustering No Responses »

Aug 242009

Today’s tutorial will be on the infamous Quorum disk. When I first setup my GFS2 shared Cluster of 3 nodes, I was quite impressed with the fact that 3 nodes were sharing the same file system. Now that everything was up and running, I wanted to see what would happen if I brought down, 2 out of the 3 nodes in the cluster. I turned off 1st node and all was well, I was still able to access my GFS2 mount on the other 2 nodes. Then I decided to reboot the 2nd node, and guess what happened???? QUORUM DISSOLVED!!! Now on my final node the GFS2 file system was still mounted but I could not touch a file or run a ls on the mount… It just hung there!!

Well I knew this was not going to be acceptable…. Since if I still have 1 node available, the node should still be able to use the GFS2 mount. So I did some research about this quorum disk and what it can do for me. Let me tell you, this was exactly what I was looking for. 1st let me start out my explaining what a quorum is ( relating to clustering ). A quorum is the minimal number of votes that is needed in a cluster, usually the majority. So if you have 3 nodes in a cluster, that means you have a total of 3 votes in the cluster and you will need a minimum of 2 votes to remain in a quorate state. Which means you can lose 1 node in the cluster and the other nodes are still functional. But if you lose 2 nodes, your quorum will be dissolved. Which means even though your GFS2 file system is still mounted on your final node, it will not be accessible to you.

The quorum disk will help this particular situation… You ask how???? Well here it is… Qdisk needs at a minimum of a 10MB disk partition shared across the cluster.
Qdiskd runs on each node in the cluster, periodically checking its
own health and then placing its state information into its assigned
portion of the shared disk. On each node qdiskd then looks at the state of
the other nodes in the cluster as posted in their area of the qdisk
partition. When all the nodes that are running qdiskd are in a healthy state, the quorum of the cluster is increased by the value of the shared Quorum Disk.

The Value of the Quorum Disk should be n-1 ( Number of nodes – 1 ). In this case the Value should be 2 ( 3 -1 ).

Continue reading “HowTo setup a Quorum Disk” »

HowTo Increase GFS2 Performance in a Cluster

Clustering No Responses »

Aug 142009

In the last HowTo, I showed you how to setup GFS2 file system with Red Hat Clustering. I will now show you how to optimize the performance of your GFS2 mounts. The gfs_controld daemon manages the mounting, unmounting, and the recovery of the GFS2 mounts. gfs_controld also manages the posix lock.

By default the plock_rate_limit option is set to 100. This will allow a maximum of 100 locks per second, which will decrease your GFS2 performance. See below…

You can test the performance of you cluster by downloading the program ping_pong.c. This program was very helpful to me in debugging the poor performance in my GFS2 cluster.
The instructions on how to compile the program and run it is on the site http://wiki.samba.org/index.php/Ping_pong.When I initially ran ping_pong, I only got a max of 97 plocks per second. After removing the rate limit I was able to get about 3000 Plocks per second.

Continue reading “HowTo Increase GFS2 Performance in a Cluster” »

HowTo setup GFS2 with Clustering

Clustering 3 Responses »

Aug 142009

In my last project at work, I had to replace NFS with GFS2 and Clustering. So in this tutorial I will show you how to create a Red Hat or CentOS cluster with GFS2. I will also show you how to optimize GFS2 performance in the next HowTo, because you will quickly notice some loss of performance until you do a little optimization first.I will 1st show you how do build a Cluster with GFS2 on the Command Line and in the next tutorial I will show you how to do the same thing using Conga.

In this tutorial I am using 3 CentOS Virtual Machines running CentOS 5.3 in VMware ESX 3.5. For the GFS2 File System I am using a vmdk built with the thick option, that is shared among all the Virtual Machines. You also can use iscsi or fiber… This option is up to you.

Always make sure your iptables (If you know the port’s and protocols for clustering, then add it to iptables ) and selinux is OFF. If not you will run into issues.

The 3 machines I am using are called

gfs1 == 192.168.101.100
gfs2 == 192.168.101.101
gfs3 == 192.168.101.103

Continue reading “HowTo setup GFS2 with Clustering” »