|
Clustering HowTo's -
RedHat Clustering
|
|
Written by Allen Sanabria
|
|
Friday, 28 August 2009 10:05 |
|
Now that I am done with the implementation of RHE Cluster with GFS2, I now need to setup monitoring. As you all know, monitoring is a vital part of any environment. Even though we have a cluster of nodes setup, we still need to be aware of what is happening. Currently here are 2 very important tools, for checking the cluster status.
clustat ( which is installed by the rgmanager rpm ) The clustat command will give you a quick status about all the nodes in the cluster and of the services running.
cman_tool ( which is installed by the cman rpm ) The cman_tool command is for managing a node in the cluster ( leaving, joining, votes, and status ).
Here is an example of clustat...
clustat
Cluster Status for MyCluster @ Sat Aug 29 18:37:27 2009
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
gfs1 1 Online, Local, rgmanager
gfs2 2 Online, rgmanager
gfs3 3 Online, rgmanager
/dev/disk/by-path/pci-0000:00:11.0-scsi-0:0:1:0-part1 0 Online, Quorum Disk
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:CIM gfs1 started
service:Pirahna (gfs2) disabled
|
|
Last Updated on Tuesday, 01 December 2009 18:08 |
|
Read more...
|
|
|
Clustering HowTo's -
RedHat Clustering
|
|
Written by Allen Sanabria
|
|
Sunday, 23 August 2009 20:25 |
|
Today's tutorial will be on the infamous Quorum disk. When I first setup my GFS2 shared Cluster of 3 nodes, I was quite impressed with the fact that 3 nodes were sharing the same file system. Now that everything was up and running, I wanted to see what would happen if I brought down, 2 out of the 3 nodes in the cluster. I turned off 1st node and all was well, I was still able to access my GFS2 mount on the other 2 nodes. Then I decided to reboot the 2nd node, and guess what happened???? QUORUM DISSOLVED!!! Now on my final node the GFS2 file system was still mounted but I could not touch a file or run a ls on the mount... It just hung there!!
Well I knew this was not going to be acceptable.... Since if I still have 1 node available, the node should still be able to use the GFS2 mount. So I did some research about this quorum disk and what it can do for me. Let me tell you, this was exactly what I was looking for. 1st let me start out my explaining what a quorum is ( relating to clustering ). A quorum is the minimal number of votes that is needed in a cluster, usually the majority. So if you have 3 nodes in a cluster, that means you have a total of 3 votes in the cluster and you will need a minimum of 2 votes to remain in a quorate state. Which means you can lose 1 node in the cluster and the other nodes are still functional. But if you lose 2 nodes, your quorum will be dissolved. Which means even though your GFS2 file system is still mounted on your final node, it will not be accessible to you.
The quorum disk will help this particular situation... You ask how???? Well here it is... Qdisk needs at a minimum of a 10MB disk partition shared across the cluster.
Qdiskd runs on each node in the cluster, periodically checking its
own health and then placing its state information into its assigned
portion of the shared disk. On each node qdiskd then looks at the state of
the other nodes in the cluster as posted in their area of the qdisk
partition. When all the nodes that are running qdiskd are in a healthy state, the quorum of the cluster is increased by the value of the shared Quorum Disk.
The Value of the Quorum Disk should be n-1 ( Number of nodes - 1 ). In this case the Value should be 2 ( 3 -1 ).
|
|
Last Updated on Saturday, 29 August 2009 07:50 |
|
Read more...
|
|
Clustering HowTo's -
RedHat Clustering
|
|
Written by Allen Sanabria
|
|
Friday, 14 August 2009 12:05 |
|
In the last HowTo, I showed you how to setup GFS2 file system with Red Hat Clustering. I will now show you how to optimize the performance of your GFS2 mounts. The gfs_controld daemon manages the mounting, unmounting, and the recovery of the GFS2 mounts. gfs_controld also manages the posix lock.
By default the plock_rate_limit option is set to 100. This will allow a maximum of 100 locks per second, which will decrease your GFS2 performance. See below...
<dlm plock_ownership="0" plock_rate_limit="100"/>
<gfs_controld plock_rate_limit="100"/>
You can test the performance of you cluster by downloading the program ping_pong.c. This program was very helpful to me in debugging the poor performance in my GFS2 cluster.
The instructions on how to compile the program and run it is on the site http://wiki.samba.org/index.php/Ping_pong.When I initially ran ping_pong, I only got a max of 97 plocks per second. After removing the rate limit I was able to get about 3000 Plocks per second.
|
|
Last Updated on Wednesday, 26 August 2009 18:35 |
|
Read more...
|
|
|
Clustering HowTo's -
RedHat Clustering
|
|
Written by Allen Sanabria
|
|
Friday, 14 August 2009 11:48 |
|
In my last project at work, I had to replace NFS with GFS2 and Clustering. So in this tutorial I will show you how to create a Red Hat or CentOS cluster with GFS2. I will also show you how to optimize GFS2 performance in the next HowTo, because you will quickly notice some loss of performance until you do a little optimization first.I will 1st show you how do build a Cluster with GFS2 on the Command Line and in the next tutorial I will show you how to do the same thing using Conga.
In this tutorial I am using 3 CentOS Virtual Machines running CentOS 5.3 in VMware ESX 3.5. For the GFS2 File System I am using a vmdk built with the thick option, that is shared among all the Virtual Machines. You also can use iscsi or fiber... This option is up to you.
Always make sure your iptables (If you know the port's and protocols for clustering, then add it to iptables ) and selinux is OFF. If not you will run into issues.
The 3 machines I am using are called
- gfs1 == 192.168.101.100
- gfs2 == 192.168.101.101
- gfs3 == 192.168.101.103
|
|
Last Updated on Monday, 15 March 2010 11:45 |
|
Read more...
|
|
|
|
|
|
Aug.04
