Zenoss | LinuxDynasty

HowTo check redis availability and get stats using Redis-py, Zenoss, and Python

Mar 202012

My team was assigned to create a redis slave status check to be ran under Zenoss. So while they are creating that check, I decided to google for redis checks written in python that work under Nagios and or Zenoss and none of the checks i found did exactly what I wanted.

So I decided to write my own check, that will grab every piece of data that the redis info() command was able to retrieve. Since this info is all in a python dictionary, I was able to get all the stats that were labeled as type int or as type float, which made my job that much easier.

So 1st I had to install 2 Redis instances on my local Ubuntu server at home. Now that i completed that, I had to make sure the slave was syncing off the master server. I used this link “http://redis.io/topics/replication” to help me configure redis replication.

Now that all that stuff is out of the way, I wrote an easy_peasy python script to connect to redis and grab all of the performance stats. As well as verify if the instance is a master or a slave instance. If it is a slave instance, than it also verifies if it is syncing to the master or not.

The script is using Redis-py that was installed using PIP.

Here is the script I wrote

Download

Continue reading “HowTo check redis availability and get stats using Redis-py, Zenoss, and Python” »

HowTo add Aggregate Data Graphs from existing datapoints in Zenoss

Python, Zenoss No Responses »

Feb 022011

Recently I had to aggregate the amount of 200, 400, 404, 500, 503, and 504 HTTP 1.1 Codes from all of our Nginx and Ruby On Rails systems. So I decided to write a quick plugin for Zenoss that all you need to do is pass the devices or Device Class you want to aggregate the data from and the DataPoint name.

Also this makes it simple to use the Zenoss Thresholds and create Aggregate Reports based on the datapoints you use with out having to do some fancy python coding… Enjoy

Example..

python AllenZenossAggregate.py -h
Usage: AllenZenossAggregate.py -d "nginx-1" -d "nginx-2" -p "nginx_codes_count200"
OK Aggregate for devices  nginx-1 nginx-2 is 34|aggregate=34
     AllenZenossAggregate.py -o "/Server/Linux/Nginx" -p "nginx_codes_count200"
OK Aggregate for /Server/Linux/Nginx class is 22|aggregate=22
 Options:
  -h, --help            show this help message and exit
  -d DEVICE, --device=DEVICE
                        The device you want to grab the datapoints from.
  -o ORGANIZER, --organizer=ORGANIZER
                        The Class you want to get your list of devices from.
  -p DPOINTS, --dpoints=DPOINTS
                        Name of DataPoint nginx_codes_count200

Download

Monitoring RabbitMQ with Zenoss

Zenoss No Responses »

Sep 102010

RabbitMQ was recently deployed in the company I currently work for. At the last minute ( as always ) they came to me and ask me to please add RabbitMQ monitoring to Zenoss. They said here is the url and now please monitor for a few stats ( using the RabbitMQ Status Plugin ). So I said to myself, I could easily just write a quick shell script to get the 3 stats that they needed and add them into Zenoss. After thinking about it…. In the near future they might ask for more than those 3 stats. So I decided to write a quick python script ( Zenoss Compatible ) to get all the stats from that status page and input them into Zenoss…

Prerequisites

Zenoss 2.5 and above ( I have not tested on 3.+ or <2 .4 )
lxml Python module ”easy_install lxml” as the zenoss user
“Zenoss_Template_Manager.py” Optional, Only needed if you do not want to add all the datapoints manually… I DON’T!!!!
RabbitMQ Installed
RabbitMQ Status Plugin from http://www.lshift.net/blog/2009/11/30/introducing-rabbitmq-status-plugin
“check_rabbitmq.py“

Zenoss Template Manager = Download

CheckRabbitMQ.py = Download

Once you have all the above, we are ready to go..

copy both Python Scripts above into the /opt/zenoss/libexec/ folder ( If you are using RedHat/CentOS ) and make sure they are executable.

As the Zenoss user run the script ….
/opt/zenoss/libexec/check_rabbitmq.py -u ‘http://rabbitmq-server:55672′ -a ‘mon-user mon-passwd’ |sed -re “s/^OK|/ /g” |sed -re “s/([A-Za-z0-9_.]+*)?=[0-9]+/-p “1,G”/g” |xargs /opt/zenoss/libexec/Zenoss_Template_Manager.py -o “/Devices/Server/Linux/RabbitMQ” -c ‘/opt/zenoss/libexec/check_rabbitmq.py -u “http://rabbitmq-server:55672″ -a “mon-user mon-passwd”‘ –template=RabbitMQ –dsource=RabbitMQStats -V $1

If you need to know how to use the Zenoss_Template_Manager.y script, check here http://www.linuxdynasty.org/howto-add-multiple-datapoints-to-zenoss-using-the-zenoss-api.html

So let me explain what the sed statements above are doing…

“sed -re “s/^OK|/ /g”" This sed statement is removing the OK| from the beginning of the line
“sed -re “s/([A-Za-z0-9_.]+*)?=[0-9]+/-p “1,G”/g”
This 1st part of the sed statement is matching any letter,number,underscore, and period, any number of times until it reaches the equal “=” ”[A-Za-z0-9_.]+*)?=”
The 2nd part of this statement is going to match the “=” and any number of integers after it. “[0-9]+”
Now we need to make the substitution…. So we are going to substitute, every match with a -p, then a space and then the 1st group match in escaped quotes, then a comma and G for GAUGE.

The 2 sed statements above will do that for every match it finds. If you were to add each datapoint by hand, it would look like this….
/opt/zenoss/libexec/Zenoss_Template_Manager.py -o “/Devices/Server/Linux//RabbitMQ” -c ‘/opt/zenoss/libexec/check_rabbitmq.py -u “http://rabbitmq-server:55672″ -a “mon-user mon-passwd”‘ –template=RabbitMQ –dsource=RabbitMQStats -V -p “queue.conversion.event.tracking_msg_unack,G” -p “connections,G” -p “erlang_processes_used,G” -p “erlang_processes_avail,G” -p “file_descriptors_used,G” -p “file_descriptors_avai,G” -p “binary_memory,G” -p “memory_used,G” -p “memory_avail,G”

I hope the above scripts will save someone time and frustration……

connections=26 erlang_processes_used=252 erlang_processes_avail=1048576 file_descriptors_used=1 file_descriptors_avail=1024 pid=3167 binary_memory=0 ets_memory=0 memory_used=27 memory_avail=99

HowTo Send Splunk Alerts To Zenoss, And make them Look Like Splunk

Zenoss 2 Responses »

Aug 032010

We needed to integrate the Splunk Alerts into Zenoss, because even though Splunk can indeed send out alerts. Splunk does not have any clue about what an “Escalation Process” is. With Zenoss you can create an “Escalation Process”.

I have 2 ways to send events to Zenoss from Splunk..

Write a Script that uses the snmptrap command.
Write a Script that uses the Zenoss zensendevent command.

I decided to go with the Zenoss zensendevent command ( Which is a python script with no external dependencies, which can be copied from the Zenoss Server at $ZENHOME/bin/zensendevent ).

Now it’s time to get the ball rolling..

On the Splunk Server I copied the zensendevent script from the Zenoss Server to Splunk on /opt/splunk/bin/scripts/zensendevent.
I then created a shell script called Splunk2Zenoss.sh. ( This script takes the Saved Splunk Search and passes it over to Zenoss )This script will also be located in /opt/splunk/bin/scripts/Splunk2Zenoss.sh
You will then need to modify the options in the script. (For instance the severity of the alert, the zenoss server, the event mapping, event key, login and passwd )
I then created the saved search in Splunk and make sure to check the Trigger Shell Script option. ( Make sure to put the script name in here )

Continue reading “HowTo Send Splunk Alerts To Zenoss, And make them Look Like Splunk” »

HowTo get 3par disk IO stats into Zenoss

Zenoss No Responses »

Mar 292010

I’ve come to realize, that CIM is the new SNMP, but on steroids. Most new SAN, NAS, Network, and Operating Systems now support CIM/WBEM. To me it is easier to gather statistics and information through CIM, then it is through SNMP. In this article I am going to give you a script that will allow you to query the 3par for Disk IO stats. You will be able to grab Disk IO stats on a per Volume, per Port, or per Disk basis.You will also be able to search for a Volume, Port, or Disk, instead of just dumping all the Volumes, Ports, or Disk.

Before you download this script, you will need to download pywbem from sourceforge and install it. .

get3ParIOstats.py == Download

All the data that you get from the script, must be saved as a COUNTER and not a GAUGE.

Here is an example of searching for statistics by DISK for DISK 2:6:1..

python get3parIOstats.py -u "http://3par" -a 'login passwd' -s '2:6:1' --diskOK|2:6:1_ReadIOs=67236384 2:6:1_WriteIOs=28457131 2:6:1_TotalIOs=95693515

Continue reading “HowTo get 3par disk IO stats into Zenoss” »

How to find Alerting Rules using the Zenoss API

Zenoss No Responses »

Mar 192010

Good Afternoon my fellow Zenoss users. here I bring you another Python tool to use with Zenoss. This tool will allow you to list all of your alerting rules, either by Group, User, or Query. If you are like me and have over 100 Alerting Rules. Managing these rules are more then a pain in the butt. Especially when you create a new rule and you know you created it correctly, but it just does not work… Well you might have another conflicting rule. Well what do you do in this situation?? You can go through each rule manually and sooner or later you will find it.

Or you can use my tool to find it for you. You might ask… How will yout tool find the conflicting rule for me??? Well to be honest it will not find the rule for you. What it can do is search all the rules for a certain query. Which can return one or more alerting rules if they are similiar in nature. or you can have the tool, list all of your Alerting rules. Or you can have the script list all of your alerting rules by group or user. The toll will report back to you The Group/User, The Alerting Rules attached to that Group/user, and the Schedules Attached to that Alerting Rules.

Where this tool really helped me, was when the DST changed happened. None of my schedules were working anymore, so the work around is to move all the schedules ahead of time. Using this script I found all my schedules very quickly, with out haveing to go through each of my alerting rules one by one.

Download

Continue reading “How to find Alerting Rules using the Zenoss API” »

HowTo add Multiple DataPoints to Zenoss using the Zenoss API

Zenoss 4 Responses »

Mar 052010

I’ve seen a few people in the Zenoss community forums asking for an easier way to add multiple Datapoints into Zenoss, instead of adding 1 by 1. Well, I’ve decided to try and build such a tool today. Now even though the tool is not complete (IN MY EYES). It is already has been helpful to me! As of right now the script can List Templates in a Organizer ( aka /Devices/Server/Linux ) or in a Device ( zenoss1.linuxdynasty ).

It can also create a Template, with a Command DataSource, with Multiple DataPoints. This means you can now automate the creation of Templates and DataSources and DataPoints. So if you are using a tool like Puppet, Cfengine, Bcfg2, or Chef, this process can be automated. I will be adding more features in the next few weeks.

Any question or help about this script, please post them here http://www.linuxdynasty.org/forums/Scripting/scripting/ZenossTemplateManager
UPDATES!!!!!!

Update 1.0.6, Fixed an issue where if youdo not pass the -G option for graph it will fail.
Update 1.0.5, Now have the ability to add multiple graphs with multiple datapoints
Update 1.0.4, Now have the ability to create thresholds, bind templates to class or device, set severities on thresholds and
datasources, attach datapoints to thresholds.
Update 1.0.3, I broke down the templateManager function into 3 smaller functions. You also can now list templates and datasources for either device or organizer, and list datapoints for a device. I also added more verbosity with the -V option.
Update 1.0.2, You now do not have to pass a command with the script. So i f you already have a command in the DataSource, it wll not get overwritten.. Thank you Eangel, for telling me about this issue..

Upcoming Features such as….

Deleteing Templates, DataSources, and DataPoints
Copying Templates
Adding Graphs

The tool is called Zenoss_Template_Manager.py. You can download it here..
Download

Continue reading “HowTo add Multiple DataPoints to Zenoss using the Zenoss API” »

HowTo backup all of your Zenoss Templates the easy way.

Python, Zenoss 2 Responses »

Feb 122010

Recently I had to prepare for a Zenoss upgrade. During my prep work I had to create a zenpack of all of our Templates. For those of you who use Zenoss, you know how many templates you can start to accumulate in a short amount of time. You can have Templates attached to single Devices, to SubClasses, and to Classes. Now if you have a few devices this is not a big deal. But if you have a couple hundred to a couple of thousand devices, this could be a real hassle.

Now you can take a ton of your time and review Class by Class and Device by Device until you finally finish. You will eventually get it all in a Zenpack……. Well lucky for you guys I created a Python Script that runs as the Zenoss user and create a Zenpack for you. All you have to do is pick a name for the ZenPack and optionally the Device Class you want to scan. The script will scan the Device Class that you specified ( or by default scan the entire /Device Class and its Sub Classes). It will then create the ZenPack with all the locally attached Device Templates. I am thinking of also adding the Events class as part of the next release of this script.

Update 1.0.1, I added the –unique option. If you decide to use this option, This Zenpack will only add, The Device Templates that are not already part of an existing ZenPack. I also added the –verbose option, so that you can see which Device Templates are being added or being dismissed.

You can download the script here
Download

Continue reading “HowTo backup all of your Zenoss Templates the easy way.” »

Syncing up Zenoss with the Citrix Netscaler using Python

Zenoss No Responses »

Jun 172008

The script below I created to sync up Zenoss with the Citrix Netscalers. What this will do essentially is make sure that all of our devices in Zenoss are in the correct Systems in Zenoss. So for instance lets say we have a cluster called Foo-Cluster and we have 100 devices that are in that cluster as per the netscaler. This script will move those 100 devices into that System aka {Cluster} in Zenoss.

The reason this is so important is that when you update the Citrix Netscaler you will also have to update Zenoss, but if you run this script in cron then you will not have to update Zenoss at all. This script was written in Python This script was tested with Zenoss 2.0.2, Also some modifications may have to be made to fit your environment.

Please post any questions to this script here http://linuxdynasty.org/phpBB3/viewtopic.php?f=5&t=3
You can download the script here … zenoss_netscaler_snmp.py

Continue reading “Syncing up Zenoss with the Citrix Netscaler using Python” »

Discovered Zenoss Devices

Zenoss No Responses »

Apr 262008

For those Zenoss users out there, this script will send a emailof devices that are in the discovered class.

So once you receive this email you will know that you need to move those devices into there appropriate classes.


#!/bin/env python
#Copyright (C) 2008  Allen Sanabria

#This program is free software; you can redistribute it and/or modify
#it under the terms of the GNU General Public License as published by
#the Free Software Foundation; either version 2 of the License, or
#(at your option) any later version.

#This program is distributed in the hope that it will be useful,
#but WITHOUT ANY WARRANTY; without even the implied warranty of
#MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#GNU General Public License for more details.

#You should have received a copy of the GNU General Public License along
#with this program; if not, write to the Free Software Foundation, Inc.,
#51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
##############################################################
#Created by Allen Sanabria aka LinuxDynasty aka PrNino69
#This script is to check how many devices are in the
#Discovered Class
#Started Nov 28th
#Completed, Nov 28th
##############################################################

import os, sys
from re import sub
from string import split
from string import join
from urllib import urlopen
from smtplib import SMTP
from time import sleep


user = "zenoss"
passwd = 'zenoss'
util = '@zenoss'
base = "http://%s:%s%s:8080" % (user,passwd,util)
discovered_url = urlopen(base+'/zport/dmd/Devices/Discovered/getSubDevices').read()
discovered_sub = sub("<Device at /zport/dmd/Devices/Discovered/devices/|>|^[|]$|,", "", discovered_url)
discovered_list = list(split(discovered_sub))


message = """nThe boxes below were discovered in the last run of zendisc.nThey are all located under /Devices/Discovered Class.n
                     Please move Devices to appropriate Device class, if one does not exist please create one.n
                     This script runs on the zenoss (cc17-22) server."""
devices = sub(",|[|]", "n", str(discovered_list))
BODY = join((message, devices),"n")
print BODY
FROM = "zenoss@linuxdynasty.org"
TO = "sa@linuxdynasty.org"
SUBJECT = "Devices That Were Discovered During The Network Scan!"
body = join(("From: %s" % FROM, "To: %s" % TO, "Subject: %s" % SUBJECT, "", BODY), "n")
server = SMTP('localhost')
server.set_debuglevel(1)
server.sendmail(FROM, [TO], body)
sleep(10)
server.quit()