HowTo Send Splunk Alerts To Zenoss, And make them Look Like Splunk

We needed to integrate the Splunk Alerts into Zenoss, because even though Splunk can indeed send out alerts. Splunk does not have any clue about what an “Escalation Process” is. With Zenoss you can create an “Escalation Process”.

I have 2 ways to send events to Zenoss from Splunk..

1. Write a Script that uses the snmptrap command.
2. Write a Script that uses the Zenoss zensendevent command.

I decided to go with the Zenoss zensendevent command ( Which is a python script with no external dependencies, which can be copied from the Zenoss Server at $ZENHOME/bin/zensendevent ).

Now it’s time to get the ball rolling..

1. On the Splunk Server I copied the zensendevent script from the Zenoss Server to Splunk on /opt/splunk/bin/scripts/zensendevent.
2. I then created a shell script called Splunk2Zenoss.sh. ( This script takes the Saved Splunk Search and passes it over to Zenoss )This script will also be located in /opt/splunk/bin/scripts/Splunk2Zenoss.sh
3. You will then need to modify the options in the script. (For instance the severity of the alert, the zenoss server, the event mapping, event key, login and passwd )
4. I then created the saved search in Splunk and make sure to check the Trigger Shell Script option. ( Make sure to put the script name in here )

I did my best to mimic the Splunk Saved Search Alert that you get by email through Splunk, into Zenoss. As of right now any searches that you have saved, can now be sent to Zenoss by using zensendevent that comes with Zenoss.

There are 2 caveats to this though….

1. You will need to add to the following ( | fields – _raw ) to all your saved searches, with out the ().
2. modify zensendevent on line number 170,
   from..
   field, value = line.split(‘=’)
   to..
   field, value = line.split(‘=’,1)
   The reason for the change above, is because Splunk will send “=” signs in the message output, and zensendevent is splitting by “=”. So to get rid if this issue, you set the maxsplit to 1.

 

Here are ScreenShot Examples of a Splunk Email and a Zenoss Email

Splunk2Zenoss.sh == Download

SplunkTransfor.py == Download