Show me examples of Enriched Events

A number of you have asked to see what an Enriched Event looks like. The following three examples illustrate enriched events received from Nagios, Zabbix, and Zenoss.

[NAGIOS]

Notification Type: PROBLEM
HOST: dbserver1
State: DOWN
Address: 70.86.17.12
Info: CRITICAL – Host Unreachable (70.86.17.12)
Date/Time: Sat Jan 16 11:09:23 JST 2013

ESCALATION:
This is a CRITICAL alert which needs immediate escalation to the site DEVOPS team. Use Pagerduty DEVOPS service.

REMEDIATION:
1) Attempt to ping the host from the nagios server
2) If ping is successful, attempt to ssh to the host (ops1@70.86.17.12)
3) if ssh is not successful then initiate DB_HOST_DOWN recipe sequence

[ZABBIX]

Name:SSH server is down on Morbo
ID:13053
Status:PROBLEM
Hostname:Morbo
IP:172.16.0.199
Value:1
Event_id:8733
Severity:Average

ESCALATION:
This is a CRITICAL alert which needs immediate escalation to the on call SYSTEMS team. Use Pagerduty SYSTEMS service.

REMEDIATION:
1) Attempt to ssh to the site
2) If ssh access is available then review the system logs to see if there was a restart event
3) If ssh access is NOT available then initiate the steps described in the SERVER_SSH_DOWN recipe

[ZENOSS]

Device: www.eventenrichment.org (http://www.eventenrichment.org)
IP Address: 173.236.226.200
Component: HealthMonitor-public_site
Severity: Critical
Time: 2013/06/03 15:18:35.000
Message: CRITICAL: Unable to open http://www.eventenrichment.org/healthchk (http://www.eventenrichment.org/healthchk) in 10 seconds.

ESCALATION:
This is a CRITICAL alert which needs immediate escalation to the site DEVOPS team. Use Pagerduty DEVOPS service.

REMEDIATION:
1) Attempt to manually open the site
2) If site comes up fine, then attempt to open up the URL from a remote site.
3) If that is also successful then resolve the event
4) If not, then initiate the steps described in the SITE_DOWN recipe

These three simple examples are just the tip of the iceberg in terms of what is possible with Event Enrichment. Enrichments can include links to specific Runbook entries, specific steps to follow to fix the problem, and in advanced usage, actions.

For specific instruction on how to utilize Event Enrichment in your operations, check out the Beginner’s Guide:

Questions? Ask in the comments.

What's your opinion?