When an on-call person receives an alert it is essential that he or she understands it – immediately.
The Problem:
The information available from the backend systems (e.g. monitoring tools, facility management, etc.) is often quite cryptic and not self-explanatory.
Sometimes alert messages are hard to read and understand:
- Alert on XC3o84-CCP-DEC07, Error code: 1254, Timestamp: 1421920446797, Location: 52.4027477,13.0641529
- Ticket TCK42564422 assigned to analyst cz1447.
- NtpClient was unable to …. NtpClient will try again in 3473457 minutes ….
- STOP: 0x00000F4 (0x0000000000000003, 0xFFFFFA8008BB7B30, 0xFFFFFA80008BB7E10, 0xFFFFF800031E88B0)
- An error occurred while creating an error report.
This is the case when the alert message is received by SMS or email and it gets even worse if the message is automatically spoken to the user in a voice call.
Such information is good for machines but not for humans.
- What does error code 1254 mean?
- Is severity 3 high or low?
- What server was behind IP 192.168.2.105 again?
- Who is the user “cz1447”? How can I call him or her now?
The Solution:
So, how can you translate “machine-speak” into a more “human-speak”? The solution offered by Enterprise Alert is Alert Enrichment. You can replace or add additional information for the alert notifications to be sent out to the users.
One option is to use scripting. Enterprise Alert offers the Scripting Host which is an add-on for enhancing alert workflows by using own scripts (Java Script or Visual Basic Script). All you need to do is to create a normal policy and then under the “Destination” tab chose a script (“Through Custom Workflows” under “Mode Options”).
You can create or upload your script under “System” –> “Scripting Host”.
You can then handle the alerting workflow in the script.
A very simple script for forwarding an incoming event to a script and then generating an alert might look like follows:
| function OnNewEvent (eventObject) { var strExternalId = “”; var objTicket; var strBuilding = eventObject.getProperty(“Building”); var strServer = eventObject.getProperty(“Server”); var strErrorCode = eventObject.getProperty(“ErrorCode”); var strSeverity = eventObject.getProperty(“Severity”); // Send the alert message // Priority: 0: low priority, 1: major priority, 2: critical priority objTicket = EAScriptHost.TicketCreate(“rczachara”, “FF”, 2, “Alert in building: ” + strBuilding + “, on server: ” + strServer + “, severity: ” + strSeverity + “, error code: ” + strErrorCode); if (objTicket == null) { EAScriptHost.LogError(“Ticket not created”); } objTicket.SetProperty(“externalTicketId”, strExternalId); // Set the notification type to User alert // 0 = Automatic, 1: User alert, 2: Team Broadcast, 3: Team Broadcast to a schedule, 4: Team Escalation, 5: Team Escalation to a schedule, 6: Subscription Feed objTicket.SetProperty(“notificationType”, “1”); // Now send the ticket to Enterprise Alert for processing if (objTicket.Send() == false) { EAScriptHost.LogError(“Ticket not dispatched to Enterprise Alert”); } } |
The output of the alert text looks like follows:
Alert in building: Berlin Office, on server: 192.168.2.105, severity: 2, error code: 1254
So, how can you make this more readable? For example you can replace the severity number by an explanatory word:
// Replace severity if (strSeverity == “1”) strSeverity = “low”; else if (strSeverity == “2”) strSeverity = “medium”; else if (strSeverity == “3”) strSeverity = “high”; |
Or, you can add some command line information like from “nslookup”:
| // Replace the IP address by the server name var obShell = new ActiveXObject(“WScript.shell”); var e = obShell.Exec(“%comspec% /c nslookup ” + + ” 2>&1 “) while(!e.StdOut.AtEndofStream) { var line = e.StdOut.ReadLine(); strServer = line.replace(“Server: “, “”);break; } |
Or, you can even add extra information information from a database:
| // Add site information var strResult; // Get database connection var strDbConnectionString = “Driver=SQL Server Native Client 11.0;Server=sqlserver;Trusted_Connection=No;UID=SA;PWD=Derdack!;Database=EnterpriseAlert”; var oConn = DbGetConnection(strDbConnectionString); if (oConn) { if (DbCheckConnection(oConn)) {// Make the SQL query var rsUsers = DbGetRS(oConn, “SELECT Description FROM [MMEA].[dbo].[DemoSites] WHERE Site LIKE \’%” + strBuilding + “\’;”); if (!rsUsers.EOF) { strResult = rsUsers.Fields.Item(“Description”).Value;// Add the additional information strBuilding = strBuilding + ” (” + strResult + “)”; } // Close the database connection oConn.Close();} } |
At the end the alert text might look much better:
Alert in building: Berlin Office (The key is at the reception.), on server: SQLSERVER2, severity: medium, error code: 1254 (Low disk space.)
More examples where alert enrichment is helpful:
• Send one-time passwords
• Add contact information
• Add links to tickets or KB
• Reformat error codes
• Add location information
• Format notification depending on media
• Add site information
• ….
Do do not let the user guess what an alert message really means and use Alert Enrichment …. 😉
Please also do not miss the related video under https://vimeo.com/118599348.