A couple of weeks ago our partner Rok Ponikvar from S&T contacted me about an issue one of his customers faced. His customer complained that Enterprise Alert is not alerting on current issues and even if he creates a test ticket in his OBM system no alert goes out. After a little back and forth we concluded that Enterprise Alert is still processing historic data from an Event Storm in OBM earlier that day. The situation resolved itself rather quickly so the impact on the customer was not too severe.
What is the technical background?
Enterprise Alert will, for the protection of its own and of the 3rd party systems operations, only poll a certain amount of Events in a given interval for example 500 events or tickets every 30 seconds. If there are more than 500 events or tickets created, let’s say 750, within these 30 seconds Enterprise Alert will only poll the 500 oldest will be polled and the rest, in this theoretical case 250, will remain for the next cycle. As you can imagine this can cause quite a backlog to build up over time.
Enterprise Alert will work through that backlog and catch up to the current data as soon as the 3rd party system generates fewer events or tickets than being polled from Enterprise Alert.
What was the solution?
In further discussion, Rok asked if there was a way to monitor the system for a delay. I pointed him to the ModuleServiceProperties table as most of our modern smart connectors store polling information there in order to have a starting timestamp upon service restart and to provide sync data for High Availability setups. Rok then went ahead and wrote a query to monitor his system for any desynchronization between Enterprise Alert and his OBM system. He kindly shared this query with me so everyone can benefit. Shoutout to Rok Ponikvar for this. Thanks a lot. 😊
DECLARE @lastevent datetime DECLARE @obmsynctime datetime SELECT @obmsynctime= [Value] FROM [<your_database_name>].[dbo].[ModuleServiceProperties] WHERE [Name] = 'LastTimestamp' and [ModuleServiceID]= '<your_Service_ID>' SELECT TOP (1) @lastevent = [Timestamp] FROM [<your_database_name>].[dbo].[EAEvents] WHERE [ModuleServiceID]= '<your_Service_ID>' order by [Timestamp] desc SELECT ABS(datediff(MINUTE, @lastevent , @obmsynctime )) as SyncDiff
Please note you will have to replace <your_database_name> with your database name, default is either MMEA or EnterpriseAlert, and <your_Service_ID> with the ID of the service you want to monitor. You can get the ID from the ModuleServiceProperties as well as there will be one entry for the service type and one for the timestamp. If you have several connectors of the same type things get a little more complicated but that’s what firstname.lastname@example.org is for. 😉