Cluster Monitoring Model Description

AlarmCondition Model

ConditionDef

Types and attributes:

The ConditionDef will have each of these attributes but will be populated based on the type of the Condition

Type: Facet

Test: Check if the Facet is running

Type: HTTP

Test: Check if the HTTP endpoint is UP

Type: WS

Test: Check if the websocket endpoint is UP

Type: Sequence

Test: Check if the sequence is executing

Type: Log

Test: Check if the sequence is executing

RecoveryActionModel

Type: ExecuteQuery

Test: Check if the sequence is executing

Type: Script

Test: Check if the sequence is executing

Alert Model

Implementation

Daemon Configuration

The following steps will be performed on the deamon:

1. Rules are added by the use through the API or csv file upload
2. A job is executed every minute that does the following:
   1. Picks up a AlarmCondition
   2. Executes the condition
   3. On Success
      1. Updates the LastRun with current time and LastStatus to 'Success'
   4. On Failure
      1. Updates the LastRun with current time and LastStatus to 'Failure'
      2. Generates an alert with Condition, ClusterId, InstanceName, and Error Reason
      3. If Recovery Action is present try and execute the action recovery could be
         1. Try and restart failing component
         2. Try and restart the A-Stack engine on the failing node
3. Find new alerts (Newer than the last run) Sends email notification for the alert

Alarm State Transition Diagram

Provisioning Alarms Using CSV File

Command Line API

Load Alarm Conditions

This Command loads the AlarmConditions into the monitoring system.

• Command: tql -monitoringconfig -load Path=<Path of the File>/AlarmConditions.csv

Delete Alarm Conditions

This command lets you delete one or all AlarmConditions from the system based on the inputs provided.

• Command: tql -monitoringconfig -delete ClusterId=Cluster-1,Instance=Instance-1,Name=Test_Endpoint_Facet

Deletes AlarmCondtion called Test_Endpoint_Facet from instance Instance-1 on cluster Cluster-1

• Command: tql -monitoringconfig -delete ClusterId=Cluster-1,Instance=Instance-1

Deletes all AlarmCondtions from instance Instance-1 on cluster Cluster-1

Get Alarm Conditions

This command gets AlarmCondtions from the system based on the inputs provided. And writes it to the file provided in the Path variable

• Command: tql -monitoringconfig -get ClusterId=Cluster-1,Instance=Instance-1,Name=Test_Endpoint_Facet,Path=<File Path>\Result.json

Gets AlarmCondtion called Test_Endpoint_Facet from instance Instance-1 on cluster Cluster-1

• Command: tql -monitoringconfig -get ClusterId=Cluster-1,Instance=Instance-1,Path=<File Path>\Result.json

Gets all AlarmCondtions from instance Instance-1 on cluster Cluster-1

Update Alarm Conditions

Updates an AlarmCondition values. First run the get Command to get the AlarmCondition, make the Necessary changes and provide this file path in the "Path" variable.

• Command: tql -monitoringconfig -update ClusterId=Cluster-1,Path=<File Path>\Result.json

Update Email Configuration

Updates the Email configuration. ClusterId is optional, if not specified updates will happen on all cluster.

• Command: tql -monitoringconfig -config Path=<File Path>\EmailConfig.json,Type=setemail,ClusterId=Cluster-1

Updates configuration on cluster Cluster-1

• Command: tql -monitoringconfig -config Path=<File Path>\EmailConfig.json,Type=setemail,ClusterId=Cluster-1

Updates configuration on all clusters

Sample Configuration:

{  
   "NotificationConfig":{
	  "Host":"HOST_NAME",
      "Port":"PORT_NUMBER",
      "Username":"UNAME",
      "Password":"PWD",
      "From":"from@domain.com",
      "To":"recepient@domain.com,recepient2@domain1.com",
      "Subject":"Alert Generated Notifications"
   }
}

Get Email Configuration

Get the Email configuration and prints it on the screen. ClusterId is mandatory.

• Command: tql -monitoringconfig -config Type=getemail,ClusterId=Cluster-1

Update Schedule Configuration

Updates the schedule configuration. ClusterId is optional, if not specified updates will happen on all cluster.

• Command: tql -monitoringconfig -config Path=E:\Atomiton\Builds\TQLEngineConfiguratorNode1\resources\atomiton\configurator\spaces\SysConfig.json,Type=setschedule,ClusterId=Cluster-1

{  
   "SysConfig":[
		{"EMAIL_FREQ":"1min"},
		{"FACET_FREQ":"1min"},
		{"HTTP_FREQ":"1min"},
		{"SEQ_FREQ":"1min"},
		{"LOG_FREQ":"1min"},
		{"INFO_FREQ":"1min"},
		{"WS_FREQ":"1min"},
		{"ALERT_PURGE_FREQ":"1min"},
		{"ALERT_PURGE_LIMIT_DAYS":"25"}
   ]
}

• FACET_FREQ: Frequency with which Facet type AlarmConditions are executed.
• HTTP_FREQ: Frequency with which HTTP type AlarmConditions are executed.
• SEQ_FREQ: Frequency with which Sequence type AlarmConditions are executed.
• LOG_FREQ: Frequency with which Log type AlarmConditions are executed.
• INFO_FREQ: Frequency with which Info type AlarmConditions are executed.
• WS_FREQ: Frequency with which WS type AlarmConditions are executed.
• ALERT_PURGE_FREQ: Frequency with which stale alerts are purged.
• ALERT_PURGE_LIMIT_DAYS: How old should an alert be before it is considered stale.
• EMAIL_FREQ: Frequency with which Alert emails are sent.

Get Schedule Configuration

Get the schedule configuration and prints it on the screen. ClusterId is mandatory.

• Command: tql -monitoringconfig -config Type=getschedule,ClusterId=Cluster-1

Stop Monitoring

Pause a monitoring service. ClusterId is mandatory.

Command: tql -monitoringconfig -stopmonitoring ClusterId=Cluster-1

Start Moniting

Resume a monitoring service. ClusterId is mandatory.

Command: tql -monitoringconfig -startmonitoring ClusterId=Cluster-1

Update purge configuration for management dashboard'

Update Alert purging configuration on the Dashboard.

Command: tql -monitoringconfig -alertpurgedashboard Purgefreq=1min,Purgelimit=20

• Purgefreq: Frequency with which stale alerts are purged.

• Purgelimit: How old should an alert be before it is considered stale

Management Dashboard

A scheduled job will run on the dashboard back-end that will periodically (Every minute) pull the AlarmConditions, Alerts and Notifications sent and store it onto itself for displaying on the UI.

FAQs

Common

Q: Monitoring Page you had shared earlier was running on port 9000. Is this port configurable?

A: No, we cannot change port from 9000 to any other port.

Q: What are possible states of the alert – is there an Alert Lifecycle?

A: An alert is just a notification of a failure it will just have ‘New’ and ‘Read’ states. This in set using the ‘HasRead’ flag in the Alerts model

Q: How does alert correspond to email notifications ? where is this configured ?

A: Emails will consolidate and send Alerts in a fixed (Configurable schedule). It will only include Alerts generated since the last email.

     For Fatal messages, email will be sent instantly

    Configuration CLI command:

     tql –dashboard –configure <Config file Path>

      This is the design level config file:

{  
   "DashBoardConfig":{  
      "NotificationConfig":{  
         "EmailTo":"abc@xyz.com",
         "Frequency":"1min"
      },
      "ExecutionConfig":[  
         {  
            "ClusterId":"Cluster-1",
            "ExecConfig":{  
               "FacetExecFrequency":"30sec",
               "HttpExecFrequency":"1min"
            }
         },
         {  
            "ClusterId":"Cluster-2",
            "ExecConfig":{  
               "FacetExecFrequency":"45sec",
               "HttpExecFrequency":"2min"
            }
         }
      ]
   }
}

Q: How are we ensuring that we are not sending too many mails when an alert condition arises (for e.g. are we going to have one mail per minute)

A: Emails will consolidate and send Alerts in a fixed (Configurable schedule with "NotificationConfig"."Frequency" in the above JSON). It will only include Alerts generated since the last email. 

     For Fatal messages, email will be sent instantly

Q: How is the pause / unpause of the monitoring / alerting aspect going to be handled – this is in relation to query sent earlier on maintenance windows.

A: This will be a CLI option. On the daemons:

tql –cluster –monitoring -start

tql –cluster –monitoring –stop

This will basically stop the scheduled jobs that execute the alarms