Cluster Monitoring Model Description

AlarmCondition Model

ConditionDef

Types and

...

attributes:

The ConditionDef will have each of these attributes but will be populated based on the type of the Condition

Type: Facet

Test: Check if the Facet is running

Type: HTTP

Test: Check if the HTTP endpoint is UP

Type: WS

Test: Check if the websocket endpoint is UP

Type: Sequence

Test: Check if the sequence is executing

Type: Log

Test: Check if the sequence is executing

RecoveryActionModel

Type: ExecuteQuery

Test: Check if the sequence is executing

Type: Script

Test: Check if the sequence is executing

Alert Model

...

The following steps will be performed on the deamondaemon:

1. Rules are added by the use through the API or csv file upload
2. A job is executed every minute that does the following:
   1. Picks up a AlarmCondition
   2. Executes the condition
   3. On Success
      1. Updates the LastRun with current time and LastStatus to 'Success'
   4. On Failure
      1. Updates the LastRun with current time and LastStatus to 'Failure'
      2. Generates an alert with Condition, ClusterId, InstanceName, and Error Reason
      3. If Recovery Action is present try and execute the action recovery could be
         1. Try and restart failing component
         2. Try and restart the A-Stack engine on the failing node
3. Find new alerts (Newer than the last run) Sends email notification for the alert

Alarm State Transition Diagram

Provisioning Alarms Using CSV File

Command Line API

Load Alarm Conditions

This Command loads the AlarmConditions into the monitoring system.

• Command: tql -monitoringconfig -load Path=

...

• <Path of the File>/AlarmConditions.csv

Delete Alarm Conditions

This command lets you delete one or all AlarmConditions from the system based on the inputs provided.

• Command: tql -monitoringconfig -delete ClusterId=Cluster-1,Instance=Instance-1,Name=Test_Endpoint_

...

• Facet

Deletes AlarmCondtion called Test_Endpoint_Facet from instance Instance-1 on cluster Cluster-1

• Command: tql -monitoringconfig -delete ClusterId=Cluster-1,Instance=Instance-1

Deletes all AlarmCondtions from instance Instance-1 on cluster Cluster-1

Get Alarm Conditions

This command gets AlarmCondtions from the system based on the inputs provided. And writes it to the file provided in the Path variable

• Command: tql -monitoringconfig -get ClusterId=Cluster-1,Instance=Instance-1,Name=Test_Endpoint_

...

• Facet,Path=

...

• <File Path>\Result.json

Gets AlarmCondtion called Test_Endpoint_Facet from instance Instance-1 on cluster Cluster-1

• Command: tql -monitoringconfig -get ClusterId=Cluster-1,Instance=Instance-1,Path=

...

• <File Path>\Result.json

Gets all AlarmCondtions from instance Instance-1 on cluster Cluster-1

Update Alarm Conditions

Updates an AlarmCondition values. First run the get Command to get the AlarmCondition, make the Necessary changes and provide this file path in the "Path" variable.

• Command: tql -monitoringconfig -update ClusterId=Cluster-1,Path=

...

• <File Path>\Result.json

Update Email Configuration

Updates the Email configuration. ClusterId is optional, if not specified updates will happen on all cluster.

• Command: tql -monitoringconfig -config Path=

...

• <File Path>\EmailConfig.json,Type=setemail,ClusterId=Cluster-1

Updates configuration on cluster Cluster-1

• Command: tql -monitoringconfig -config Path=<File Path>\EmailConfig.json,Type=setemail,ClusterId=Cluster-1

Updates configuration on all clusters

Sample Configuration:

{  
   "NotificationConfig":{
	  "Host":"HOST_NAME",
      "Port":"PORT_NUMBER",
      "Username":"UNAME",
      "Password":"PWD",
      "From":"from@domain.com",
      "To":"recepient@domain.com,recepient2@domain1.com",
      "Subject":"Alert Generated Notifications"
   }
}

Get Email Configuration

Get the Email configuration and prints it on the screen. ClusterId is mandatory.

• Command: tql -monitoringconfig -config Type=getemail,ClusterId=Cluster-1

Update Schedule Configuration

Updates the schedule configuration. ClusterId is optional, if not specified updates will happen on all cluster.

• Command: tql -monitoringconfig -config Path=E:\Atomiton\Builds\TQLEngineConfiguratorNode1\resources\atomiton\configurator\spaces\SysConfig.json,Type=setschedule,ClusterId=Cluster-1

{  
   "SysConfig":[
		{"EMAIL_FREQ":"1min"},
		{"FACET_FREQ":"1min"},
		{"HTTP_FREQ":"1min"},
		{"SEQ_FREQ":"1min"},
		{"LOG_FREQ":"1min"},
		{"INFO_FREQ":"1min"},
		{"WS_FREQ":"1min"},
		{"ALERT_PURGE_FREQ":"1min"},
		{"ALERT_PURGE_LIMIT_DAYS":"25"}
   ]
}

• FACET_FREQ: Frequency with which Facet type AlarmConditions are executed.
• HTTP_FREQ: Frequency with which HTTP type AlarmConditions are executed.
• SEQ_FREQ: Frequency with which Sequence type AlarmConditions are executed.
• LOG_FREQ: Frequency with which Log type AlarmConditions are executed.
• INFO_FREQ: Frequency with which Info type AlarmConditions are executed.
• WS_FREQ: Frequency with which WS type AlarmConditions are executed.
• ALERT_PURGE_FREQ: Frequency with which stale alerts are purged.
• ALERT_PURGE_LIMIT_DAYS: How old should an alert be before it is considered stale.
• EMAIL_FREQ: Frequency with which Alert emails are sent.

Get Schedule Configuration

Get the schedule configuration and prints it on the screen. ClusterId is mandatory.

• Command: tql -monitoringconfig -config Type=getschedule,ClusterId=Cluster-1

Stop Monitoring

Pause a monitoring service. ClusterId is mandatory.

Command: tql -monitoringconfig -stopmonitoring ClusterId=Cluster-1

Start Moniting

Resume a monitoring service. ClusterId is mandatory.

Command: tql -monitoringconfig -startmonitoring ClusterId=Cluster-1

Update purge configuration for management dashboard'

Update Alert purging configuration on the Dashboard.

Command: tql -monitoringconfig -alertpurgedashboard Purgefreq=1min,Purgelimit=20

• Purgefreq: Frequency with which stale alerts are purged.

• Purgelimit: How old should an alert be before it is considered stale

Management Dashboard

A scheduled job will run on the dashboard back-end that will periodically (Every minute) pull the AlarmConditions, Alerts and Notifications sent and store it onto itself for displaying on the UI.

...