Table of Contents | ||||||
---|---|---|---|---|---|---|
|
Cluster Terminology
This topic introduces some of the concepts unique to the cluster creation and communication feature of A-Stack.
Terminology | Description | Synonym Terms |
---|---|---|
Network | A collection of A-Stack running instances that communicates (notifications) and keep persistent data store in sync. | Cluster, Clustering |
Peer | A-Stack running instance that is participating in forming a Network | |
Node | A-Stack running instance | Peer if participating in forming a network. |
Group | Logical Name defined to make communication type between a collection of Nodes | |
Advertise | Send Notifications to peer or group | |
Replicate | Replication of data (TQL Storage) between nodes or a group of nodes |
...
...
language | xml |
---|---|
theme | Emacs |
title | SffNetworkFacet with AMQP Transport |
linenumbers | true |
...
- Please refer to AMQP Protocol Handler for details configurtion parameters. AMQP Protocol Handler parmeters are used in defining Borker Exchange, bindings and Routing Key.
- Group also subscribes to the exchange so it can receive messages published by other peers to that group.
- Group communication over AMQP must have separate Send and Receive configurations which corresponds to two separate connections for publish and subscribe. Engine recognizes pub/sub style of group communication by looking whether these configurations are present.
- New PeerQName parameter is now available to aid in defining peer-unique names like queue name. Each distinct peer must have its own unique queue on the exchange. This parameter is resolved to self QName of the cluster peer. That is, If you’re running on port 8080 it will be “N1.P0” and on port 8081 it will be “N1.P1” etc
- Routing key should be unique per cluster per group. In the above example I defined it as cluster.[:QName:], which would be resolved as cluster.G1. In the presence of multiple clusters you should probably use something like [:fid:].[:QName:] instead. Same goes for peer queue names (i.e. QueueName="[:fid:].[:PeerQName:]" or so).
For the Network Definiton below: It results in following variable values.
Name | Value |
---|---|
QName | AMQPCluster.Region1_Cluster1_Server1 |
GroupName | Region1_Cluster |
RoutingKey | cluster.Region1_Cluster1 (Note that RoutingKey is unique per Exchange / Cluster definiton) |
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
<NewPackage> <RuntimeParams> <RegionName>Region1</RegionName> <RegionSectionName>Cluster1</RegionSectionName> <ClusterName>[:RuntimeParams.RegionName:]_[:RuntimeParams.RegionSectionName:]</ClusterName> </RuntimeParams> <NewFacetInstance Name="wsNotificationCluster" Type="SffNetworkFacet" Context="new" fid="wsNotificationCluster"> <OnActivate> <AddProcessData Key="ClusterPeers.Peer"> <ImportFacet>[:RuntimeParams.MacroFacetID:]</ImportFacet> <SetLocalData Key="peers"> <Value> <Value> <Server <Name>Id="[:$LocalDataRuntimeParams.server.IdClusterName:]</Name> _Server1" Host="bkhan" Port="8082"/> <URL>ws://<Server Id="[:$LocalDataRuntimeParams.server.Host:]:[:$LocalData.server.Port:]/fid-wsNotificationCluster</URL>ClusterName:]_Server2" Host="bkhan" Port="8083"/> <AdvertiseTo><Server Id="[:RuntimeParams.ClusterName:]</AdvertiseTo> _Server3" Host="bkhan" Port="8084"/> <Server <Group>Id="[:RuntimeParams.ClusterName:]</Group> _Server4" Host="bkhan" Port="8085"/> </Value> </AddProcessData>SetLocalData> </For> <Log Level="INFO" Message="TQLNotificationCluster : Federation AdvertiseTo servers [:$ProcessData.DeviceEngines:]"/><For Each="server" From="$LocalData.peers" In="Server"> <AddProcessData Key="ClusterPeers.Peer"> <Process> <Network> <Value> <OnPeerError> <Name>[:$LocalData.server.Id:]</Name> <Log Level="error" Message="TQLNotificationCluster : Peer [:$ContextData.$Peer.Key:] error: [:$Error:]"/><URL>ws://[:$LocalData.server.Host:]:[:$LocalData.server.Port:]/fid-wsNotificationCluster</URL> <AdvertiseTo>[:RuntimeParams.ClusterName:]</OnPeerError>AdvertiseTo> <Namespace Name="AMQPCluster"> <Group>[:RuntimeParams.ClusterName:]</Group> <Include>$ProcessData.ClusterPeers</Include></Value> </AddProcessData> </Namespace> </For> <Group<Log nameLevel="[:RuntimeParams.ClusterName:]INFO" UserNameMessage="tql" Password="tql12345" TQLNotificationCluster : Federation AdvertiseTo servers [:$ProcessData.DeviceEngines:]"/> <Process> VirtualHost="/" Durability="true" <Network> <OnPeerError> Host="mqtt.atomiton.com" Port="5672" ExchangeName<Log Level="AtomitonFanoutExchangeerror" Message="TQLNotificationCluster : Peer [:$ContextData.$Peer.Key:] error: [:$Error:]"/> ExchangeType="fanout" QueueName="peer.[:PeerQName:]"></OnPeerError> <Namespace Name="AMQPCluster"> <Send post="amqp://?ClientType=publisher& <Include>$ProcessData.ClusterPeers</Include> </Namespace> UserName=[:UserName:]& <Group name="[:RuntimeParams.ClusterName:]" UserName="tql" Password="tql12345" Password=[:Password:]& VirtualHost="/" Durability="true" Host="mqtt.atomiton.com" Port="5672" VirtualHost=[:VirtualHost:]& ExchangeName="AtomitonFanoutExchange" HostName=ExchangeType="fanout" QueueName="peer.[:HostPeerQName:]&"> <Send post="amqp://?ClientType=publisher& PortNumber=[:Port:]& ExchangeName=[:ExchangeNameUserName=[:UserName:]& ExchangeTypePassword=[:ExchangeTypePassword:]& QueueNameVirtualHost=[:QueueNameVirtualHost:]& DurabilityHostName=[:DurabilityHost:]">& <RoutingKey>cluster.PortNumber=[:QNamePort:]</RoutingKey>& <Template> ExchangeName=[:ExchangeName:]& <Message> ExchangeType=[:ExchangeType:]& <Value> QueueName=[:QueueName:]& <PublishMessage>[:$Request.Message.Value:]</PublishMessage> Durability=[:Durability:]"> </Value> <RoutingKey>cluster.[:QName:]</RoutingKey> </Message> <Template> </Template> <Message> </Send> <Value> <Receive get="amqp://?ClientType=subscriber& <PublishMessage>[:$Request.Message.Value:]</PublishMessage> UserName=[:UserName:]& </Value> Password=[:Password:]& </Message> </Template> VirtualHost=[:VirtualHost:]& </Send> <Receive HostNameget=[:Host:]"amqp://?ClientType=subscriber& PortNumberUserName=[:PortUserName:]& ExchangeNamePassword=[:ExchangeNamePassword:]& ExchangeTypeVirtualHost=[:ExchangeTypeVirtualHost:]& QueueNameHostName=[:QueueNameHost:]& DurabilityPortNumber=[:DurabilityPort:]"& as="ServerPipeline" disable="CMD_SEND"> ExchangeName=[:ExchangeName:]& <Message> ExchangeType=[:ExchangeType:]& <Value> QueueName=[:QueueName:]& <AMQPSubscriptionConfig> <RoutingKey value="cluster.Durability=[:QNameDurability:]"/> </AMQPSubscriptionConfig> as="ServerPipeline" disable="CMD_SEND"> </Value> <Message> </Message> <Value> </Receive> <AMQPSubscriptionConfig> </Group> </Network> </Process> <RoutingKey value="cluster.[:QName:]"/> <SffReport Name="wsNotificationCluster_OnActivate"/> </OnActivate> <OnOpen ModifyPipeline="WsServerExtensionArgs" Context="keep"/> </AMQPSubscriptionConfig> <Policy name="MyMetering" type="SffSimplePolicy" kind="metering"> <inc type="integer" target="RequestCount"/> <add type="integer" target="TotalBytes" value="[:$RawData.TotalBytesRead:]"/> </Value> <set type="integer" target="AvgRequestSize" value="[:$ThisData/(number(TotalBytes div RequestCount)):]"/> </Policy> </Message> <OnError> <SetLocalData Key="guard" Value=""/> </Receive> <ReportError> <Error>[:[:$LocalData.guard:]$Error:] </Error>Group> <occurredIn>TQLNotificationCluster<</occurredIn>Network> </ReportError>Process> <SffReport Name="wsNotificationCluster_OnErrorOnActivate"/> </OnError>OnActivate> </NewFacetInstance> </NewPackage> <OnOpen ModifyPipeline="WsServerExtensionArgs" Context="keep"/> <Policy name="MyMetering" type="SffSimplePolicy" kind="metering"> <inc type="integer" target="RequestCount"/> <add type="integer" target="TotalBytes" value="[:$RawData.TotalBytesRead:]"/> <set type="integer" target="AvgRequestSize" value="[:$ThisData/(number(TotalBytes div RequestCount)):]"/> </Policy> <OnError> <SetLocalData Key="guard" Value=""/> <ReportError> <Error>[:[:$LocalData.guard:]$Error:]</Error> <occurredIn>TQLNotificationCluster</occurredIn> </ReportError> <SffReport Name="wsNotificationCluster_OnError"/> </OnError> </NewFacetInstance> </NewPackage> |
Broker Exchange and Bindings
AMQP Vs WS Performance
Using AMQP as a cluster transport does have a cost associated with it due to network involving communication with AMQP borker. Sample Test setup will help us understand the performance cost comparison between the two transports.
Test Setup
Transport | Broker Instance Type | Boroker Verison | Cluster Size | Cluster Relationship | Database |
---|---|---|---|---|---|
AMQP | AWS EC2 m1.medium | RabbitMQ 3.2.4 | 4 | AdvertiseTo | Remote Postgress RDS |
WS | - | - | 4 | AdvertiseTo | Remote Postgress RDS |
Test Results
Cluster Topologies
Choosing the correct data storage strategy depends on the type of application (simple to complex), deployment and interoperability to other platforms. Here are some basic recommended guidelines
...
o For on-the-same-box deployment this is best used with configured port ranges (e.g. sff.server.port=”[8080..8089], see v1.1.1.201704130000). This way your configuration can be completely static and shared across all peers. Any new peer will simply pick up next port from the range and join the cluster. Everything is nice and predictable (i.e. no messing with sff.server.port=0 ephemerals)
...
- Transport Protocol Limitation: Only web socket protocol is supported for peer communications.
- No Auto / Mutual Discovery: There is no mutual discovery yet so each peer must be provided with the same network definition. For example, the exact same definition above can be deployed on three different engine instances, bound to ports 8080, 8082 and 8083 correspondently. Each instance will have one instance of SffNetworkFacet deployed with same fid-cluster. Each instance will figure out its own name/position in the network topology based on host, port and fid
- No inter-group communication: There is no inter-group communications yet, only along relationship links
- No detection of Virtual Peer: It is not yet smart enough to figure out a virtual peer deployed on the same engine so if you configure two related peers and deploy both on the same box, it will try to open a WS to itself instead of using internal communications
- No detecton of Duplex Links: It is not yet smart enough to figure out duplex links so each peer will open client WS to its target even if server WS is already open.
- Peer relationships are supported:
- AdvertizeTo sends data change notifications to the target peer. This is useful while working with a shared DB. This way subscribers will get notified about DB change even if they are connected to a different peer
- ReplicateTo sends data change requests to the target peer. This effectively replicates changes made to the source DB on the target. Only visible (by target) models/attributes will be updated so facet security is enforced. Multiple sources are allowed as well as local modifications on the target (but local modifications will currently NOT be propagated back to source). In case of multiple source-originated modifications, only the latest one will be applied. Timestamps and versions are used for collision disambiguation. In other words, version numbers work like Lamport clocks across the network. Local modifications follows the usual rules
- In other words, this cluster works like a tree structure where children can advertise/replicate local changes to the parent. Although possible and legal, it is NOT recommended to have the same instances on different sibling children to avoid any problems with consistency at least until we test it some more. If you need to move instance from one child to another then you need to explicitly delete instance on the first child (which should propagate to parent) and then re-create instance on another child (which is also propagated). This will allow versions on the parent to start from 1 again and should not result in any problems with further updates as versions will remain synchronized between child and parent.fid
- No inter-group communication: There is no inter-group communications yet, only along relationship links
- No detection of Virtual Peer: It is not yet smart enough to figure out a virtual peer deployed on the same engine so if you configure two related peers and deploy both on the same box, it will try to open a WS to itself instead of using internal communications
- No detecton of Duplex Links: It is not yet smart enough to figure out duplex links so each peer will open client WS to its target even if server WS is already open.
- Peer relationships are supported:
- AdvertizeTo sends data change notifications to the target peer. This is useful while working with a shared DB. This way subscribers will get notified about DB change even if they are connected to a different peer
- ReplicateTo sends data change requests to the target peer. This effectively replicates changes made to the source DB on the target. Only visible (by target) models/attributes will be updated so facet security is enforced. Multiple sources are allowed as well as local modifications on the target (but local modifications will currently NOT be propagated back to source). In case of multiple source-originated modifications, only the latest one will be applied. Timestamps and versions are used for collision disambiguation. In other words, version numbers work like Lamport clocks across the network. Local modifications follows the usual rules
- In other words, this cluster works like a tree structure where children can advertise/replicate local changes to the parent. Although possible and legal, it is NOT recommended to have the same instances on different sibling children to avoid any problems with consistency at least until we test it some more. If you need to move instance from one child to another then you need to explicitly delete instance on the first child (which should propagate to parent) and then re-create instance on another child (which is also propagated). This will allow versions on the parent to start from 1 again and should not result in any problems with further updates as versions will remain synchronized between child and parent.
Deploy, Manage and Monitor Cluster
In this section we discuss how to deploy, manage and monitor cluster. A-Stack support to deploy, manage and monitor cluster can described using following high level picture
High Level Architecture of Cluster Management
Managing Multiple Clusters
A-Stack offers following components to help deploy, manage and monitor array of clusters
Component | Description |
---|---|
Configurator Daemon | A-Stack Runtime with configurator models to help signal cluster management on a given Cluster host |
TQLConsole - ThingSpace Configurator | User Interface based deployment of a remote cluster |
TQL Command Line Interface (Cluster Option) | Command line interface to create, start, stop, list a remote cluster |
TQL Command Line Interface (MonitorDashboard Option) | Command line interface to provision alarms, notification options |
Cluster Monitoring Dashboard | Read-onlty view of the cluster, alarms, and alerts |
Atomiton IT Infrastructure and Cluster Moniotring Dashboard
Cluster Monitoring Dashboard is utilitzed within Atomiton IT infrastrcuture as well. Below the current Cluster Configuration of Atomiton IT infrastructure.