Page Comparison

...

1. Smart Bridge Framework (SBF) - A subcomponent of Device Orchestration Service responsible for ingesting data from a number of configured Device Providers. Device Providers can either push the data to the transport enabled by Smart Bridge Service or Smart Bridge Service can pull the data a scheduled interval from the Device Provider.

2. Stream Processing Engine (SPE)  - A subcomponent of Device Orchestration Service responsible for converting the Device Provider data bridged via Smart Bridge Service into a stream. Stream Processing Engine maintains the device data in memory with periodic disk writes in case of overflow. Interaction between Smart Bridge Service (SBF)  and Stram Processing Engine (SPE) is either local (colocated) or remote mode (via HTTP interface)

3. Device Processor Framework (DPF) - A subcomponent of Device Orchestration Service responsible for processing the device data and store into Device models.

Terminology

...

Term

...

Description

...

Provider

...

“Provider” as the name suggests is a component that provides the functionality to interact with a physical device connected locally or a cloud-based service that in-turn interacts with a device. All providers provide a simple contract via API over a given transport.

...

ServiceInstance

...

ServiceInstance is a collection of all the parameters organized by Service Subcomponents that capture the metadata to a successful runtime execution of a Provider interaction.

...

ServiceConext

...

ServiceContext is the runtime instantiation of a single ServiceInstance. In addition to the ServiceInstance parameters, it contains runtime contextual parameters - for example, Model and attributes with which the Serviceinstance is attached using ServiceServiceFacet.

...

Model

...

Model is a type that represents the information and relationships with other models. In TDL - A model has three types of substructures: attributes, properties, and operations. There are three types of models: Thing Models, Data Models, and App Models. In the context of Service, we are mostly talking about ThingModel.

...

Model Instance

...

A Model Instance is a single concrete occurrence of a given model, existing usually during the runtime.

...

Relationship between Model, Model Instance, ServiceInstance, Provider and ServiceContext

...

• M1, M2, … Mn represents model instances

• D1, D2, … Dn represents Service Instances

• P1, P2, …. Pn represents different providers

• (D1, P1) ordered pair represents single READ and/or Write interaction with a given provider. Note that it is possible for the same provider to give multiple different READ/Write interactions each to capture different aspects of the device. It is recommended to use a different Service Instance for the same provider. Combination of ProviderName and ProviderInteractionName is a unique combination of Provider (See below)

• (M1, D1, P1) ordered pair represents ServiceContext which is a runtime artifact.

• Model Instance M2 can reuse the same ServiceContext (M1, D1, P1); In fact, the same ServiceContext can be applied to entire model instances i.e. in effect to the entire Model.

• Please note that in some cases reusing the same ServiceContext might not be possible due to the underlying limitation of ProtocolHandler connection management with the Device. In such case, its recommended to create a different ServiceContext pair.

Service OpenWeatherMap as Cloud-based Environment Domain Provider Example

Implementation without Service

Without Service, it is implemented as pure ThingModel-ThingFacet by the TQL Developer as explained 7. Weather cloud service

Creating Service Instance

# Device Orcheastration Service Single Instance definition for a
# OpenWeatherMap Weather Cloud Service Provider
Service:
  ServiceName: AtomitonWeatherService;
  ServiceDescription: "This is an example of HTTP (Client) based interaction with OpenWeatherMap Service"
  Tags:
    KeyValue(Key: CustomerName, Value: AtomitonInternal);
    KeyValue(Key: LicenseAgreement, Value: Unlimited);
  DataEncrpytion:
    EncrpytionKeyName: Atomiton.KMS.Key
  ServiceTrigger:
    OnDemand: false
    OnSchedule:
      ScheduleFrequency: PT5S
  SBFOptions:
    ProviderInfo:
      ProviderName: OpenWeatherMap
      ProviderDescription: Open Weather Map Free Licensing APIS
      ProviderInteractionName: FreeSubscription
      Interaction:
        CustomInteractionAction: None #Or override with your own MyCustomInteraction
        InteractionType: Read #(or Write)
        Protocol:
          ProtocolHandlerName: HTTP
          ClientServer: Client
          ConnectionInfo:
              Method: GET
              HostName: http://api.openweathermap.org
              URI: /data/2.5/weather
              QueryString:
                KeyValue:
                  Key: q
                  Value: Pune,IN
                KeyValue:
                  Key: mode
                  Value: xml
                KeyValue:
                  Key: APPID
                  Value: 2b8f32bc27def9f56804c03516516f4d
              RequestHeaders:
                KeyValue:
                  Key: Content-Type
                  Value: text/xml
              RequestMessageValue:
          MessageProcessingHandler:
            MPHanlderName: ASTFW_Service_MessageProcessingHandler_Activity #Can be custom (Rest of parameters will be ignored)
            MapFields:
              Field:
                SourceFieldName: current.temperature.value
                DestinationFieldName: Environment.EnvDetails.Temperature
                Formula:
                DataModification: ASTFW_Service_DataModificationFunc_Truncate
                Validation:
                  ValidationCondition: ASTFW_Service_ValidationFunc_ValuesAreZero #Can be custom
                  ValidationAction: ASTFW_Service_ValidationAct_IgnoreRecord # Can be custom
              Field:
                SourceFieldName: current.humidity.value
                DestinationFieldName: Environment.EnvDetails.Humidity
                Formula: None
                DataModification: ASTFW_Service_DataModificationFunc_Truncate
                Validation:
                  ValidationCondition: ASTFW_Service_ValidationFunc_ValuesAreZero #Can be custom
                  ValidationAction: ASTFW_Service_ValidationAct_IgnoreRecord # Can be custom
              Field:
                SourceFieldName: current.pressure.value
                DestinationFieldName: Environment.EnvDetails.Pressure
                Formula: None
                DataModification: ASTFW_Service_DataModificationFunc_Truncate
                Validation:
                  ValidationCondition: ASTFW_Service_ValidationFunc_ValuesAreZero #Can be custom
                  ValidationAction: ASTFW_Service_ValidationAct_IgnoreRecord # Can be custom
              Field:
                SourceFieldName: context.sid
                DestinationFieldName: Environment.ExtEnvID
                Formula: contextstore
                DataModification: ASTFW_Service_DataModificationFunc_Truncate
                Validation:
                  ValidationCondition: ASTFW_Service_ValidationFunc_ValuesAreZero #Can be custom
                  ValidationAction: ASTFW_Service_ValidationAct_IgnoreRecord # Can be custom
  SPFOptions:
    SPFMode: local #Can be Remote
    SPFTargetFacet: ExtensionInterface
    ReadSubscriber:
      ChannelName: Atomiton.DFBoundStream.EnvironmentDomain #Standard Event will be published directly to this channel
    WriteSubscriber:
      ChannelName: Atomiton.DFBoundStream.WriteChannle.EnvironmentDomain
      ChannelHandler: ASTFW_Service_WriteChannelHandler
      MapFields: # Reverse Mapping information
        Field:
          SourceFieldName: DeviceCloudProvider.CloudThings.ExternalEnv.RequestEnvInfo.Location
          DestinationFieldName: setlocation.value
#Dynamically Added Context is here..
#Context:
#  EntityType: Environment;
#  Sid:

Description of Service Instance Parameters

...

Parameter Name

...

Description

...

An example or Interface Definition

...

Service

...

An overall container of ServiceInstance

...

ServiceName

...

Unique name of the ServiceInstance

...

ServiceDescription

...

Description of ServiceInstance

...

Tags

...

Choose key-value pairs to tag your ServiceInstance. Use tags to organize, track, or control access for this ServiceInstance. For example, a tag can include CustomerName and license agreement

...

DataEncrpytion

...

Encryption ensures that your data and your sensitive information are not accessible by any user or application without a valid key. Encrypting data in transit and data at rest is vital for regulatory compliance.

Service always encrypts your data and sensitive information during transit and at rest. You can either choose a customer master key (CMK) managed by Atomtion (default) or bring your own key to encrypt your data and sensitive information.

...

ServiceTrigger

...

A trigger causes ServiceInstance to run. You can choose from one of three triggers types:

Run on-schedule. Use when you want the ServiceInstance to run automatically at a scheduled frequency.

Event-driven. Use when you want the ServiceInstance to run when an external event happens (As described by the Provider)

Run on-demand. Use when you want to run the ServiceInstance

...

SBFOptions

...

An overall container to hold SBF Options

...

ProviderInfo

...

An overall container to hold Provider Information

...

ProviderName

...

Name of the Provider

...

ProviderDescription

...

A detailed description of the provider

...

ProviderInteractionName

...

Provider interaction name. Combination of ProviderName and ProviderInteractionName must be unique

...

Interaction

...

An overall container to hold the provider interaction

...

CustomInteractionAction

...

Users can specify their own actions than can handle the complete provider interaction. This is useful when there are multiple steps involved in an interaction.

...

InteractionType

...

Interaction can be a “READ” or “WRITE” type. Read is for reading from the provider and write is to write back to the provider.

...

Protocol

...

An overall container to hold the provider protocol communication information

...

ProtocolHandlerName

...

Protocol Handler name - must be one of the A-Stack provided protocol handler name (HttpServerExtensionArgs) or short prefix like - HTTP or OPCUA, etc.

...

ClientServer

...

The interaction can be Client i.e. ServiceInstance initiates a connection at specific trigger interval OR Server i.e. provider pushes data to ServiceInstance

...

ConnectionInfo

...

An overall container to hold protocol connection information. It includes hostname, port, URI constructed via an array of name-value pairs, Request Message Value to be sent as a payload. Header parameters, Protocol Method (GET/PUT/POST..)

...

MessageProcessingHandler

...

An overall container to hold information on how to handle and process the message received from the provider using a given interaction.

...

MessageProcessingHandlerAction

...

Action name that can handle the message. Values can be ASTFW_Service_MessageProcessingHandler_Action or provided by the user. If it is provided by the user, it must take care of handling the mapping of source and destination fields.

...

#  ASTFW Service Message Processing Handler Action
# --- Constants ----------------------------------------------------------------
# --- Arguments ----------------------------------------------------------------
var $cntx = ServiceContext();
var $interactionreponse = $$.argument.valueBy(0);
$ --- Process Message --------------------------------------------------------------------
var $standardEvent = MLP.newMl();
$standardEvent.UpdateSaveEntity.Request.entity.entityType = $cntxt.modelName;
foreach(value: var $field, in: $cntxt.MessageProcessingHandler.MapFields.Field, from:$cntxt.MessageProcessingHandler.MapFields ):
  #Validate
  #Modify
  var fieldValue = modifiedValue
# ------------------------------------------------------------------------------
return = $standardEvent;

Standard Event Structure with an example can be found here

# Standard Event Structure
UpdateSaveEntity:
  Request:
    entity:
      entityType: Environment #or any model
      body:
        #Array of desitnation attributes

...

MapFields

...

An overall container to hold source to the destination field mapping. From the Studio one can Upload a .csv file with mapped fields - Use a comma-separated values (CSV) file to specify the field mappings. Each line in the CSV file contains the source field name, followed by a comma, followed by the destination field name.

...

Field

...

An overall container to hold a single field mapping information

...

SourceFieldName

...

Source filed name from the response of a provider interaction

...

DestinationFieldName

...

Destination field name pointing to the user Thing Model attribute

...

Formula

...

This step is optional. To add a formula that concatenates fields, select two fields from Mapped fields, and then provide a custom formula to chose one of the standard formulas provided. Example: Concatenation.

...

DataModificationAction

...

This step is optional. Some of the data modifications are truncate a string value. Users can specify custom action to handle data modification.

...

Validation

...

Overall container to hold singe data validation. This can be used to discriminate if the same value is sent by the provider.

...

ValidationCondition

...

For Validations, add validations to check whether a field has bad data. For each field, choose the condition that indicates bad data. Service provided conditions are:

• Value is null

• Value is 0

• Value is missing

• Value is negative

• Value contains text

Users can provide their validation own condition.

...

ValidationAction

...

What action ServiceInstance should take when a field in a record is bad. Users can provide their own validation action

ServiceService Facet

ThingFacet(name: ASTFW_ServiceService_Facet):
  String(name: strActivate, KnownBy: ASTFW_ServiceService_ActivateAction, modifiers: virtual);
  String(name: strDisable,  KnownBy: ASTFW_ServiceService_DisableAction, modifiers: virtual);
  String(name: strListen, KnownBy: ASTFW_ServiceService_ListenAction, modifiers: virtual);
  String(name: strStopListen, KnownBy: ASTFW_ServiceService_StopListenAction, modifiers: virtual);
  String(name: strWrite, KnownBy: ASTFW_ServiceService_WriteAction, modifiers: virtual);
  String(name: strServiceConfigID);
  Action(name: ASTFW_ServiceService_ActivateAction, type: BML, limit: 1):
    onDiscard:  # optional cleanup handler
      logInfo = "ASTFW_ServiceService_ActivateAction discarded";
      discard: closed
    #logInfo = ">>>>> strActivate: arguments:" + string($$.argument);
    logInfo = ">>>>> strActivate: ActionArgument:" + $ActionArgument;
    logInfo = ">>>>> strActivate: ActionUpdate:" + $ActionUpdate;
    logInfo = ">>>>>Activating ServiceContext... sid ..." + $ActionArgument.$sid;
    #Check if already deployed; then do nothing
    $modelsid = $ActionArgument.$sid;
    $ServiceContextID = "ServiceContext_" + $ActionArgument.$sid;
    logInfo = ">>>> ServiceContextID is.." + $ServiceContextID;
    $ServiceConfigID = $ActionArgument.strServiceConfigID.Value;
    logInfo = ">>>> ServiceConfigID is..." + $ServiceConfigID;
    scope (fid: SffFacetAgentFacet):
      DoRequest => Process => Message => Value:
        GetFacetInfo(fid = $ServiceContextID);
    $fafresponse = $$.v2.getResponse();
    if ($fafresponse.Message.Value.FacetAgent.Status == 200):
      logInfo = ">>> Noting to do. Already Activated";
    else:
      scope(fid: SffFacetAgentFacet3):
        DoRequest:
          Process => Message => Value:
            $TL.RuntimeParams.ServiceConfigFacetID = $ServiceContextID;
            @: 'resources/Service/spaces/ServiceActivityTemplate.mqp.cdm'
      scope(fid: Service):
        DoRequest => Process => Message => Value:
          Query:
           Find:
            Service:
              ServiceConfigID(eq = $ServiceConfigID);
      $TL.finddata = $$.v2.getResponse();
      var $T.guard = true;
      if ($finddata.Message.Value.Find.Status == "Success"):
        logInfo = ">>>> Found ServiceConfig";
        $TL.RuntimeParams.ServiceConfigFacetID = $ServiceContextID;
        scope(fid = $ServiceContextID):
          $F.Servicecontext = $finddata.Message.Value.Find.Result;
          $F.context.sid = $modelsid;
          $F.context.EntityType = "Environment"                                 #Fixme: Replace dynamically
          StartServiceContext();
      else:
        logInfo = ">>> ServiceConfig Not found";
    return:
      strActivate: activated

  Action(name: ASTFW_ServiceService_DisableAction, type: BML, limit: 1):
    onDiscard:  # optional cleanup handler
      logInfo = "ASTFW_ServiceService_DisableAction discarded";
      discard: closed
    #logInfo = ">>>>> strActivate: arguments:" + string($$.argument);
    #logInfo = ">>>>> strActivate: ActionArgument:" + $ActionArgument;
    #logInfo = ">>>>> strActivate: ActionUpdate:" + $ActionUpdate;
    $ServiceContextID = "ServiceContext_" + $ActionArgument.$sid;
    loginfo = ">>>> ServiceContextID is ..." + $TL.ServiceContextID;
    scope(fid: SffFacetAgentFacet3):
      DoRequest:
        Process => Message => Value:
           DelFacetInstance(fid = $ServiceContextID)
    return:
      strActivate: disabled

  Action(name: ASTFW_ServiceService_ListenAction, type: BML, limit: 1):
    onDiscard:  # optional cleanup handler
      logInfo = "ASTFW_ServiceService_ListenAction discarded";
      discard: closed
    logInfo = ">>>>> ListenAction: arguments:" + string($$.argument);
    logInfo = ">>>>> ListenAction: ActionArgument:" + $ActionArgument;
    logInfo = ">>>>> ListenAction: ActionUpdate:" + $ActionUpdate;
    logInfo = ">>>> Starting to Listen";
    $ServiceContextID = "ServiceContext_" + $ActionArgument.$sid;
    loginfo = ">>>> ServiceContextID is ..." + $TL.ServiceContextID;
    scope(fid = $ServiceContextID):
      DoRequest => Process => Message => Value:
        startProviderConnection();
    return:
      strActivate: StartListen

  Action(name: ASTFW_ServiceService_StopListenAction, type: BML, limit: 1):
    onDiscard:  # optional cleanup handler
      logInfo = "ASTFW_ServiceService_StopListenAction discarded";
      discard: closed
    #logInfo = ">>>>> strActivate: arguments:" + string($$.argument);
    #logInfo = ">>>>> strActivate: ActionArgument:" + $ActionArgument;
    #logInfo = ">>>>> strActivate: ActionUpdate:" + $ActionUpdate;
    $ServiceContextID = "ServiceContext_" + $ActionArgument.$sid;
    loginfo = ">>>> ServiceContextID is ..." + $TL.ServiceContextID;
    scope(fid = $ServiceContextID):
      DoRequest => Process => Message => Value:
        stopProviderConnection();
    return:
      strActivate: StopListen

  Action(name: ASTFW_ServiceService_WriteAction, type: BML, limit: 1):
    onDiscard:  # optional cleanup handler
      logInfo = "ASTFW_ServiceService_WriteAction discarded";
      discard: closed
    #logInfo = ">>>>> strActivate: arguments:" + string($$.argument);
    #logInfo = ">>>>> strActivate: ActionArgument:" + $ActionArgument;
    #logInfo = ">>>>> strActivate: ActionUpdate:" + $ActionUpdate;
    var $postReq:
      Message => Value:
        DelFacetInstance:
          fid: Service
    scope(fid: SffFacetAgentFacet3):
      DoRequest:
        Target: SffFacetAgentFacet3
    return:
      strActivate: Write

Service Config Meta Data

# Service Meta Model Definition
Namespace(Name: Service):
 Domain(Name: ServiceMetaModel):
  Def(Name: DefKeyValue):
   String(Name: Key)
   String(Name: Value)

  Def(Name: DefTags):
   DefKeyValue(Name: KeyValue, cardinality: 0..m)

  Def(Name: DefDataEncryption):
   String(Name: EncrpytionKeyName)

  Def(Name: DefQueryString):
   DefKeyValue(Name: KeyValue, cardinality: 0..m)

  Def(Name: DefRequestHeaders):
   DefKeyValue(Name: KeyValue, cardinality: 0..m)

  Def(Name: DefConnectionInfo):
   String(Name: Method)
   String(Name: HostName)
   String(Name: URI)
   DefQueryString(Name: QueryString)
   DefRequestHeaders(Name: RequestHeaders)
   String(Name: RequestMessageValue)

  Def(Name: DefValidation):
   String(Name: ValidationCondition)
   String(Name: ValidationAction)

  Def(Name: DefOnSchedule):
   String(Name: ScheduleFrequency)

  Def(Name: DefServiceTrigger):
   Boolean(Name: OnDemand, default: false);
   DefOnSchedule(Name: OnSchedule)

  Def(Name: DefMessageProcessingHandler):
   String(Name: MPHanlderName)
   DefMapFields(Name: MapFields)

  Def(Name: DefProtocol):
   String(Name: ProtocolHandlerName)
   String(Name: ClientServer)
   DefConnectionInfo(Name: ConnectionInfo)
   DefMessageProcessingHandler(Name: MessageProcessingHandler)

  Def(Name: DefInteraction):
   String(Name: CustomInteractionAction)
   String(Name: InteractionType)
   DefProtocol(Name: Protocol)

  Def(Name: DefProviderInfo):
   String(Name: ProviderName)
   String(Name: ProviderDescription)
   String(Name: ProviderInteractionName)
   DefInteraction(Name: Interaction)

  Def(Name: DefField):
   String(Name: SourceFieldName)
   String(Name: DestinationFieldName)
   String(Name: Formula)
   String(Name: DataModification)
   DefValidation(Name: Validation)

  Def(Name: DefMapFields):
   DefField(Name: Field, cardinality: 0..m)

  Def(Name: DefSBFOptions):
   DefProviderInfo(Name: ProviderInfo)

  Def(Name: DefReadSubscriber):
   String(Name: ChannelName)

  Def(Name: DefWriteSubscriber):
   String(Name: ChannelName)
   String(Name: ChannelHandler)
   DefMapFields(Name: MapFields)

  Def(Name: DefSPFOptions):
   String(Name: SPFMode)
   String(Name: SPFTargetFacet)
   DefReadSubscriber(Name: ReadSubscriber)
   DefWriteSubscriber(Name: WriteSubscriber)

  Def(Name: DefService, inherits: DataModel):
   Sid(Name: ServiceConfigID)
   String(Name: ServiceName)
   String(Name: ServiceDescription)
   DefTags(Name: Tags)
   DefDataEncryption(Name: DataEncrpytion)
   DefServiceTrigger(Name: ServiceTrigger)
   DefSBFOptions(Name: SBFOptions)
   DefSPFOptions(Name: SPFOptions)

  DefService(Name: Service)

Service End-2-End Flow

Design Time Flow

Runtime Flow

#
Name: DeviceCloudProvider
Domain(Name: CloudThings):
  ThingModel(Combines: ASTFW_ServiceService_Facet, Name: ExternalEnv):
    Sid(Name: ExtEnvId);
    EnvInfo(Name: EnvDetails, KnownBy: ASTFW_ServiceService.ListenAction);
  Def:
    Include: "[:RuntimeParams.TypesLocation:]/envinfo.types.xml"

Northbound to Soundbound Communication

Service Monitoring and Metering

ServiceRunHistory

Querying Service

You can query Service using a ServiceName, ProviderName, or any fields part of the ServiceInstance meta-model.

Dashboard APIs

Service Deployment Options

Service Design Review Comments

Service Initial TImelines

...

Task Details

...

Completion Date

...

Status

...

Modeling of Service Meta-Model (ServiceInstances, etc) (Venkata)

...

6/5/20

...

Done

...

Service ServiceFacet (Baseer)

...

6/9/20

...

Done

...

Core Logic to implement the execution of one ServiceContext. (Baseer / Venkata)

• HTTP Connection

• Perif Connection

• MessageHandler

• Standard Event to Model

...

6/12/20

...

Done

...

Refactor Design taking Activity Modeling into consideration (Baseer)

...

6/12/20

...

Done

...

Integration testing for two use cases:

• Simple Perif

• Smart Cities use case

...

6/19/20

...

Done

...

Deployment Models

...

6/16/20

...

Use cases demo

...

6/22/20

AMQP, OPC, MQTT Connection handler

History Logging

Dynamically invoke custom overridden actions

...

Requirements for DOS

The latest version of A-Stack TQLStudio

Connections to HTTP, WS, AMQP, MQTT, Perif, and OPC protocol handlers are supported out of the box.

Getting Started with DOS

This tutorial provides a hands-on introduction to Atomiton DOS. You create a DOS configuration to move data between a provider data source and your models.

Tasks:

• Pre-requisites

• Step 1: Create a new TQLStudio Project

• Step 2: Prepare DOS Configuration files

• Step 3: Map data fields

• Step 4: Add filters, custom handlers (if any)

• Step 5: Review and Create DOS Configuration

• Step 6: Activate and Listen to Data

Step 1: Create a new TQLStudio Project

Step 2: Prepare DOS Configuration files

Step 3: Map data fields

Step 4: Add filters, custom handlers (if any)

Step 5: Review and Create DOS Configuration

Step 6: Activate and Listen to Data

DOS Supported Handlers and Functions

Message Handlers

Filter Functions

Description of DOS Configuration Parameters

Service End-2-End Runtime Flow

Runtime Flow

Service Monitoring and Metering

ServiceRunHistory