Device Orchestration Service (DOS)

Device Orchestration Service Overview

A-Stack Device Orchestration Service (DOS) is a component that allows device data from a number of device providers to bridge into the device models. As part of bridging multiple devices, data streams are created leveraging stream processing engines with the ability to process the raw device data into the selected device processors functions. Device Orchestration Service can be thought of three major components -

Smart Bridge Framework (SBF) - A subcomponent of Device Orchestration Service responsible for ingesting data from a number of configured Device Providers. Device Providers can either push the data to the transport enabled by Smart Bridge Service or Smart Bridge Service can pull the data a scheduled interval from the Device Provider.
Stream Processing Engine (SPE) - A subcomponent of Device Orchestration Service responsible for converting the Device Provider data bridged via Smart Bridge Service into a stream. Stream Processing Engine maintains the device data in memory with periodic disk writes in case of overflow. Interaction between Smart Bridge Service (SBF) and Stram Processing Engine (SPE) is either local (colocated) or remote mode (via HTTP interface)
Device Processor Framework (DPF) - A subcomponent of Device Orchestration Service responsible for processing the device data and store into Device models.

Terminology

Term	Description
Provider	“Provider” as the name suggests is a component that provides the functionality to interact with a physical device connected locally or a cloud-based service that in-turn interacts with a device. All providers provide a simple contract via API over a given transport.
ServiceInstance	ServiceInstance is a collection of all the parameters organized by Service Subcomponents that capture the metadata to a successful runtime execution of a Provider interaction.
ServiceConext	ServiceContext is the runtime instantiation of a single ServiceInstance. In addition to the ServiceInstance parameters, it contains runtime contextual parameters - for example, Model and attributes with which the Serviceinstance is attached using ServiceServiceFacet.
Model	Model is a type that represents the information and relationships with other models. In TDL - A model has three types of substructures: attributes, properties, and operations. There are three types of models: Thing Models, Data Models, and App Models. In the context of Service, we are mostly talking about ThingModel.
Model Instance	A Model Instance is a single concrete occurrence of a given model, existing usually during the runtime.
Relationship between Model, Model Instance, ServiceInstance, Provider and ServiceContext	M1, M2, … Mn represents model instances D1, D2, … Dn represents Service Instances P1, P2, …. Pn represents different providers (D1, P1) ordered pair represents single READ and/or Write interaction with a given provider. Note that it is possible for the same provider to give multiple different READ/Write interactions each to capture different aspects of the device. It is recommended to use a different Service Instance for the same provider. Combination of ProviderName and ProviderInteractionName is a unique combination of Provider (See below) (M1, D1, P1) ordered pair represents ServiceContext which is a runtime artifact. Model Instance M2 can reuse the same ServiceContext (M1, D1, P1); In fact, the same ServiceContext can be applied to entire model instances i.e. in effect to the entire Model. Please note that in some cases reusing the same ServiceContext might not be possible due to the underlying limitation of ProtocolHandler connection management with the Device. In such case, its recommended to create a different ServiceContext pair.

Service OpenWeatherMap as Cloud-based Environment Domain Provider Example

Implementation without Service

Without Service, it is implemented as pure ThingModel-ThingFacet by the TQL Developer as explained 7. Weather cloud service

Creating Service Instance

# Device Orcheastration Service Single Instance definition for a
# OpenWeatherMap Weather Cloud Service Provider
Service:
  ServiceName: AtomitonWeatherService;
  ServiceDescription: "This is an example of HTTP (Client) based interaction with OpenWeatherMap Service"
  Tags:
    KeyValue(Key: CustomerName, Value: AtomitonInternal);
    KeyValue(Key: LicenseAgreement, Value: Unlimited);
  DataEncrpytion:
    EncrpytionKeyName: Atomiton.KMS.Key
  ServiceTrigger:
    OnDemand: false
    OnSchedule:
      ScheduleFrequency: PT5S
  SBFOptions:
    ProviderInfo:
      ProviderName: OpenWeatherMap
      ProviderDescription: Open Weather Map Free Licensing APIS
      ProviderInteractionName: FreeSubscription
      Interaction:
        CustomInteractionAction: None #Or override with your own MyCustomInteraction
        InteractionType: Read #(or Write)
        Protocol:
          ProtocolHandlerName: HTTP
          ClientServer: Client
          ConnectionInfo:
              Method: GET
              HostName: http://api.openweathermap.org
              URI: /data/2.5/weather
              QueryString:
                KeyValue:
                  Key: q
                  Value: Pune,IN
                KeyValue:
                  Key: mode
                  Value: xml
                KeyValue:
                  Key: APPID
                  Value: 2b8f32bc27def9f56804c03516516f4d
              RequestHeaders:
                KeyValue:
                  Key: Content-Type
                  Value: text/xml
              RequestMessageValue:
          MessageProcessingHandler:
            MPHanlderName: ASTFW_Service_MessageProcessingHandler_Activity #Can be custom (Rest of parameters will be ignored)
            MapFields:
              Field:
                SourceFieldName: current.temperature.value
                DestinationFieldName: Environment.EnvDetails.Temperature
                Formula:
                DataModification: ASTFW_Service_DataModificationFunc_Truncate
                Validation:
                  ValidationCondition: ASTFW_Service_ValidationFunc_ValuesAreZero #Can be custom
                  ValidationAction: ASTFW_Service_ValidationAct_IgnoreRecord # Can be custom
              Field:
                SourceFieldName: current.humidity.value
                DestinationFieldName: Environment.EnvDetails.Humidity
                Formula: None
                DataModification: ASTFW_Service_DataModificationFunc_Truncate
                Validation:
                  ValidationCondition: ASTFW_Service_ValidationFunc_ValuesAreZero #Can be custom
                  ValidationAction: ASTFW_Service_ValidationAct_IgnoreRecord # Can be custom
              Field:
                SourceFieldName: current.pressure.value
                DestinationFieldName: Environment.EnvDetails.Pressure
                Formula: None
                DataModification: ASTFW_Service_DataModificationFunc_Truncate
                Validation:
                  ValidationCondition: ASTFW_Service_ValidationFunc_ValuesAreZero #Can be custom
                  ValidationAction: ASTFW_Service_ValidationAct_IgnoreRecord # Can be custom
              Field:
                SourceFieldName: context.sid
                DestinationFieldName: Environment.ExtEnvID
                Formula: contextstore
                DataModification: ASTFW_Service_DataModificationFunc_Truncate
                Validation:
                  ValidationCondition: ASTFW_Service_ValidationFunc_ValuesAreZero #Can be custom
                  ValidationAction: ASTFW_Service_ValidationAct_IgnoreRecord # Can be custom
  SPFOptions:
    SPFMode: local #Can be Remote
    SPFTargetFacet: ExtensionInterface
    ReadSubscriber:
      ChannelName: Atomiton.DFBoundStream.EnvironmentDomain #Standard Event will be published directly to this channel
    WriteSubscriber:
      ChannelName: Atomiton.DFBoundStream.WriteChannle.EnvironmentDomain
      ChannelHandler: ASTFW_Service_WriteChannelHandler
      MapFields: # Reverse Mapping information
        Field:
          SourceFieldName: DeviceCloudProvider.CloudThings.ExternalEnv.RequestEnvInfo.Location
          DestinationFieldName: setlocation.value
#Dynamically Added Context is here..
#Context:
#  EntityType: Environment;
#  Sid:

Description of Service Instance Parameters

Parameter Name	Description	An example or Interface Definition
Service	An overall container of ServiceInstance
ServiceName	Unique name of the ServiceInstance
ServiceDescription	Description of ServiceInstance
Tags	Choose key-value pairs to tag your ServiceInstance. Use tags to organize, track, or control access for this ServiceInstance. For example, a tag can include CustomerName and license agreement
DataEncrpytion	Encryption ensures that your data and your sensitive information are not accessible by any user or application without a valid key. Encrypting data in transit and data at rest is vital for regulatory compliance. Service always encrypts your data and sensitive information during transit and at rest. You can either choose a customer master key (CMK) managed by Atomtion (default) or bring your own key to encrypt your data and sensitive information.
ServiceTrigger	A trigger causes ServiceInstance to run. You can choose from one of three triggers types: Run on-schedule. Use when you want the ServiceInstance to run automatically at a scheduled frequency. Event-driven. Use when you want the ServiceInstance to run when an external event happens (As described by the Provider) Run on-demand. Use when you want to run the ServiceInstance
SBFOptions	An overall container to hold SBF Options
ProviderInfo	An overall container to hold Provider Information
ProviderName	Name of the Provider
ProviderDescription	A detailed description of the provider
ProviderInteractionName	Provider interaction name. Combination of ProviderName and ProviderInteractionName must be unique
Interaction	An overall container to hold the provider interaction
CustomInteractionAction	Users can specify their own actions than can handle the complete provider interaction. This is useful when there are multiple steps involved in an interaction.
InteractionType	Interaction can be a “READ” or “WRITE” type. Read is for reading from the provider and write is to write back to the provider.
Protocol	An overall container to hold the provider protocol communication information
ProtocolHandlerName	Protocol Handler name - must be one of the A-Stack provided protocol handler name (HttpServerExtensionArgs) or short prefix like - HTTP or OPCUA, etc.
ClientServer	The interaction can be Client i.e. ServiceInstance initiates a connection at specific trigger interval OR Server i.e. provider pushes data to ServiceInstance
ConnectionInfo	An overall container to hold protocol connection information. It includes hostname, port, URI constructed via an array of name-value pairs, Request Message Value to be sent as a payload. Header parameters, Protocol Method (GET/PUT/POST..)
MessageProcessingHandler	An overall container to hold information on how to handle and process the message received from the provider using a given interaction.
MessageProcessingHandlerAction	Action name that can handle the message. Values can be `ASTFW_Service_MessageProcessingHandler_Action` or provided by the user. If it is provided by the user, it must take care of handling the mapping of source and destination fields.	# ASTFW Service Message Processing Handler Action # --- Constants ---------------------------------------------------------------- # --- Arguments ---------------------------------------------------------------- var $cntx = ServiceContext(); var $interactionreponse = $$.argument.valueBy(0); $ --- Process Message -------------------------------------------------------------------- var $standardEvent = MLP.newMl(); $standardEvent.UpdateSaveEntity.Request.entity.entityType = $cntxt.modelName; foreach(value: var $field, in: $cntxt.MessageProcessingHandler.MapFields.Field, from:$cntxt.MessageProcessingHandler.MapFields ): #Validate #Modify var fieldValue = modifiedValue # ------------------------------------------------------------------------------ return = $standardEvent; Standard Event Structure with an example can be found here # Standard Event Structure UpdateSaveEntity: Request: entity: entityType: Environment #or any model body: #Array of desitnation attributes
MapFields	An overall container to hold source to the destination field mapping. From the Studio one can Upload a .csv file with mapped fields - Use a comma-separated values (CSV) file to specify the field mappings. Each line in the CSV file contains the source field name, followed by a comma, followed by the destination field name.
Field	An overall container to hold a single field mapping information
SourceFieldName	Source filed name from the response of a provider interaction
DestinationFieldName	Destination field name pointing to the user Thing Model attribute
Formula	This step is optional. To add a formula that concatenates fields, select two fields from Mapped fields, and then provide a custom formula to chose one of the standard formulas provided. Example: Concatenation.
DataModificationAction	This step is optional. Some of the data modifications are truncate a string value. Users can specify custom action to handle data modification.
Validation	Overall container to hold singe data validation. This can be used to discriminate if the same value is sent by the provider.
ValidationCondition	For Validations, add validations to check whether a field has bad data. For each field, choose the condition that indicates bad data. Service provided conditions are: Value is null Value is 0 Value is missing Value is negative Value contains text Users can provide their validation own condition.
ValidationAction	What action ServiceInstance should take when a field in a record is bad. Users can provide their own validation action

ServiceService Facet

ThingFacet(name: ASTFW_ServiceService_Facet):
  String(name: strActivate, KnownBy: ASTFW_ServiceService_ActivateAction, modifiers: virtual);
  String(name: strDisable,  KnownBy: ASTFW_ServiceService_DisableAction, modifiers: virtual);
  String(name: strListen, KnownBy: ASTFW_ServiceService_ListenAction, modifiers: virtual);
  String(name: strStopListen, KnownBy: ASTFW_ServiceService_StopListenAction, modifiers: virtual);
  String(name: strWrite, KnownBy: ASTFW_ServiceService_WriteAction, modifiers: virtual);
  String(name: strServiceConfigID);
  Action(name: ASTFW_ServiceService_ActivateAction, type: BML, limit: 1):
    onDiscard:  # optional cleanup handler
      logInfo = "ASTFW_ServiceService_ActivateAction discarded";
      discard: closed
    #logInfo = ">>>>> strActivate: arguments:" + string($$.argument);
    logInfo = ">>>>> strActivate: ActionArgument:" + $ActionArgument;
    logInfo = ">>>>> strActivate: ActionUpdate:" + $ActionUpdate;
    logInfo = ">>>>>Activating ServiceContext... sid ..." + $ActionArgument.$sid;
    #Check if already deployed; then do nothing
    $modelsid = $ActionArgument.$sid;
    $ServiceContextID = "ServiceContext_" + $ActionArgument.$sid;
    logInfo = ">>>> ServiceContextID is.." + $ServiceContextID;
    $ServiceConfigID = $ActionArgument.strServiceConfigID.Value;
    logInfo = ">>>> ServiceConfigID is..." + $ServiceConfigID;
    scope (fid: SffFacetAgentFacet):
      DoRequest => Process => Message => Value:
        GetFacetInfo(fid = $ServiceContextID);
    $fafresponse = $$.v2.getResponse();
    if ($fafresponse.Message.Value.FacetAgent.Status == 200):
      logInfo = ">>> Noting to do. Already Activated";
    else:
      scope(fid: SffFacetAgentFacet3):
        DoRequest:
          Process => Message => Value:
            $TL.RuntimeParams.ServiceConfigFacetID = $ServiceContextID;
            @: 'resources/Service/spaces/ServiceActivityTemplate.mqp.cdm'
      scope(fid: Service):
        DoRequest => Process => Message => Value:
          Query:
           Find:
            Service:
              ServiceConfigID(eq = $ServiceConfigID);
      $TL.finddata = $$.v2.getResponse();
      var $T.guard = true;
      if ($finddata.Message.Value.Find.Status == "Success"):
        logInfo = ">>>> Found ServiceConfig";
        $TL.RuntimeParams.ServiceConfigFacetID = $ServiceContextID;
        scope(fid = $ServiceContextID):
          $F.Servicecontext = $finddata.Message.Value.Find.Result;
          $F.context.sid = $modelsid;
          $F.context.EntityType = "Environment"                                 #Fixme: Replace dynamically
          StartServiceContext();
      else:
        logInfo = ">>> ServiceConfig Not found";
    return:
      strActivate: activated

  Action(name: ASTFW_ServiceService_DisableAction, type: BML, limit: 1):
    onDiscard:  # optional cleanup handler
      logInfo = "ASTFW_ServiceService_DisableAction discarded";
      discard: closed
    #logInfo = ">>>>> strActivate: arguments:" + string($$.argument);
    #logInfo = ">>>>> strActivate: ActionArgument:" + $ActionArgument;
    #logInfo = ">>>>> strActivate: ActionUpdate:" + $ActionUpdate;
    $ServiceContextID = "ServiceContext_" + $ActionArgument.$sid;
    loginfo = ">>>> ServiceContextID is ..." + $TL.ServiceContextID;
    scope(fid: SffFacetAgentFacet3):
      DoRequest:
        Process => Message => Value:
           DelFacetInstance(fid = $ServiceContextID)
    return:
      strActivate: disabled

  Action(name: ASTFW_ServiceService_ListenAction, type: BML, limit: 1):
    onDiscard:  # optional cleanup handler
      logInfo = "ASTFW_ServiceService_ListenAction discarded";
      discard: closed
    logInfo = ">>>>> ListenAction: arguments:" + string($$.argument);
    logInfo = ">>>>> ListenAction: ActionArgument:" + $ActionArgument;
    logInfo = ">>>>> ListenAction: ActionUpdate:" + $ActionUpdate;
    logInfo = ">>>> Starting to Listen";
    $ServiceContextID = "ServiceContext_" + $ActionArgument.$sid;
    loginfo = ">>>> ServiceContextID is ..." + $TL.ServiceContextID;
    scope(fid = $ServiceContextID):
      DoRequest => Process => Message => Value:
        startProviderConnection();
    return:
      strActivate: StartListen

  Action(name: ASTFW_ServiceService_StopListenAction, type: BML, limit: 1):
    onDiscard:  # optional cleanup handler
      logInfo = "ASTFW_ServiceService_StopListenAction discarded";
      discard: closed
    #logInfo = ">>>>> strActivate: arguments:" + string($$.argument);
    #logInfo = ">>>>> strActivate: ActionArgument:" + $ActionArgument;
    #logInfo = ">>>>> strActivate: ActionUpdate:" + $ActionUpdate;
    $ServiceContextID = "ServiceContext_" + $ActionArgument.$sid;
    loginfo = ">>>> ServiceContextID is ..." + $TL.ServiceContextID;
    scope(fid = $ServiceContextID):
      DoRequest => Process => Message => Value:
        stopProviderConnection();
    return:
      strActivate: StopListen

  Action(name: ASTFW_ServiceService_WriteAction, type: BML, limit: 1):
    onDiscard:  # optional cleanup handler
      logInfo = "ASTFW_ServiceService_WriteAction discarded";
      discard: closed
    #logInfo = ">>>>> strActivate: arguments:" + string($$.argument);
    #logInfo = ">>>>> strActivate: ActionArgument:" + $ActionArgument;
    #logInfo = ">>>>> strActivate: ActionUpdate:" + $ActionUpdate;
    var $postReq:
      Message => Value:
        DelFacetInstance:
          fid: Service
    scope(fid: SffFacetAgentFacet3):
      DoRequest:
        Target: SffFacetAgentFacet3
    return:
      strActivate: Write

Service Config Meta Data

# Service Meta Model Definition
Namespace(Name: Service):
 Domain(Name: ServiceMetaModel):
  Def(Name: DefKeyValue):
   String(Name: Key)
   String(Name: Value)

  Def(Name: DefTags):
   DefKeyValue(Name: KeyValue, cardinality: 0..m)

  Def(Name: DefDataEncryption):
   String(Name: EncrpytionKeyName)

  Def(Name: DefQueryString):
   DefKeyValue(Name: KeyValue, cardinality: 0..m)

  Def(Name: DefRequestHeaders):
   DefKeyValue(Name: KeyValue, cardinality: 0..m)

  Def(Name: DefConnectionInfo):
   String(Name: Method)
   String(Name: HostName)
   String(Name: URI)
   DefQueryString(Name: QueryString)
   DefRequestHeaders(Name: RequestHeaders)
   String(Name: RequestMessageValue)

  Def(Name: DefValidation):
   String(Name: ValidationCondition)
   String(Name: ValidationAction)

  Def(Name: DefOnSchedule):
   String(Name: ScheduleFrequency)

  Def(Name: DefServiceTrigger):
   Boolean(Name: OnDemand, default: false);
   DefOnSchedule(Name: OnSchedule)

  Def(Name: DefMessageProcessingHandler):
   String(Name: MPHanlderName)
   DefMapFields(Name: MapFields)

  Def(Name: DefProtocol):
   String(Name: ProtocolHandlerName)
   String(Name: ClientServer)
   DefConnectionInfo(Name: ConnectionInfo)
   DefMessageProcessingHandler(Name: MessageProcessingHandler)

  Def(Name: DefInteraction):
   String(Name: CustomInteractionAction)
   String(Name: InteractionType)
   DefProtocol(Name: Protocol)

  Def(Name: DefProviderInfo):
   String(Name: ProviderName)
   String(Name: ProviderDescription)
   String(Name: ProviderInteractionName)
   DefInteraction(Name: Interaction)

  Def(Name: DefField):
   String(Name: SourceFieldName)
   String(Name: DestinationFieldName)
   String(Name: Formula)
   String(Name: DataModification)
   DefValidation(Name: Validation)

  Def(Name: DefMapFields):
   DefField(Name: Field, cardinality: 0..m)

  Def(Name: DefSBFOptions):
   DefProviderInfo(Name: ProviderInfo)

  Def(Name: DefReadSubscriber):
   String(Name: ChannelName)

  Def(Name: DefWriteSubscriber):
   String(Name: ChannelName)
   String(Name: ChannelHandler)
   DefMapFields(Name: MapFields)

  Def(Name: DefSPFOptions):
   String(Name: SPFMode)
   String(Name: SPFTargetFacet)
   DefReadSubscriber(Name: ReadSubscriber)
   DefWriteSubscriber(Name: WriteSubscriber)

  Def(Name: DefService, inherits: DataModel):
   Sid(Name: ServiceConfigID)
   String(Name: ServiceName)
   String(Name: ServiceDescription)
   DefTags(Name: Tags)
   DefDataEncryption(Name: DataEncrpytion)
   DefServiceTrigger(Name: ServiceTrigger)
   DefSBFOptions(Name: SBFOptions)
   DefSPFOptions(Name: SPFOptions)

  DefService(Name: Service)

Service End-2-End Flow

Design Time Flow

Runtime Flow

#
Name: DeviceCloudProvider
Domain(Name: CloudThings):
  ThingModel(Combines: ASTFW_ServiceService_Facet, Name: ExternalEnv):
    Sid(Name: ExtEnvId);
    EnvInfo(Name: EnvDetails, KnownBy: ASTFW_ServiceService.ListenAction);
  Def:
    Include: "[:RuntimeParams.TypesLocation:]/envinfo.types.xml"

Northbound to Soundbound Communication

Service Monitoring and Metering

ServiceRunHistory

Column	Description
ServiceName	ServiceInstance Name whose history is being maintained.
DataCreated	Data time when ServiceInstance was created
Status	The Status column tells you if the ServiceInstance is active, in the draft, or has been suspended. ServiceInstances triggered on a schedule that has been created but not yet activated are in “draft” mode. You can suspend ServiceInstances that is triggered on a schedule or by an external event. Suspended ServiceInstances can be resumed by calling StartListen action again
RunStatus	Run Status column shows the status of the last ServiceInstance run indicating if it was a success or if there were errors.
LastRunDate	Date time when ServiceInstance last run

Querying Service

You can query Service using a ServiceName, ProviderName, or any fields part of the ServiceInstance meta-model.

Dashboard APIs

Service Deployment Options

Service Design Review Comments

Service Initial TImelines

Task Details	Completion Date	Status
Modeling of Service Meta-Model (ServiceInstances, etc) (Venkata)	6/5/20	Done
Service ServiceFacet (Baseer)	6/9/20	Done
Core Logic to implement the execution of one ServiceContext. (Baseer / Venkata) HTTP Connection Perif Connection MessageHandler Standard Event to Model	6/12/20	Done
Refactor Design taking Activity Modeling into consideration (Baseer)	6/12/20	Done
Integration testing for two use cases: Simple Perif Smart Cities use case	6/19/20	Done
Deployment Models	6/16/20
Use cases demo	6/22/20
AMQP, OPC, MQTT Connection handler History Logging Dynamically invoke custom overridden actions Fix Output->Input; Input-Output; (with new A-Stack; Integate with Activity Model?)