12. Bulk initialization of Things and Data

Key Concepts

	Related docs
For loop
Show query

Background

Device Management

Bulk initialization of Things is closely tied to the device management aspect of IoT Platform. The devices layer is the most important component of an IoT solution. A mature IoT platform comes with comprehensive device management features that let customers:

On-board existing and new devices with rich metadata.
Identical devices that share the same metadata are grouped together. This feature makes it possible to search devices based on their capabilities.
The platform also provides per-device authentication and authorization to enforce enhanced security.
It makes it easy to define which devices can connect, send, and receive messages.
Devices can be easily blacklisted and whitelisted through declarative policies.

TQLEngine platform provide mechanisms to easily address above device management requirements.

Bulk Initialization Design Pattern

At the time of writing this tutorial Atomiton TQLEngine does not provide specific language constructs to perform bulk initialization of Things or Data. But the existing Atomic domain languages capabilities can be exploited to achieve bulk initialization requirements. The bulk initialization design pattern can be broken down into two steps:

Bulk Initialization Steps:

Step #

Name

Description

Output

Implementation Flow

1

Gather Metadata

Input to gathering of metadata step can be:

Manual entires
Code that generates metadata
Reading metadata from external device management source

The output of gathering metadata step could be:

Generated CSV, JSON Files
Generated Create / Save TQL Query statement

@TQL Macro, AppModel, AppFacet
Gather and Initialize can be combined into same flow.
Decoupled implementation from TQLEngine
(Example: NodeJS generating series of JSON files)

2

Initialize

Initialize step includes:

Load the data into TQLEngine.
Metadata gathered from step-1 serves as input.
This steps has to be repeatable.

The output of initialize steps are:

Data and Things loaded into TQLEngine.
May or may not note trigger actions.

@TQL Query
Can be integrated into Gather Metadata step.

Greenhouse UseCase

Please refer to 8. Simulating sensors, actuators, and devices for the description of Greenhouse usecase. The data initalization requirement for this case is shown below

Greenhouse Configurator Application Flow

Greenhouse Configuration Requirements

Ability to add greenhouses on-the-fly
Each greenhouse must be able to dynamically generate number of lanes, zones per lane
Add Crop type per lane

Bulk Initialization Implementation

Import baseline Greenhouse Data and Thing Models

Create a new Project in TQLStudio by importing content from Baseline models.

Pipeline Macros

Execute Internal Query

Pipeline Macros

<Macro Name="executeQuery">
  <Argument>
    <QueryString>
      <Query/>
    </QueryString>
  </Argument>
  <Result>
    <OnRequest Target="[:RuntimeParams.FacetIDName:]" Disable="CMD_SERVER">
      <Process Return="CMD_NOP">
        <Message>
          <Value>
            [:$Macro.Argument.QueryString:]
          </Value>
        </Message>
      </Process>
    </OnRequest>
  </Result>
</Macro>

Main CreateGreenhouse Macro

Main Create Greenhouse Macro

<Macro Name="CreateGreenhouse">
  <Argument>
    <Greenhouses></Greenhouses>
  </Argument>
  <Result>
    <SetLocalData key="Greenhouses">
      <Value>
        <Include>$Macro.Argument.Greenhouses</Include>
      </Value>
    </SetLocalData>
    <for each="record" in="Greenhouse" from="$LocalData.Greenhouses" using="$ProcessData">
       <JavaScript>
        var NumberOfLanes = [:$ProcessData.record.NumberOfLanes:];
        var ZonesPerLane = [:$ProcessData.record.ZonesPerLane:];
        var NumberOfZones = NumberOfLanes * ZonesPerLane;
        sffContext.execute("SetLocalData","key","NumberOfZones","value",NumberOfZones);
       </JavaScript>
       <!-- Create Query for Greenhouse -->
          <SetContextData key="FirstLane">
           <Value>Yes</Value>
          </SetContextData>
          <For each="laneRecord" in="Lane" from="$ProcessData.record.Lanes" using="$LocalData">
            <!-- Add Lanes -->
            <addLanes>
              <GreenhouseID>[:$ContextData.GreenhouseID:]</GreenhouseID>
              <LaneName>[:$LocalData.laneRecord.LaneName:]</LaneName>
              <LaneLength>110</LaneLength>
              <LaneWidth>40</LaneWidth>
              <ZoneInLane>[:$ProcessData.record.ZonesPerLane:]</ZoneInLane>
              <BorderLane>No</BorderLane>
            </addLanes>
            <if condition="$Response.Message.Value/Create/Status eq 'Success'">
              <then>
                <SetLocalData key="LaneID">
                  <Value>[:$Response.Message.Value.Create.Lane.LaneID:]</Value>
                </SetLocalData>
                <loadZoneModels>
                  <LaneID>[:$LocalData.LaneID:]</LaneID>
                  <ZonesPerLane>[:$ProcessData.record.ZonesPerLane:]</ZonesPerLane>
                  <CropType>[:$LocalData.laneRecord.CropType:]</CropType>
                  <FirstLane>[:$ContextData.FirstLane:]</FirstLane>
                </loadZoneModels>
              </then>
            </if>
            <SetContextData key="FirstLane">
              <Value>No</Value>
            </SetContextData>
          </For>
        </then>
      </if>
    </for>
  </Result>
</Macro>

Load Zone Models

Data is randomized and generated

Load Zone Models

<Macro Name="loadZoneModels">
	<Argument>
        <ZonesPerLane></ZonesPerLane>
        <LaneID></LaneID>
        <CropType></CropType>
        <FirstLane></FirstLane>
	</Argument>
	<Result>
      <JavaScript>
            var ZonesPerLane = 0;
            if(!isNaN([:$Macro.Argument.ZonesPerLane:])){
                ZonesPerLane = [:$Macro.Argument.ZonesPerLane:];
            }
      		var loadContent = ListMap.static.newInstance();
      		for (var row=1; row &lt;= ZonesPerLane; row++) {
      			var svi = loadContent.instanceAdd("addZone");
      			svi.put("ZoneName", "Zone"+row);
      			svi.put("LaneID", "[:$Macro.Argument.LaneID:]");
      			svi.put("ZoneLength", 10);
      			svi.put("ZoneWidth", 10);
      			svi.put("Vertex1", 5);
      			svi.put("Vertex2", 2);
      			svi.put("Vertex3", 15);
      			svi.put("Vertex4", 12);
      			svi.put("EndZone", "No");
                svi.put("ZoneAvgSM", 45);
                svi.put("CropType", "[:$Macro.Argument.CropType:]");
                if("[:$Macro.Argument.FirstLane:]" == "Yes" &amp;&amp; row==1){
                    svi.put("Live", "Yes");
                }else{
                    svi.put("Live", "No");
                }
      			svi.put("GridOnLength", 3);
      			svi.put("GridOnWidth", 3);
      			svi.put("GridCount", 9);
      			svi.put("GridLength", 3.33);
      			svi.put("GridWidth", 3.33);
      			svi.put("ZoneTemperature", "0");
      			svi.put("ZoneHumidity", "0");
                svi.put("ZoneAvgLight", "0");
                svi.put("NoOfPlants", Math.floor((Math.random() * 100) + 1));
      		}
      		sffLog.info("load Zone Content ======== "+loadContent);
      		loadContent;
      </JavaScript>
	</Result>
</Macro>

Queries

Let's start writing some queries to create and read the data.

Reset All Data

Delete Greenhouse

<DeleteAllModelsData/>

Create Greenhouse

Macro can be called directly

Create Greenhouse

<CreateGreenhouse>
  <Greenhouses>
	<Greenhouse>
		<GreenHouseName>GreenHouse-1</GreenHouseName>
		<Location>
			<latitude>1.11</latitude>
			<longitude>1.12</longitude>
		</Location>
		<ZonesPerLane>5</ZonesPerLane>
		<NumberOfLanes>2</NumberOfLanes>
		<Lanes>
			<Lane>
				<LaneName>Lane-1</LaneName>
				<CropType>Tomato</CropType>
			</Lane>
          	<Lane>
				<LaneName>Lane-2</LaneName>
				<CropType>Tomato</CropType>
			</Lane>
		</Lanes>
	</Greenhouse>
  </Greenhouses>
</CreateGreenhouse>

Show Query

Show Query

<Query>
  <Show>
    <Entity inherits="DataModel" not.qname="*.TqlSystem.*"/>
  </Show>
</Query>

Find Greenhouse / TempSensor

Find Greenhouse

<Query>
  <Find>
    <Greenhouse>
      <GreenhouseID ne=""/>
    </Greenhouse>
  </Find>
</Query>
 
<!-- Custom container -->
<Query as="Khans.GreenDen">
  <Find>
    <Greenhouse>
      <GreenhouseID ne=""/>
    </Greenhouse>
  </Find>
</Query>
 
<!-- Filter the result -->
<Query as="Khans.GreenDen">
  <Find only="Greenhouse:ExternalEnv">
    <Greenhouse>
      <GreenhouseID ne=""/>
    </Greenhouse>
  </Find>
</Query>
 
<!-- Filter the result -->
<Query as="$none:Greenhouse.Names"> <!-- What happens if we don't use $none -->
 <Find only="Greenhouse:GreenhouseName,Greenhouse:ZoneCount">
   <Greenhouse>
     <GreenhouseID ne=""/>     
   </Greenhouse>
 </Find>
</Query>
 
<!-- Filter the result -->
<Query as="$none:Greenhouse.Names"> <!-- What happens if we don't use $none -->
 <Find only="Greenhouse:GreenhouseName,Greenhouse:ZoneCount" as="$none:$none:$none"> <!-- Knock off Containers -->
   <Greenhouse>
     <GreenhouseID ne=""/>     
   </Greenhouse>
 </Find>
</Query>

Complete Model

You can import this model into your engine from this location