TQL Code Coverage

Owned by Baseer Khan
Last updated: Sept 28, 2020 by Long Tran
4 min read


Introduction

Domain Specific Languages are languages created to support a particular set of tasks, as they are performed in a specific domain. Atomiton's TQL is a collection of number of DSL's each supporting specific function. Just like any GPL, in order for DSL to be successful, it must to have advanced tooling support around it.  Ideally we want all all sort of simulators, debuggers, tools to analyze code coverage. Everything built on one platform. 

In this section we describe the simple reporting that A-Stack (runtime for TQL) offers to deduce your TQL Code coverage. The advantage of finding the TQL Code coverage is to avoid or minimize all runtime errors

                

Enabling Code Coverage

In order to enable code coverage report in A-Stack runtime following steps needs to be performed.

Steps to enable code coverage

Step NumberStep NameDescription
1

Add a configuration parameter in sff.local.config.xml file - 

<sff.report.FsCoverage>512</sff.report.FsCoverage>

  • Initial histogram array size will be 512 and array will grow roughly by this increment
  • It is recommended to set this to your typical source file number of lines.
  • Zero or negative values or absent parameter means that code coverage collection is disabled (default)
2

Auto deploy your TQL Code by specifying the package files or directories in 

<sff.local.deploy/>

  • The facility will collect histogram of code usage for each known TQL source file.

  • Note that it will ignore all and any code which does not have file name information in it. For example, packages or facets deployed via submission to TQLEngineManager or FacetAngentFacet over a network connection will not be registered unless they employ code included from some named file.

3Start the A-Stack runtime using -ea flag
4Run your automation / manual tests against the deployed TQL code
  • This is normal execution of your code using your unit / automated test.
5Gracefully shutdown the A-Stack runtime
6Gather the code coverage matrix data from log file
  • Data will be written in your configured log folder - log file in INF

Sample data format:

{
"filename": [0,1,1,0,0],
"Bundle: tqlenginemanager/macros/executeRemoteQuery.mc.xml":[0,0,0,0,0,0,0,0,0,2,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
...
}

Each array element (an integer value) will contain number of hits for the corresponding line of code so array element with index 1 will contain count of all executions of line 1, index 2 will contain hit counter for line 2 etc. Described in detail below.

Format description of code coverage Matrix Data

Code Coverage Matrix Data will be written to your configured log file in the log folder. Please note that all the deployed package file names along with each and every included filename within the package file will be reported as a single JSON object.

{
"filename": [0,1,1,0,0],
"Bundle: tqlenginemanager/macros/executeRemoteQuery.mc.xml":[0,0,0,0,0,0,0,0,0,2,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
...
}

Example of Code Coverage Data

In this section we cover generating the code coverage data for sample HelloTQL Project.

  1. Download and install A-StackPrime runtime - Version 1.3.1 and beyond.
  2. Download and unzip this test project in the A-Stack 
  3. Add 

    <sff.local.deploy>resources/hellotql/deploy</sff.local.deploy> to sff.local.config.xml file

  4. Let's run two sample queries : Endpoint; ws://{host}:{port}/fid-MYGIDI2UAAAH6AAAAEXDEUQIWS
Query to execute
#
Query:
  DeleteAll:
      TempSensor:
        sensorId(ne: "")
  Save(format="version, current"):
    #This will read
    TempSensor:
      Simulation: true
      PerifMsg: $Null()
      Peripheral: serial
      Baudrate: 9600
      InterfacePort: "/dev/cu.usbmodem1411"
      Interface: serial
      Format: ascii
      Operation: receive
      UniqueId: 76522
      Payload: $Null()
      TempValue: $Null() 
Query(Storage: "TqlSubscription"):
  Save:
    TqlSubscription(Label: "TempSensor", sid: "20"):
      Topic: *Atomiton.Sensors.TempSensor.TempValue*


The output will be in this format

JSON output
{
"Bundle: tqlenginemanager/facets/nwChangeDetectionFacets.fs.xml":[0,0,0,0,1,2,1,0,0,1,1,0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,0,2,1,0,0,1,1,0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,2,1,0,0,0,1,0,1,0,0,0,0,1,0,0,0,0,1,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,1,0,0,0,0,1,1],
"resources/atomiton/AStackDevManagment/macros/devmanagmacros.xml":[0,0,2,2,2,2,2],
"Bundle: tqlenginemanager/facets/macroFacet.fs.xml":[0,0,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,1],
"SffFacetAgentFacet.xml":[0,0,0,1,1],
"Bundle: tqlenginemanager/macros/getEngineSettings.mc.xml":[0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],
"Bundle: tqlenginemanager/facets/networkStatus.af.xml":[0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1],
"Bundle: tqlenginemanager/macros/executeSffFacetAgentQuery.mc.xml":[0,0,0,0,0,0,0,2,2],
"SffTqlFacet.xml":[0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,1,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,1,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,2,2,0,0,0,0,0,0,0,0,0,0,0,0,2,1],
"Resource SffTqlFacet.xml":[0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,38,19,0,0,38,19,0,0,0,0,0,0,0,19,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,0,0,0,0,0,0,0,0,0,0,0,0,38,19,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,22,11,0,0,11,11,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,11,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,20,10,0,0,10,0,0,0,0,0,0,10,0,0,0,0,0,0,0,0,0,2,2,0,0,0,0,0,0,0,0,0,0,0,0,20,10],
"resources/hellotql/facets//tempfacetserial.tf.xml":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,10,0,0,0,0,0,0,0,0,10,0,0,0,0,0,0,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,1],
"resources/hellotql/macros/executequery.mc.xml":[0,0,0,0,0,0,0,0,36,0,0,0,18],
"Bundle: tqlenginemanager/facets/logSearchFacet.fs.xml":[0,0,0,0,1],
"Bundle: tqlenginemanager/macros/checkForInternetAvailability.mc.xml":[0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,1,1,1,1],
"Bundle: tqlenginemanager/macros/getMemoryInfo.mc.xml":[0,0,0,0,0,1,0,0,1,1,0,0,0,1,0,1,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1],
"/Users/baseerkhan/iot/atomiton/development/latest/prime/resources/hellotql/deploy/GenericTQLInterface.mqp.xml":[0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,0,0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,0,0,0,0,0,0,0,2,0,2,0,0,0,0,0,2,0,0,0,0,0,0,0,2,0,4,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1],
"Bundle: tqlenginemanager/macros/executeQuery.mc.xml":[0,0,0,0,0,0,0,0,20,10],
"Bundle: tqlenginemanager/facets/upgradeService.fs.xml":[0,0,0,0,1,1,2,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,1,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1],
"Inline":[0,2],
"Bundle: tqlenginemanager/macros/executeRemoteQuery.mc.xml":[0,0,0,0,0,0,0,0,0,2,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],
"/sff/tqlenginemanager/deploy/tqlenginemanager.mqp.xml":[0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,1,0,0,0,0,0,1,0,0,0,0,1,0,0,0,0,1,0,0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1],
"SffTqlPolicyFacet.xml":[0,0,0,0,0,0,0,0,0,2],
"Bundle: tqlenginemanager/macros/scheduleJob.mc.xml":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2],
"resources/hellotql/macros/setsimulatedtemp.mc.xml":[0,0,0,0,9,0,0,0,9],
"/sff/tqlenginemanager/facets/appStartupInitialization.fs.xml":[0,1,0,0,2,1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,1,0,0,0,0,0,0,0,1,1,1],
"Bundle: tqlenginemanager/facets/loggerFacet.fs.xml":[0,0,0,0,0,1,0,0,0,0,0,0,0,1,1,1],
"Bundle: tqlenginemanager/facets/engineActivationFacet.fs.xml":[0,0,0,0,0,1,2,1,0,0,1,1,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,1,0,1,0,0,1,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],
"Bundle: tqlenginemanager/macros/checkForEngineUpgrade.mc.xml":[0,0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1],
"resources/hellotql/macros/schedulejob.mc.xml":[0,0,0,0,0,0,0,0,0,0,0,1,1,1,0,0,0,0,9],
"Bundle: tqlenginemanager/facets/engineMonitoringActivation.fs.xml":[0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1]
}

Determine TQL Code Coverage

Prerequisites

One of the prerequisite is to determine the number of source files, executable code line number for each of the source file. It would be ideal to generate the source code line numbers in a same array format as reported by A-Stack code coverage.

Pseudo code to generate source file line number array

  1. Input to the pre-processing code can be parent directory name under which all the source code files are present
  2. Make a list of all the source code file names
  3. Iterate through each file in the list from (2) and create a array whose index is the line number and value is 1 if the line is a TQL instruction (example, Process, JavaScript, Workflow, Invoke, etc)
  4. Output the array in the format [ 0,0,0,1,1,1,0,1 ]

Calculating code coverage

Once you have prerequisite data, you can take the code coverage data generated by A-Stack and determine the coverage. The table below will help you give examples - 

Source File

Lines # of executable instruction

(Found from prerequisite)

Line Number reported in array with value > 0Number of non-zero value source lines (say, sl)Number of lines covered by tests (say, cl)

Code Coverage

cc = (100*cl)/sl

resources/hellotql/macros/executequery.mc.xml[0,0,0,0,0,0,0,0,1,0,0,0,1][0,0,0,0,0,0,0,0,36,0,0,0,18]22100%
list all your files here....

..
TOTAL...

Total coverage value