Nerve Data Services Gateway

The Gateway is the central application of the Data Services. Its purpose is receiving data from a source via a certain protocol on an input interface and forwarding it to a destination using a different protocol on an output interface. This behavior is reflected in the user configuration as well. In general, the configuration file consists of inputs, outputs and connections between them:

• Inputs are collection interfaces where data is received.
• Outputs are providing interfaces where data can be received from.
• Connections are logical links between inputs and outputs.

Each input and output provides connectors where a connection can be attached to. A connector is a subset of the data available at an input. This subset is forwarded to the connector of an output where it is distributed further to the data receiver. On an abstract level, this concept is illustrated in the figure below.

Inputs and outputs hold general configuration data for the protocol they implement. This can be a URL to a server, a port number or something related to timing. A connector of an input or output holds more specific information on how to get or where to send data, for example the node IDs of an OPC UA Server an OPC UA Client input connects to, or the name of a topic for an MQTT Publisher. Each input and output can provide multiple connectors at a time. Data from an input connector can also be connected to multiple output connectors at the same time to distribute data to multiple data consumers. The figure below shows a more specific example of that concept.

Data Gateway UI

The Data Gateway UI is the central page of the Data Services and can be reached by selecting Data > Gateway in the navigation on the left in either Local UI or Management System. Gateway configurations are applied through the interface in the middle, either through the graphical configuration tool or with a pre-written JSON file that is uploaded. When accessing this page, the panes of the configuration tool displays the last deployed configuration.

Item	Description
Configuration and Logs tabs (1)	The Configuration tab is displayed by default, displaying the graphical configuration tool. The latest logs can be viewed within the Logs tab. The entirety of the logs can be exported into a file using the Download button in this tab.
Edit configuration (2)	Select this button to enable editing mode and enter a configuration into the editor to configure the Gateway. In editing mode, the configuration with the graphical configuration tool is enabled, as well as options to import or export JSON configurations. After selecting the edit button, these buttons appear instead: The following descriptions describe the buttons from left to right. Deploy Select this to apply the current configuration to the Gateway. The button becomes active once changes have been performed. Import Select this to upload a Gateway configuration as a JSON file from the workstation into the Gateway. Once the file is uploaded, the graphical configuration tool will reflect the configuration from the JSON file. Export Select this to download the currently deployed Gateway configuration as a JSON file. Discard current changes Discard all the changes that have been done in editing mode. This will revert everything to the state before the edit button was selected. Import certificates Import certificates and keys from a workstation to be used by the Gateway. When configuring inputs and outputs, only the name of the certificates and key files is required. Note the following: All certificates are stored in the same directory on the node. Therefore, certificates must have unique names when imported. Certificates need to be in DER format. To exit editing mode, select the arrow on the left.
Inputs (3)	Configured inputs are displayed here. Details can be viewed by selecting the magnifying glass icon. Editing options appear when editing mode is active.
Connections (4)	Configured connections are displayed here. Details can be viewed by selecting the magnifying glass icon. Editing options appear when editing mode is active.
Outputs (5)	Configured outputs are displayed here. Details can be viewed by selecting the magnifying glass icon. Editing options appear when editing mode is active.

Applying a configuration to the Gateway

The configuration of the Gateway is applied through the graphical configuration tool in the Gateway section of the Data Services UI. The current configuration of the Gateway is displayed in the separate input, connection and output panes if there is a configuration present in the Gateway. The panes will be empty on initial startup. A new configuration can be entered using the GUI or imported from an existing JSON file. After applying a configuration, check the Logs tab for any warnings or errors.

The basic structure of the configuration consist of a list of inputs such as an OPC UA server or an MQTT subscriber, outputs such as a Timescale database or an MQTT publisher, and connections between these inputs and outputs. Refer to Gateway configuration parameter descriptions below for the exact syntax. While studying the syntax is required to write a Gateway configuration in JSON format, note that the syntax is also helpful if the graphical configuration tool is used.

The examples in the Examples section show step by step applications in common use cases.

Applying a configuration through the GUI

The graphical configuration tool offers a guided way to configure the Gateway. Required fields are marked with a red asterisk. The recommended order is defining an input, defining an output and then defining the connections last.

As a demonstration, the instructions below show how to recreate the pre-written JSON configuration file from the OPC UA Server to cloud for visualization example with the graphical configuration tool.

First, inputs need to be configured. This input will have two connectors configured that are required for connecting to two database outputs.

1. Select the Edit configuration icon on the right.

   

2. Select the plus icon next to Inputs.

   

3. Select OPC UA Client from the drop-down menu. A list of parameters will appear.

   

4. Enter the following information:

   Setting	Value
   Name	Enter any name, for example OPC UA Client.
   Server URL	opc.tcp://localhost:4848
   Polling interval in ms	1000
   Connector 1	Select the plus icon and enter the following information: Name Enter a name for the connector. This example uses temperature. Access type polling Nodes Select the plus icon and enter the following information: Node ID ns=2;i=2
   Connector 2	Select the plus icon and enter the following information: Name Enter a name for the connector. This example uses humidity. Access type subscription Publishing interval at server in ms 1000 Sampling interval at server in ms 1000 Nodes Select the plus icon and enter the following information: Node ID ns=2;i=4

5. Select Save.

   

The graphical configuration tool now reflects the configuration above, showing an OPC UA Client input. Next, it is required to define an output before defining connections. In the OPC UA Server to cloud example, two outputs are required, as temperature data from a demo sensor will be saved in the local NerveDB for visualization while humidity data from the same demo sensor will be saved in the central NerveDB in the Management System and be visualized there.

1. Select the plus icon next to Outputs.

   

2. Select Nerve database from the drop-down menu. A list of parameters will appear.

3. Enter the following information:

   Setting	Value
   Name	Enter any name, for example NerveDB local.
   Location	LOCAL
   Connector	Select the plus icon and enter the following information: Name Enter a name for the connector. This example uses NerveDB local connector. Table name Enter a name for the data table. If no table name is provided, the name of the connection will be used as a table name. Note that this field is optional when using the local NerveDB.

4. Select Save.

   

5. Select the plus icon next to Outputs to define a second output.

6. Select Nerve database from the drop-down menu. A list of parameters will appear.

7. Enter the following information:

   Setting	Value
   Name	Enter any name, for example NerveDB central.
   Location	CENTRAL
   Connector	Select the plus icon and enter the following information: Name Enter a name for the connector. This example uses NerveDB central connector. Table name Enter a name for the data table. This example uses Demo sensor to central NerveDB.

8. Select Save.

The graphical configuration tool now reflects the configuration above, showing two database outputs in addition to the OPC UA Client input. With inputs, outputs and their respective connectors defined, connections can now be set up to finalize the Gateway configuration. In this example temperature data is sent to the local NerveDB and humidity data is sent to the central NerveDB.

1. Select the plus icon next to Connections.

   

2. Enter a connection name, for example demo sensor to local NerveDB.

3. Select the following information from the drop-down menus under Input:

   Setting	Value
   Input	Select the OPC UA Client input, named OPC UA Client in this example.
   Connector	Select the first connector, named [0]: temperature in this example.

4. Select the following information from the drop-down menus under Output:

   Setting	Value
   Output	Select the local NerveDB output, named NerveDB local in this example.
   Connector	Select the connector, named [0]: NerveDB local connector in this example.

5. Select Save.

   

6. Select the plus icon next to Connections to add the second connection.

7. Enter a connection name, for example demo sensor to central NerveDB.

8. Select the following information from the drop-down menus under Input:

   Setting	Value
   Input	Select the OPC UA Client input, named OPC UA Client in this example.
   Connector	Select the second connector, named [1]: humidity in this example.

9. Select the following information from the drop-down menus under Output:

   Setting	Value
   Output	Select the NerveDB output, named NerveDB central in this example.
   Connector	Select the connector, named [0]: NerveDB central connector in this example.

10. Select Save.

    

11. Select the Deploy button. A success message pops up in the upper-right corner.

    

    Note

    In case of an already existing configuration, the Gateway might give a warning about which connections will be affected by a short data flow interruption. All other connections will continue to operate and stay unaffected.

After deploying the configuration, the graphical configuration tool is still in editing mode. Select the arrow on the left side to exit editing mode.

Uploading an existing JSON configuration file

If writing a JSON configuration file is the preferred method of configuring the Gateway, a pre-written JSON file can be uploaded and applied to the Gateway. This method is also useful when following any of the examples.

1. Log in to the Local UI.
2. Expand Data > Gateway in the navigation on the left.

3. Select the edit button on the right.

   

4. Select the Import button.

   

5. Add a pre-written JSON configuration file in the file browser.

6. Select the Deploy button. A success message pops up in the upper-right corner.

   

The configuration is now deployed. The graphical configuration tool now reflects the settings from the JSON configuration file. Edit the elements in each pane by selecting the edit button in each element. Exit editing mode by selecting the arrow on the left.

For writing JSON configuration files, have a look at the Nerve Data Services Data Format below and consult the list of parameters for more information on all input, output and connection parameters.

Writing a Gateway configuration file in the Nerve Data Services data format

Inputs and Outputs at the local Gateway, using a protocol that does not define a data format (i.e. MQTT), send or receive data in JSON format. The same applies to the Python API in the Analytics Data SDK. As a result, data within the Nerve Data Services is normalized to this format.

The JSON schema below describes the data format in more detail. The JSON schema is also used to validate data upon reception. If a data frame is received that does not comply with the schema, it is silently dropped by the Nerve Data Services.

{
    "$schema": "http://json-schema.org/draft/2019-09/schema#",
    "$id": "https://nerve.cloud/dp/dp_data_model.schema.json",
    "title": "Nerve Data Services JSON data model schema",
    "description": "Schema that represents default data model that Nerve Data Services use.",
    "type": "object",
    "properties": {
        "variables": {
            "type": "object",
            "additionalProperties": {
                "anyOf": [
                    {
                        "type": [ "boolean", "number", "string" ]
                    },
                    {
                        "type": "array",
                        "items": {
                            "type": "boolean"
                        }
                    },
                    {
                        "type": "array",
                        "items": {
                            "type": "number"
                        }
                    },
                    {
                        "type": "array",
                        "items": {
                            "type": "string"
                        }
                    }
                ]
            }
        },
        "timestamp": {
            "type": [ "number", "string", "array" ],
            "items": {
                "type": [ "number", "string" ]
            }
        }
    },
    "required": [ "variables" ],
    "additionalProperties": false
}

The schema defines and allows only two properties in the root object: timestamp and variables. Timestamp is optional, and can be a single value or an array of values. Variables is an object containing arbitrary number of properties, which can be single values or arrays of values.

Custom JSON format

In order to be able to fetch values from all kinds of JSON messages, the Nerve Data Services Gateway supports custom JSON data formats. The value of a field from a JSON message can be fetched using a so-called JSON path. A JSON path is comparable to a file path in a Unix-like file system. As an example, if a file is located in the following file structure:

• Documents
  • Personal
    • journal.txt

its file path would be: /Documents/Personal/journal.txt. Based on this analogy, a JSON field from the following JSON:

{
    "Values": {
        "Temperature": 20
    }
}

would have the following JSON path: .Values.Temperature. Arrays are represented with square brackets:

{
    "Values": {
        "Temperature": [ 20, 30, 40 ]
    }
}

The third value of the Temperature field would be represented the following way: .Values.Temperature[2]

Gateway configuration for custom JSON formats

The fields name and path of the variables array objects inside of the connectors object are used to construct the JSON path. The full JSON path is always constructed by adding name to path:

The example above will construct the following JSON path: .measurement.temperature

• The variable name through the rest of the Gateway will always be just the value of the name field. (temperature in the example above).
• If name is omitted, the value of path will become the name of the variable.
• If path is omitted, the value of path will be the default path .variables. This allows the Gateway to use the default Data Services data model JSON schema.
• To define a path for timestamps, a special timestamp object is used in the root of the input. The only field in the timestamp object is path, which is used to specify the path where to fetch timestamp values.
• This object is optional, and if omitted, the Gateway will look for timestamps under the default JSON path, .timestamp. This also allows the Gateway to use the default Data Services data model JSON schema.

Timestamp JSON path example

The variable expansion operator [*] cannot be used in a timestamp path. All additional requirements from the section above still apply.

Extended JSON paths

The Gateway extends the JSON paths explained above with two new operators for fetching values, in order to reduce the need to write JSON paths for recurring fields. The following data is used for both expansion examples below:

{
    "machineB": {
        "sensor1": [
            {
                "temperature": 10,
                "humidity": 51
            },
            {
                "temperature": 20,
                "humidity": 52
            },
            {
                "temperature": 30,
                "humidity": 53
            }
        ]
    }


}

Operator [] - List expansion

The list expansion is an operator used to fetch multiple values of the same variable. It allows processing large chunks of data in a simple way.

The data above, when received with the following configuration:

generates 3 entries on any Gateway output.

Operator [*] - Variable expansion

The variable expansion is an operator used to fetch multiple values of a variable as separate variables. It allows processing large chunks of data in a compact way.

The data above, when received with the following configuration:

generates all data in one entry. Note that in the configuration above, only the operator was changed from [] to [*].

An additional parameter is required when working with the variable expansion, maxValues. maxValues is required for outputs that need to know the exact number of data before actually receiving it (TimescaleDB, for example). If more values are received than specified in maxValues, the Gateway will simply ignore them. If less values than specified are received, the Gateway will write either the last known value at that position or a default value based on the variable type. In the example above, humidity-3 is set to 0 because of this.

Gateway configuration parameter descriptions

The following list gives more information on the parameters that can be set when configuring the Gateway. Parameters are listed by available inputs and outputs in the list below. The description of each parameter also includes information on additional parameters. The following inputs and outputs are included in the list:

Inputs

In the graphical configuration tool, mandatory fields are marked with a red asterisk.

OPC UA Client

MQTT Subscriber

ZeroMQ Subscriber

OPC UA PubSub Subscriber

S7 Client

Modbus Client

Kafka Consumer

Outputs

In the graphical configuration tool, mandatory fields are marked with a red asterisk.

TimescaleDB

InfluxDB

MQTT Publisher

OPC UA PubSub Publisher

OPC UA Server

ZeroMQ Publisher

Azure IoT Hub Device

NerveDB

Kafka Producer

Connections

Gateway configuration file keywords

Parts of the Gateway configuration are eased by the use of keywords. All keywords must be put in angled brackets, for example <SN>. To be able to use keywords. the node must be registered in the Management System.

Refer to the list below for keywords that can be used with usage examples right after. Note that keywords can also be used in the graphical configuration tool.

Examples of keyword usage

LOCAL

{
    "type": "DB_TIMESCALE",
    "name": "timescaledb_0",
    "url": "<LOCAL>"
}

SN and SID

{
    "type": "MQTT_SUBSCRIBER",
    "serverUrl": "tcp://localhost:1883",
    "username": "<SN>",
    "password": "<SID>",
    "keepAliveInterval_s": 20,
    "qos": 1,
    "cleanSession": true,
    "connectors": [
        {
            "topic": "<SN>",
            "variables": [
                {
                    "name": "temperature",
                    "type": "uint64"
                }
            ]
        }
    ]
}

How to export a log file

Every element of the Data Services is producing log files locally and centrally that can be viewed in the Logs tab. These logs can also be downloaded through the Data Services UI.

1. Select Gateway in the navigation on the left.

2. Select the Logs tab.

   

3. Select Download to export a LOG file.