Nerve Data Services

The Nerve Data Services are a collection of services and interfaces that allow to collect, store, analyze, visualize and distribute data. These services and interfaces are available on the Nerve Blue node and in the Management System. Using Nerve Data Services, data can be collected at high speeds on the node and stored or visualized locally at the Node or centrally in the Management System.

Elements of Nerve Data Services

All of the above elements are individually configurable. This allows for an application in a large number of scenarios and use cases. However, this also implies deeper knowledge of the elements and their possible interactions. Further explanations of the elements can be found in separate chapters. Find a high-level graphical overview below.

Supported protocols

Model workflow

The following section gives an overview of a typical work flow, presenting what can be done with the Nerve Data Services on a high level:

• Setting up data sources
• Configuring the local Gateway on the node to collect data and defining where to distribute it
• Configuring Grafana on the node for local data visualization
• Configuring the central Gateway in the Management System to collect data from different nodes
• Configuring Grafana in the Management System for central data visualization
• Configuring Grafana to visualize data
• Implementing an analytics app and configuring the local Gateway again to send data to the analytics app
• Reconfiguring Grafana to visualize the result of the processing performed by the analytics app

The paragraphs below give short explanations to each step.

Ingesting data on the node

CODESYS can be used to translate fieldbus protocols to OPC UA Server. The local Gateway can then be configured to collect data from CODESYS and distribute it to various destinations. Data sources that support protocols which are also native to the local gateway can be linked there directly and do not need CODESYS.

Distributing data on the node

Data is distributed by the local Gateway to user defined consumers. Unless the respective protocol of a Gateway output defines a specific data format, all data received by the Gateway is normalized to the Nerve Data Services data format. Typical consumers of data are the local data storage on the node or the central data storage in the Management System, third party MQTT brokers, applications or providing the data as OPC UA Server. To enable reuse of local Gateway configurations across multiple nodes, keywords can be used to abstract node specific information from the configuration.

Storage on the node

Data is stored on the node in a TimescaleDB database. Any application running on the node can access data stored in there. For application development, a Python API is provided in the Data SDK that simplifies reading and writing data. A time window can be configured for how long data shall be kept before deleting it.

Visualization on the node

Visualization is available via Grafana which is pre-configured to allow access to data stored in the local data storage. It is also possible to configure access from Grafana to other user defined data sources without any restrictions, meaning that data in an influxDB that is installed on the node can also be visualized.

Using Data in analytics

A Python SDK is available for creating custom analytics applications. An API is available that simplifies reading data from the local storage or receiving it directly from the Gateway when it arrives there. The API also allows to write back analytics results to the local storage for visualization or send it to an MQTT broker to distribute it to other consumers. It is also possible to use third party applications such as NodeRed or Crosser for analytics together with the Nerve Data Services.

Storing and visualizing in the Management System

Storage and visualization in the Management System has a similar structure technologically speaking but offer limited configuration possibilities as the network is static and there are no option to include workloads in the Management System.

Data received on the Node can be forwarded to the Management for visualization. To allow this, the local gateway and the central gateway as well as the central visualization must be configured. Keywords are in place to simplify the configuration process. Each node has a separate database in the Central Data Storage.

Nerve Data Services data format

Inputs and Outputs at the local and central gateway using a protocol that does not define a data format (i.e. MQTT) send or receive data in JSON format. The same applies to the Python API in the Analytics Data SDK. As a result, data within the Nerve Data Services is normalized to this format.

The JSON schema below describes the data format in more detail. The JSON schema is also used to validate data upon reception. If a data frame is received that does not comply with the schema, it is silently dropped by the Nerve Data Services.

{
    "$schema": "http://json-schema.org/draft/2019-09/schema#",
    "$id": "https://nerve.cloud/dp/dp_data_model.schema.json",
    "title": "Nerve Data Services JSON data model schema",
    "description": "Schema that represents default data model that Nerve Data Services use.",
    "type": "object",
    "properties": {
        "variables": {
            "type": "object",
            "additionalProperties": {
                "anyOf": [
                    {
                        "type": [ "boolean", "number", "string" ]
                    },
                    {
                        "type": "array",
                        "items": {
                            "type": "boolean"
                        }
                    },
                    {
                        "type": "array",
                        "items": {
                            "type": "number"
                        }
                    },
                    {
                        "type": "array",
                        "items": {
                            "type": "string"
                        }
                    }
                ]
            }
        },
        "timestamp": {
            "type": [ "number", "string", "array" ],
            "items": {
                "type": [ "number", "string" ]
            }
        }
    },
    "required": [ "variables" ],
    "additionalProperties": false
}

The schema defines and allows only two properties in the root object: timestamp and variables. Timestamp is optional, and can be a single value or an array of values. Variables is an object containing arbitrary number of properties, which can be single values or arrays of values.

Data Services UI

An instance of the data Gateway is present on both the node and on the cloud. However, the instances are configured separately. The supervisor UI is identical in both instances, in the Local UI on the node and in the Management System.

Accessing the Data Services UI on the node

To access the Data Services UI on the node, access the Local UI. Connecting to the Local UI depends on the Nerve Device. Follow the link to the Local UI according to the used Nerve Device in the table below:

Select Data in the navigation on the left and select CONFIGURATION on the right to access the Data Services UI.

Accessing the Data Services UI in the Management System

By default, the permission to access to Data Services is not granted. Users that have the permission to create user roles can enable the Data Services entry in the Management System navigation by adding the UI_NAV_DATAPATH:VIEW permission to a user role. A Data Services user role is also defined in the Management System by default that can be added to a user. Refer to Users for information on how to create new users and to Roles and permissions for information on how to add a new role.

1. Log in to the Management System. Make sure the user has a role with the correct permission assigned.
2. Select Data in the navigation on the left.

3. Select CONFIGURATION to access the Data Services UI.

   

How to export a log file

Every element of the Data Services is producing log files locally and centrally that can be viewed in the Logs tab. These logs can also be downloaded through the Data Services UI.

1. Select a Data Services element in the navigation on the left.

2. Select the Logs tab on the right.

   

   Note

   Note that the visualization element does not have a Logs tab. Instead the logs are displayed on the first level and can be downloaded by selecting Download.

3. Select Download to export a LOG file.