Data analytics

The analytics element of the Nerve Data Services is primarily used for processing and analysis of data collected by the Gateway, and provides an easy way to collect and store that data. It consist of a Software Development Kit (SDK) and an Application Programming Interface (API) both written in Python. Analytics are supported on all operating systems based on Debian Buster (tested on Ubuntu 18.04 LTS and Ubuntu 20.04 LTS). The recommended Python version to use when writing analytics apps is 3.7 or later.

The release package contains:

API documentation
an installation script
the installation script README file
the SDK python wheel (containing the API python wheel)

The Nerve Data SDK can be downloaded from the Nerve Software Center.

Software Development Kit

Note

The Software Development Kit can only be used on Linux and requires sudo, python and pip for the installation.

The Analytics Data SDK is used to create, build, push and provision analytics apps as Docker images. These functions will be explained in detail in the following sections. The SDK is installed using the installation script. Navigate to the folder containing the installation script and execute the following command:

source ./nerve_dp_analytics_install.sh

Follow the instructions in the installation script. It will ask for confirmation before installing the following requirements:

Docker
- docker-ce
- curl
- apt-transport-https
- ca-certificates
- gnupg-agent
- software-properties-common
Miniconda

It will also add an official Docker GPG key to the system and the official Docker stable repository to apt repositories. The Conda environment nerve-dp-analytics will be created and activated upon installation of the Analytics Data SDK and API wheels, as well as all their dependencies.

Note

Before working with the SDK, make sure that the Conda environment is active. If the Conda environment is active it will be displayed in parentheses in front. Activate the Conda environment by entering the following command:

source miniconda/bin/activate <environmentname>

The Conda environment automatically deactivates after a restart.

Create command

The create command is used to initialize the environment for analytics app development.

Item	Description
Usage	`nerve-analytics create [-h] [-c [CLONE]] [-t {minimal,slim,standard,intel,custom}] app_name path`
Positional arguments	`app_name` The application name. `path` Path where to create the application.
Optional arguments	`-h` or `--help` Use this argument to show the help message in the command line. `-c [CLONE]` or `--clone [CLONE]` Clones base Nerve analytics environment into a new environment. If an environment name is not passed with the argument, the new environment name will be auto-generated. `-t {minimal,slim,standard,intel,custom}` or `--type {minimal,slim,standard,intel,custom}` Defines the project type to be created.

Using the create command will create a directory named after the app name. Inside, it will create a Dockerfile, a Python script, and add the analytics API wheel. If type is not provided, standard will be used.

Build command

The build command is used to build the analytics app Docker image.

Item	Description
Usage	`nerve-analytics build [-h] [-t TAG] [-p PATH]`
Optional arguments	`-h` or `--help` Use this argument to show the help message in the command line. `-t TAG` or `--tag TAG` Name of the Docker tag that will be used when tagging the Docker image. `-p PATH` or `--path PATH` Path to the application folder.

If tag is not provided, latest will be used. If path is not provided, the build command will look for a Dockerfile in the current directory.

Push command

The push command is used to push the previously built Docker image to a Docker registry.

Item	Description
Usage	`nerve-analytics push [-h] [-t TAG] [-r RE_TAG] [-n NEW_NAME] [-u USERNAME] [-e EMAIL] [-p PASSWORD] reg_path`
Positional arguments	`reg_path` Path of the Docker registry.
Optional arguments	`-h` or `--help` Use this argument to show the help message in the command line. `-t TAG` or `--tag TAG` Tag name of the Docker image to be pushed. `-r RE_TAG` or `--re-tag RE_TAG` Define tag name of the Docker image in the registry. `-n NEW_NAME` or `--name NEW_NAME` New name of the repository. `-u USERNAME` or `--username USERNAME` Username which is used to log in into registry. `-e EMAIL` or `--email EMAIL`Email which is used to log in into the registry. `-p PASSWORD` or `--password PASSWORD` Password which is used to log in into the registry.

If tag is not provided, latest will be used. If password is not provided, a prompt to enter it will be displayed to the user.

Provision command

The provision command is used to upload the previously built Docker image as a Docker workload to a Nerve Management System. The Docker image is also archived as a TAR.GZ file.

Item Description

Usage

nerve-analytics provision [-h] [-f CFG_FILE] -u URL -vn VERSION_NAME -rn RELEASE_NAME [-usr USERNAME] [-pas PASSWORD] [-n NAME] [-i IMAGE] [-d DEPLOYED_CONTAINER_NAME] [-net NETWORKS] [-desc DESCRIPTION] [-p PORT] [-e ENV] [-m MEM_LIMIT] [-cl CPU_LIMIT] [-r {no,on-failure,always,unless-stopped}] [-rel] [--verbose]

Arguments that start with -- (eg. --url) can also be set in a config file (specified via -f <filepath>). The config file syntax accepts the following separated by line breaks:

key=value
flag=true
stuff={a,b,c}

Example:
name=test-workload
verbose=true
restart={on-failure}

For details on the syntax refer to https://pypi.org/project/ConfigArgParse/. If an argument is specified in more than one place, then the command line values override the config file values which override the defaults.

Optional arguments -h or --help
Use this argument to show the help message in the command line.

-f CFG_FILE or --cfg-file CFG_FILE
Configuration file for the provision command.

-u URL or --url URL
URL of the Nerve Management System to which the workload will be uploaded. This flag is mandatory.

-vn VERSION_NAME or --version-name VERSION_NAME
Version name of the Docker workload. This flag is mandatory.

-rn RELEASE_NAME or --release-name RELEASE_NAME
Release name of the Docker workload. This flag is mandatory.

-usr USERNAME or --username USERNAME
Username which is used to log in to the Nerve Management System.

-pas PASSWORD or --password PASSWORD
Password which is used to log in to the Nerve Management System.

-n NAME or --name NAME
Name of the Docker workload which will be uploaded to the Nerve Management System.

-i IMAGE or --image IMAGE
This is the of the Docker image that has been created with the create and build commands and will be provisioned to the Nerve Management System.

-d DEPLOYED_CONTAINER_NAME or --deployed-container-name DEPLOYED_CONTAINER_NAME
Name of the Docker container upon deployment to a node.

-net NETWORKS or --networks NETWORKS
Docker network names. This is set to nerve-dp by default if not specified.

-desc DESCRIPTION or --description DESCRIPTION
Description for the Docker workload to be uploaded.

-p PORT or --port PORT
Port mapping for the container to be deployed, for example 5432:5432 or 123:234/udp.

-e ENV or --env ENV
Environment variables for the container to be deployed, for example VAR1=myval.

-m MEM_LIMIT or --mem-limit MEM_LIMIT
Memory limit in MB for the workload on the node.

-cl CPU_LIMIT or --cpu-limit CPU_LIMIT
Max number of CPUs to be used by the workload on the node. 1 by default.

-r {no,on-failure,always,unless-stopped} or --restart {no,on-failure,always,unless-stopped}
Restart policy for the container to be deployed on the node.

-rel or --released
This sets the workload version to released. Released versions cannot be updated.

--verbose
Print workload information.

The user that is used to provision the analytics app must already exist in the Management System. All parameters used when uploading a workload from the Management System can also be specified in the provision command.

If password is not provided, a prompt to enter the password will be displayed. If a workload with the same name already exists in the Management System, it must be a Docker workload in order to add a new version to it. Otherwise, the provisioning cannot be done. When adding a new version to an existing Docker workload in the Management System, version must be different than that of the existing version.

Application Programming Interface

The Analytics API is used for writing Python applications that collect and store data from and to other elements of the Data Services. The recommended Python version to use when writing analytics apps is 3.7 or later. The Analytics API provides the following modules:

Module	Description
Batch Input TimescaleDB	Used to fetch data from a TimescaleDB database.
Batch Output TimescaleDB	Used to store data in a TimescaleDB database.
Stream Input ZeroMQ	ZeroMQ Subscriber used to collect data from a ZeroMQ Publisher. Can work in both asynchronous and synchronous modes. NOTE In order to connect the ZeroMQ Publisher output of the Gateway and the Stream Input ZeroMQ of the analytics, the IP address `172.20.10.1` must be used. This is the case only if the analytics container is running inside the Docker network `nerve-dp`. If the Docker network is `host`, `localhost` can be used.
Stream Output MQTT	MQTT Publisher used to send data to a MQTT broker.

More in depth documentation regarding the API modules can be found in the release package of the analytics element that has been downloaded from the Nerve Software Center.