Logging and monitoring

Data is gathered for Docker workloads, Docker Compose workloads and system parameters regarding resource usage. Metrics are then sent to OpenSearch in the Management System and visualized in OpenSearch Dashboard. Logging and monitoring options can be configured for each node in the node details view in the Management System:

1. Log in to the Management System.
2. Select Nodes in the navigation on the left.
3. Select the node tree tab.
4. Select a node in the node tree.

5. Select the Logs tab.

   

From here, tick the corresponding checkbox of a dashboard and select Save to enable logging or monitoring on that node. Note that system logs do not need to be activated, as they are collected by default. Each dashboard can be accessed by selecting the dashboard link. A new browser tab opens, displaying the selected dashboard. Note that all dashboards are also available if the node is offline.

Node system logs

Selecting VIEW SYSTEM LOGS will open a new window and show the system logs of the node. The internal node logs are aimed at Nerve service technicians in case of error and failure. Data is stored and visualized with OpenSearch. The amount of logs can be modified through the system log settings by Nerve service technicians. Contact customer support for more information.

System monitoring

Resource utilization of the system as a whole is tracked when system monitoring is enabled. Refer to the screenshot and the table below for more information on the data that is gathered and displayed. Tick the checkbox next to VIEW SYSTEM METRICS to enable the tracking of system data.

Docker workload logging

The logs of the Docker workloads on a node are collected in the centralized logging system, allowing the analysis of logs from multiple workloads and nodes. Logs are collected from the standard Linux streams stdout (for debug messages) and stderr (for error messages). So for user created workloads this means that logs need to be sent to these streams to be collected. Note that the logs are most suitable to be read by developers with expert knowledge and should also be configured by developers. Tick the checkbox next to DOCKER WORKLOAD LOGGING to enable the tracking of Docker workload logs.

To display logs of a certain workload, collected logs can be filtered in the Docker workload logging dashboard in OpenSearch.

Docker workload monitoring

Metadata of the overall state of Docker workloads is gathered in the Management System. A list of installed containers and their resource utilization is displayed in this dashboard. Tick the checkbox next to DOCKER WORKLOAD MONITORING to enable the tracking of Docker workload data.

Audit logs for the Management System

Audit logs are a systematic and chronological record of events, activities, or transactions within a system. These logs capture a wide range of information, including who performed an action, what the action was, when it occurred, and the outcome of the action. As such, they are a distinct security capability, providing forensics capability and traceability about past actions, and are crucial for several purposes, such as security, compliance, troubleshooting, and performance monitoring.

The audit logs for the Management System are presented in an OpenSearch dashboard. Access the dashboard within the SYSTEM INFO section in the Management System:

1. Access the Management System.

2. Select SYSTEM INFO at the bottom-left.

   

3. Select the Logs tab.

4. Select VIEW AUDIT LOGS.

   

An OpenSearch Dashboard window will open in a new browser tab, showing relevant log entries.

Inspect a log entry to show the full range of fields. The fields contain the following information:

Event ID code

The Event ID code is an eight digit number, with every two digits signifying an aspect of the event. Its meaning is generated left to right, meaning that the first digits pair determines the possible value of the second digit pair and so on. The digits stand for the following:

Take a look at the flowchart below for possible digits. Keep in mind that the ID is constructed the following way: NN XX YY ZZ. To keep the overview simple the last digits ZZ are omitted in view below.

graph LR
  A[Management System - 10] --> B[System - 01];
  A[Management System - 10] --> C[Workloads - 02];
  B[System - 01] --> D[Access control - 01];

  C[Workloads - 02] --> F[Deploy - 01];
  C[Workloads - 02] --> G[CRUD - 02];
  C[Workloads - 02] --> H[Workload control - 03];
  C[Workloads - 02] --> I[DNA - 04];
  C[Workloads - 02] --> J[Workload backup - 05];
  C[Workloads - 02] --> K[Workload snapshot - 06];
  C[Workloads - 02] --> L[Remote connections - 07];

  A[Management System - 10] --> T[Role - 03];
  T[Role - 03] --> RA[CRUD - 01];

  A[Management System - 10] --> 04[Node - 04];
  04[Node - 04] --> 0401[Node reboot - 01];
  04[Node - 04] --> 0402[CRUD - 02];

  A[Management System - 10] --> 05[Node tree - 05];
  05[Node tree - 05] --> 0501[CRUD - 01];

  A[Management System - 10] --> 06[Usage reports - 06];
  06[Usage reports - 06] --> 0601[Monthly report - 01];

  A[Management System - 10] --> 07[Node update - 07];
  07[Node update - 07] --> 0701[A/B Update - 01];

  A[Management System - 10] --> 08[Notifications - 08];
  08[Notifications - 08] --> 0801[CRUD - 01]; 

  A[Management System - 10] --> 09[Labels - 09];
  09[Labels - 09] --> 0901[CRUD - 01]; 

  A[Management System - 10] --> 10[Docker Compose service - 10];
  10[Docker Compose service - 10] --> 1001[Control - 01];

  A[Management System - 10] --> 11[Monitoring and logging - 11];
  11[Monitoring and logging - 11] --> 1101[Node level - 01];

  A[Management System - 10] --> 12[Node - User management - 12];
  12[Node User management - 12] --> 1201[List updates triggered by user - 01];  

Refer to the table below for a list of events that signify the last two digits ZZ. Note that the Message and Additional info fields in the audit logs dashboard give more context to each event:

Management System logs

Management System logs can be accessed in the SYSTEM INFO section by users with the Admin role or the UI_SERVER_AUDIT:VIEW permission assigned to their role. These internal logs are aimed at Nerve service technicians in case of error and failure. Data is stored with OpenSearch and visualized with OpenSearch Dashboard.

Access the Management System logs the following way:

1. Access the Management System.

2. Select SYSTEM INFO at the bottom-left.

   

3. Select the Logs tab.

4. Select VIEW MANAGEMENT SYSTEM LOGS.

   

A new tab opens, displaying the Management System logs in an OpenSearch dashboard.

Setting up alerts

Users can monitor specific events and receive alerts when critical events are detected. The instructions below explain how to create alerts and choose a notification method for specific events detected in a Docker workload. The instructions are split up the following way to make them easier to follow:

1. Setting up a notification channel
   A notification channel is set up to use in an action. Inside the channel, sender and recipients are defined. This section is split up further into creating an SMPT sender, a recipient group and a notification channel.
   

2. Setting up a monitor
   A monitor is a job that runs on a defined schedule and queries OpenSearch indexes. The results of these queries are then used as input for one or more triggers.
   

3. Setting trigger and action
   Triggers are conditions that, if met, generate alerts, with an action that sends out information after being triggered. Actions have a destination, a message subject and a message body.
   

Since the instructions below cover steps performed in OpenSearch, OpenSearch needs to be accessed in the Nerve system to proceed:

1. Log in to the Management System.
2. Select a node from the node tree.

3. Select the Logs tab.

   

4. Select any of the dashboard links to access OpenSearch.

   

Setting up a notification channel

To send automated notifications from OpenSearch, a notification channel has to be defined. This example shows how to set up an e-mail notification channel. Before setting up a notification channel, a sender and recipient group have to be defined, which is covered in separate steps below.

Creating an SMTP sender

SSL and OpenSearch keystore are recommended to store usernames and passwords of the sender.

1. Select the burger menu in the upper-left.

2. Select Notifications under OpenSearch Plugins.

   

3. Select Email senders.

4. Select Create SMTP sender on the right.

   

5. Enter the following information:

   Setting	Value
   Sender name	nerve-alerts
   Email address	alerts@nerve.cloud
   Host	mail.nerve.cloud
   Port	587
   Encryption method	STARTTLS

6. Select Create.

   

The SMTP sender is now created. However, authentication of the SMTP sender needs to be performed by Nerve Service technicians. Contact customer support through the TTTech Industrial support portal to have the SMTP sender authenticated.

Creating a recipient group

Individuals or a group of people that should be notified about specific events are added to recipient groups.

1. Select the burger menu in the upper-left.

2. Select Notifications under OpenSearch Plugins.

   

3. Select Email recipient groups.

4. Select Create recipient group on the right.

   

5. Enter the following information:

   Setting	Value
   Name	Enter a name for the recipient group. This example uses documentation.
   Description	This field is optional. Enter a description to give more information about the recipient group.
   Emails	Add the email addresses of the recipients that should receive alerts.

6. Select Create.

   

Creating a notification channel

Notification channels can be set up in multiple ways. This example sets up an e-mail notification channel using SMTP. Refer to OpenSearch documentation for more information on other types of notification channels.

1. Select the burger menu in the upper-left.

2. Select Notifications under OpenSearch Plugins.

   

3. Select Channels.

4. Select Create channel on the right.

   

5. Enter the following information:

   Category	Settings and values
   Name and description	Name Enter a name for the channel. This example uses docs-channel. Description This field is optional. Enter a description to give more information about the channel.
   Configurations	Channel type Select Email from the drop-down menu. Sender type Select SMTP sender. SMTP sender Select the sender that was created in Creating an SMTP sender above from the drop-down menu. This example uses nerve-alerts. Default recipients Select the recipient group that was created in Creating a recipient group above from the drop-down menu. This example uses documentation.

6. Select Create.

   

Once created, the channel is automatically activated. It is suggested to temporarily deactivate a channel in case an error has been found. If a system is being repaired, logs will likely produce further error messages that would be detected. Muting a channel helps avoiding unnecessary alarms in the meantime. Tick the checkbox next to the channel and select Mute from the Actions drop-down menu.

Setting up a monitor

When setting up a monitor, it is required to define a query, which is used to detect words and phrases in log messages. This query serves as the base of the alert. This query is written in a search language called query domain-specific language (DSL), which is provided by OpenSearch. Query DSL is a flexible language with a JSON interface. Refer to OpenSearch documentation for more information on how to write queries in Query DSL.

For this example, the query is set up to find specific keywords inside custom logs from a container. The query used in the instructions below looks for the phrase ERROR in all log messages, which is set by the query parameter. Replace the value with other words or phrases to search for different terms.

1. Select the burger menu in the upper-left.

2. Select Alerting under OpenSearch Plugins.

   

3. Select the Monitors tab.

4. Select Create monitor on the right.

   

5. Enter a Monitor name. This example uses docs-monitor.

6. Select Per query monitor under Monitor type.

7. Select Extraction query editor under Monitor defining method.

   

8. Enter filebeat* under Data source to collect data stored in all filebeat indexes.

9. Copy the following script into Define extraction query.

   {
       "size": 1000,
       "query": {
           "bool": {
               "filter": [
                   {
                       "range": {
                           "@timestamp": {
                               "from": "{{ period_end }}||-1m",
                               "to": "{{ period_end }}",
                               "include_lower": true,
                               "include_upper": true,
                               "format": "epoch_millis",
                               "boost": 1
                           }
                       }
                   },
                   {
                       "match_phrase": {
                           "message": {
                               "query": "ERROR",
                               "slop": 0,
                               "zero_terms_query": "NONE",
                               "boost": 1
                           }
                       }
                   }
               ],
               "adjust_pure_negative": true,
               "boost": 1
           }
       },
       "aggregations": {}
   }
   

   

As mentioned above, the query detects log messages that contain the phrase ERROR in the last minute. Since it is executed every minute, it will detect each error only once. Continue with the next section to configure a trigger with a corresponding action.

Setting trigger and action

The trigger condition controls when notifications are sent through the notification channel. The condition is using the results of the query which was set up in Setting up a monitor above.

When the monitored event is triggered, an action is performed as a result. In this example, an e-mail is sent to the notification channel that was created in Setting up a notification channel above.

1. Select Add trigger.

   

2. Enter a Trigger name. This example uses docs-trigger.

3. Select the desired Severity level from the drop-down menu.

4. Define a trigger condition. For this example, the condition is set to trigger if there are more than five errors detected in the last minute:

   ctx.results[0].hits.total.value > 5
   

   

5. Scroll down to reach the Actions (1) section.

6. Enter the following information:

   Setting	Value and description
   Action name	Enter a name for the action. This example uses alerting-action.
   Channels	Select the channel that was created in Setting up a notification channel above. This example uses docs-channel.
   Message subject	Enter a message subject that will be the subject of the e-mail notification.
   Message	Enter a message in the Message field that will be the body of the e-mail notification. Some information using variables is already pre-filled. Refer to OpenSearch documentation for more information on how to use variables in the message.

   

7. Select Create at the bottom of the page.

With this, a monitor based on a query is created that is executed every minute and detects all log messages containing a key word or phrase. An alarm is triggered if more than five messages per minute are detected and an e-mail with error details is sent to a recipient group.

Accessing workload logs in the Local UI

Logs can be accessed locally through the Local UI. There, logs are accessed separately for each Docker workload or Docker Compose service.

1. Access the Local UI.

2. Select Workload management in the navigation on the left.

   

3. Select a Docker or Docker Compose workload.

4. Select the Logs tab.

   

Log message are displayed in a large message window. Take note of the following functions:

Note that for Docker Compose workloads, logs can be accessed for each service. Select a service from the list to display its logs.