Updated test for metrics

Application metrics requirements:

-   **[REQ-METRICS-9]** GUI and API latencies shall be tracked and reported.

-   **[REQ-METRICS-10]** GUI and API error rates shall be tracked and reported.

-   **[REQ-METRICS-9]** GUI and API latencies and error rates shall be tracked and reported.

Matching tests for these requirements are listed in [6.3.6 Metrics tests].

#### 6.1.1.2 REQ-EXPLOIT-2

**Requirement:** The product shall have OS and Application upgrade instructions which makes it possible to obtain the set High Availability targets.<br/>

**Objective:** <br/>

**Objective:** Responsibility of OS level updgrades can be elsewhere outside of the system control.<br/>

**Preparation:** None<br/>

**Activities:**

### 6.2.0.2 REQ-TECH-2

**Requirement:** When privileged information is transferred or accessed, a secure channel shall be used in transport [5.2.1 Secure channel definition].<br/>

**Objective:** <br/>

**Objective:** Protect the integrity of the data.<br/>

**Preparation:** None<br/>

**Activities:**

#### 6.2.5.0 REQ-SBOM-0

**Requirement:** Operating system dependencies and application dependencies shall be clearly separated in the provided SBOM.<br/>

**Objective:** To make clear what part of the system to upgrade, the source of the dependency should be understandable.

**Objective:** To make clear what part of the system to upgrade, the source of the dependency should be understandable.<br/>

**Preparation:** None<br/>

**Activities:**

**Requirement a:** Unique, unambiguous, and machine-readable identification of all components and dependencies are provided in the SBOM.<br/>

**Requirement b:** The SBOM identifier format is consistent with common vulnerability handling standards.<br/>

**Objective:** A linux kernel version can be 6.18, but what it contains? A refereable and exact pointer is needed.

**Objective:** A linux kernel version can be 6.18, but what it contains? A refereable and exact pointer is needed.<br/>

**Preparation:** None<br/>

**Activities:**

#### 6.2.5.2 REQ-SBOM-2

**Requirement:** The SBOM shall be consistent with [5.3.4 Secure updates] practices.<br/>

**Objective:** The deliverable erodes over time. The SBOM is one of the sources for the motivation to upgrade.

**Objective:** The deliverable erodes over time. The SBOM is one of the sources for the motivation to upgrade.<br/>

**Preparation:** None<br/>

**Activities:**

#### 6.3.6.0 REQ-METRICS-0

**Requirement:**

**Objective:** Collected and stored metrics data can not be altered.<br/>

**Preparation:**

**Activities:** Review the documentation of all components between the target and the collected and stored metrics data looking for any step that may allow alteration of the metrics data after it has left the target.<br/>

**Verdict:** Pass if no process step allows the alteration before ingestion of collected metrics data after it has left the target.<br/>

**Supporting Evidence:** The technical documentation.<br/>

**Requirement:** The product shall be designed in a way that collected and stored metrics data can not be altered.<br/>

**Objective:** An attacker wants to hide its operations. System should prepare for that.<br/>

**Preparation:** None<br/>

**Activities:**

1. Study the technical documentation.

**Verdict:**

1. Pass if no unauthorised process before ingestion of collected metrics data can alter the metrics before storage,

2. and the storage can not be altered outside of reqular cleaning cycles.

3. Fail otherwise.

**Supporting Evidence:**

1. References to to documentation sections.

#### 6.3.6.1 REQ-METRICS-1

**Objective:** Historical metrics data import overwriting an existing data point is noticed.<br/>

**Requirement:** Historical metrics data import overwriting an existing data point shall be noticed.<br/>

**Objective:** Prevent compromised managed element to hide its behaviour.<br/>

**Preparation:**

1.  Have the NMS product initialised and available with the default configuration.

1.  Create required authentication credentials for the test.

1.  Prepare an import data set, that represents natural collected values from the target.

1.  Create a copy of the import data set and modify the values.

2.  Create required authentication credentials for the test.

3.  Prepare an import data set, that represents natural collected values from the target.

4.  Create a copy of the import data set and modify the values.

**Activities:**

#### 6.3.6.2 REQ-METRICS-2

**Objective:** Metric name, purpose, and value interpretation are described for the user.<br/>

**Requirement:** Metrics name, purpose, and value interpretation shall be described for the user.<br/>

**Objective:** Understanding what is collected helps user to undrestand what happens, but also is needed for data minimisation validation.<br/>

**Preparation:**

1.  Have the NMS product initialised and available with the default configuration and required credentials.

#### 6.3.6.3 REQ-METRICS-3

**Objective:** Metrics cadence, accuracy and storage time are described for the user.<br/>

**Requirement:** Metrics cadence, accuracy and storage time shall be described for the user.<br/>

**Objective:** Metrics storage consumes alot of storage, and also affects persons privacy as rarely the data is deleted on demand.<br/>

**Preparation:**

1.  Have the NMS product initialised and available with the default configuration and required credentials.

#### 6.3.6.4 REQ-METRICS-4

**Objective:** System does not collect metrics that are not used in operative purposes.<br/>

**Preparation:**

1. Evaluate **[REQ-MON-2](#63x2-req-mon-2)** test

1. Evaluate **[REQ-MON-3](#63x3-req-mon-3)** test

**Activities:**

1. Study the monitoring data GUI.

1. Study the provided documentation.

1. Study the referred tests.

**Verdict:**

1. Pass if the metrics collection cadence, accuracy and storage time is justified by conformity to regulation or business related criteria.

**Supporting Evidence:**

1. Metrics storage plan.

1. Metrics conformity assesment.

1. Product position in relation to GDPR.

#### 6.3.6.5 REQ-METRICS-5

**Objective:** Relevant system and connected element metrics like CPU, memory, disk utilisation are tracked and reported.<br/>

**Requirement:** Relevant system and connected element metrics like CPU, memory, disk utilisation shall be tracked and reported.<br/>

**Objective:** A compromised managed device mines bitcoins, or floods other services with unwanted traffic.<br/>

**Preparation:**

1.  Have the NMS product initialised and available with the default configuration and required credentials.

1.  Have the product initialised and available with the default configuration and required credentials.

**Activities:**

1. The technical documentation.

1. Screenshot of the GUI displaying how the data is displayed.

#### 6.3.6.6 REQ-METRICS-6

**Reference:** **[REQ-MON-6a]** and **[REQ-MON-6b]** <br/>

#### 6.3.6.5 REQ-METRICS-5

**Objective a:** System process and service crashes and restarts are tracked and reported.<br/>

**Objective b:** Managed element process and service crashes and restarts are tracked and reported.<br/>

**Requirement a:** System process and service crashes and restarts shall be tracked and reported.<br/>

**Requirement b:** Managed element process and service crashes and restarts shall be tracked and reported.<br/>

**Objective:** Crashes are used to modify the program state. Abnormal crashes can be an indication of upcoming compromise.<br/>

**Preparation:**

1.  Have the NMS product initialised and available with the default configuration and required credentials.

1.  Have the managed element initialised and available with the default configuration and required credentials.

2.  Have the managed element initialised and available with the default configuration and required credentials.

**Activities:**

**Supporting Evidence:** Log or and metrics output showing detected system or managed element crash or restart with the reported cause.<br/>

#### 6.3.6.7 REQ-METRICS-7

#### 6.3.6.6 REQ-METRICS-6

**Objective:** Managed elements and system nodes and provided services availabilities and statuses are tracked and reported.<br/>

**Requirement:** Managed elements and system nodes and provided services availabilities and statuses shall be tracked and reported.<br/>

**Objective:** Bad availability can be a indication of compromise.<br/>

**Preparation:**

1.  Have the NMS product initialised and available with the default configuration and required credentials.

1. The technical documentation.

1. Screenshot of the GUI displaying how the data is displayed.

#### 6.3.6.8 REQ-METRICS-8

#### 6.3.6.7 REQ-METRICS-7

**Reference:** **[REQ-MON-8a]** and **[REQ-MON-8b]** <br/>

**Objective a:** Relevant system database and storage health metrics like queries per second, latency and throughput are tracked and reported.<br/>

**Objective b:** Relevant managed element database and storage health metrics like queries per second, latency and throughput are tracked and reported.<br/>

**Requirement a:** Relevant system database and storage health metrics like queries per second, latency and throughput shall be tracked and reported.<br/>

**Requirement b:** Relevant managed element database and storage health metrics like queries per second, latency and throughput shall be tracked and reported.<br/>

**Objective:** Bad service quality can be a indication of compromise.<br/>

**Preparation:**

1.  Have the NMS product initialised and available with the default configuration and required credentials.

1.  Have the managed element initialised and available with the default configuration and required credentials.

1.  Have the product initialised and available with the default configuration and required credentials.

2.  Have the managed element initialised and available with the default configuration and required credentials.

**Activities:**

1. The technical documentation.

1. Screenshot of the GUI displaying how the data is displayed.

#### 6.3.6.9 REQ-METRICS-9

#### 6.3.6.8 REQ-METRICS-8

**Objective:** Relevant networking metrics like throughput and protocol errros are tracked and reported.<br/>

**Requirement:** Relevant networking metrics like throughput and protocol errors shall be tracked and reported.<br/>

**Objective:** When the errors stop and the throughput returns to nominal levels, the damage has already been done.<br/>

**Preparation:**

1.  Have the NMS product initialised and available with the default configuration and required credentials.

1.  Have the managed element initialised and available with the default configuration and required credentials.

1.  Have the product initialised and available with the default configuration and required credentials.

2.  Have the managed element initialised and available with the default configuration and required credentials.

**Activities:**

1. The technical documentation.

1. Screenshot of the GUI displaying how the data is displayed.

#### 6.3.6.10 REQ-METRICS-10

#### 6.3.6.9 REQ-METRICS-9

**Objective:** GUI and API latencies are tracked and reported.<br/>

**Requirement:** GUI and API latencies and error rates shall be tracked and reported.<br/>

**Objective:** Wrong calls to the endpoints is an indication of compromise attempt.<br/>

**Preparation:**

1.  Have the NMS product initialised and available with the default configuration and required credentials.

1.  Have the managed element initialised and available with the default configuration and required credentials.

1.  Have the product initialised and available with the default configuration and required credentials.

2.  Have the managed element initialised and available with the default configuration and required credentials.

**Activities:** <br/>

1. Relevant metrics described in the technical documentation.

#### 6.3.6.11 REQ-METRICS-11

**Objective:** GUI and API error rates are tracked and reported.<br/>

**Preparation:**

1.  Have the NMS product initialised and available with the default configuration and required credentials.

1.  Have the managed element initialised and available with the default configuration and required credentials.

**Activities:**

1.  Send a malformed API request towards each system API endpoint.

**Verdict:**

1. Pass if all API endpoints notices the malformed request.

1. and reports relevant data into the logs.

1. Fail otherwise.

**Supporting Evidence:**

1. System log output.

1. Relevant metrics described in the technical documentation.

### 6.3.8 High availability tests

#### 6.3.8.0 REQ-HA-0