***MON_METRICS-1** The product shall be designed in a way that collected and stored metrics data can not be altered.
***MON_METRICS-2** Historical metrics data import overwriting an existing data point shall be noticed.
***MON_METRICS-3** Metrics name, purpose, and value interpretation shall be described for the user.
***MON_METRICS-3** Metrics name, purpose, and value interpretation shall be described for the product user.
> NOTE: **MON_METRICS-3** leaves open how the information is conveyed to the product user. It could be a dedicated portal within the product or machine readable output having the same information.
Availability, uptime, and application metrics requirements:
***MON_METRICS-4** The product shall collect, track and store metrics on, including, but not limited to:
* availability and status changes, like process and service crashes and restarts
* incidents
* relevant operative information like CPU, memory, disk utilisation
* relevant networking metrics like throughput and protocol errors
* relevant databases and storage health metrics like queries per second, latency and throughput
* GUI and API latencies, where available
1. availability and status changes, like process and service crashes and restarts
2. incidents
3. relevant operative information like CPU, memory, disk utilisation
4. relevant networking metrics like throughput and protocol errors
5. relevant databases and storage health metrics like queries per second, latency and throughput
6. GUI and API latencies, where available
from the following targets:
* managed elements, where such metrics are available or can be reasonably synthetised
* provided services
* the system itself.
1. managed elements, where such metrics are available or can be reasonably synthetised
2. provided services
3. the system itself.
The product can be a host to an ecosystem that exposes managed elements capacity for 3rd. party applications.
The product can operates as a runtime for the workload, and effectively inherits operating system like responsibilies and funcionalities.
@@ -1613,22 +1615,6 @@ This clause addresses the requirements in the CRA [\[i.1\]](#_ref_i.1) Annex 1 P
# 6 Assessment criteria for compliance with technical requirements
<mark>Editor’s Note: It has been agreed across vertical standards to define each assessment criteria following the common structure:<br>
- Requirement reference<br>
- Objective<br>
- Preparation<br>
- Activities<br>
- Verdict<br>
- Evidence<br>
The assessment criteria clause shall be structured by requirement defined in clause 5.
</mark>
<mark>Editor’s Note: The assessment clause should not contain “hidden” requirements. This means that all required specificities regarding the technical requirements should appear in clause 5. Clause 6 is only about verifying that the product complies with such technical requirements. </mark>
<mark>Editor’s Note: The assessment clause should not mandate a “self-assessment” or “third party conformity assessment”. The assessment clause should be without prejudice to specific conformity assessment procedures as detailed in Annex VIII of the CRA.</mark>
<mark>Editor’s Note: Elements of the assessment clause should map one-to-one to elements of the technical requirements clause, ensuring there is a direct correspondence both ways. This means there shall be no summary assessment criteria.</mark>
The assessment has the purpose to verify that the requirements of the present standard are met in consideration of the use case and the thereout resulting risks.
The requirements need to counter these risks with an appropriate level of security.
Consequently, those need to be fulfilled by product functions and mechanisms, and, where there is none, the related risk assessment needs to prove that non-product means, such as obligations for the operational environment and user information, even obligation, mitigate the risks left open by the product.
@@ -2008,6 +1994,29 @@ Verify that:
## 6.9 Data minimisation
#### 6.14.2.3 REQ-METRICS-3
**Objective:** The product provides the information on the meaning of the metrics data and about their memory and storage consumption.<br/>
**Preparation:**
1. Have the product initialised and available with the default configuration and required credentials.
**Activities:**
1. Study the monitoring data GUI.
1. Study the provided documentation.
1. Investigate the metrics storage.
**Verdict:**
1. Pass if the metrics collection cadence, accuracy and storage time matches the described use.
1. Fail otherwise.
**Supporting Evidence:**
1. The technical documentation.
1. Metrics storage plan.
## 6.10 Availability protection
### 6.10.0 REQ-HA-0
@@ -2201,7 +2210,7 @@ Verify that:
### 6.14.2 Metrics tests
#### 6.14.2.0 REQ-METRICS-0
#### 6.14.2.1 MON_METRICS-1
**Objective:** Verifies the ingestion pipline design upholds integrity and confidentiality requirements.<br/>
**Preparation:** None<br/>
@@ -2211,8 +2220,8 @@ Verify that:
**Verdict:**
1. Pass if no unauthorised process before ingestion of collected metrics data can alter the metrics before storage,
2. and the storage can not be altered outside of reqular cleaning cycles.
1. Pass if no unauthorised process before ingestion of collected metrics data can alter the metrics before storage
2. and if the storage can not be altered outside of reqular cleaning cycles.
3. Fail otherwise.
**Supporting Evidence:**
@@ -2220,28 +2229,28 @@ Verify that:
1. References to to documentation sections.
#### 6.14.2.1 REQ-METRICS-1
#### 6.14.2.2 MON_METRICS-2
**Objective:** Stored data wiping or overwriting can always be recognized.<br/>
**Preparation:**
1. Have the product initialised and available with the default configuration.
2. Create required authentication credentials for the test.
3. Prepare an import data set that represents normal operational metric data from a managed device.
1. Have the product initialised and available with the default configuration;
2. Create required authentication credentials for the test;
3. Prepare an import data set that represents normal operational metric data from a managed device;
4. Create a copy of the import data set and modify the data to change also related integrity protection values.
**Activities:**
1. Begin a timeseries data collection.
1. Begin a timeseries data collection;
2. Restart the collection of timeseries data with the modified data set.
**Verdict:** Pass if systems detects the modified data set.<br/>
**Supporting Evidence:**
1. Collect output showing the whether the current metrics data is being handled by the normal flow as expected.
1. Collect output showing the whether the current metrics data is being handled by the normal flow as expected;
2. Collect output showing how the modified data set was accepted or discarded.
#### 6.14.2.2 REQ-METRICS-2
#### 6.14.2.3 MON_METRICS-3
**Objective:** The user has explanations about the meaning of each metric and is enabled to interpret it. Furthermore, it clarifies the relevance of collected metric data.<br/>
**Preparation:**
@@ -2250,202 +2259,93 @@ Verify that:
**Activities:**
1. Study the monitoring data GUI.
1. Study the provided documentation.
1. Study the monitoring data GUI;
1. Study the provided documentation;
1. Investigate the metrics storage.
**Verdict:**
1. Pass if there are no unknown metrics displayed or collected.
2.Pass if there are commonly well-known metrics displayed or collected, like CPU usage, which can remain without explanation.
3.Pass if a metric is recognised and a pointer to the documentation is provided (managed element manufacturer’s reference documentation e.g.).
1. Pass if there are no unknown metrics displayed or collected
2.and if there are commonly well-known metrics displayed or collected, like CPU usage, which can remain without explanation
3.and if a metric is recognised and a pointer to the documentation is provided (managed element manufacturer’s reference documentation e.g.).
4. Fail otherwise.
**Supporting Evidence:**
1. Collect output showing whether the normal operation metrics data is operated as expected.
2. Collect output showing whether the modified data set was detected and then notified or logged
1. Output samples showing how the information is conveyed for the user.
> NOTE: Whether a modified data set gets accepted or discarded might not be an NMS but an administrator decision.
#### 6.14.2.3 REQ-METRICS-3
**Objective:** The product provides the information on the meaning of the metrics data and about their memory and storage consumption.<br/>
**Preparation:**
1. Have the product initialised and available with the default configuration and required credentials.
**Activities:**
1. Study the monitoring data GUI.
1. Study the provided documentation.
1. Investigate the metrics storage.
**Verdict:**
1. Pass if the metrics collection cadence, accuracy and storage time matches the described use.
1. Fail otherwise.
**Supporting Evidence:**
1. The technical documentation.
1. Metrics storage plan.
#### 6.14.2.4 REQ-METRICS-4
**Objective:** Support users or administrators ability to detect compromised, misconfigured, or harmful connected elements through unusual, excessive, or risky patterns of use.<br/>
**Preparation:**
1. Have the product initialised and available with the default configuration and required credentials.
2. Have at lest one managed element as part of the system the product operates
**Activities:**
1. Study the provided technical documentation to interpret the monitoring data GUI.
1. Simulate unnormal behaviour by intensionally cutting the connection with a managed element and observe the monitoring data.
1. Restart the managed element.
**Verdict:**
#### 6.14.2.4 MON_METRICS-4
1. Pass if basic administrative metrics are tracked and displayed.
1. Pass if the connection loss to the managed element is recognized.
1. Pass if by observing the metrics a baseline can be established.
1. Pass if anomalities like load spikes after a restart can be observed.
1. Fail otherwise.
**Supporting Evidence:**
1. The technical documentation of the monitoring data.
1. Screenshot of the GUI displaying how the data is displayed the normal operation and the detection of the unnormal behaviour.
#### 6.14.2.5 REQ-METRICS-5
**Objective:** A NMS crash and restart is analyzable and can allow to be forecasted based on the recorded data and the event of a crashed or restarted managed element is detected and recorded by the NMS.<br/>
**Preparation:**
1. Have the NMS product initialised and available with the default configuration and required credentials.
2. Have the managed element initialised and available with the default configuration and required credentials.
**Activities:**
1. Purposefully crash an NMS process.
2. Study the NMS behaviour in terms of process recovery or restarting the NMS.
3. Purposefully crash a process or restart a connected element.
4. Study the metrics output.
**Verdict:**
1. Pass if the NMS process or the NMS recovers to normal operation
2. and if the event can be identifed in the NMS logging
3. and if the NMS detects and displays or reports the event on the managed device.
4. Fail otherwise.
**Objective:**
**Supporting Evidence:** Log or and metrics output showing detected system or managed element crash or restart with the reported cause.<br/>
* Support users or administrators ability to detect compromised, misconfigured, or harmful connected elements through unusual, excessive, or risky patterns of use;
* To detect availability issues;
* Out of the normal behaviour, metrics indicate failures or compromise of both the managed elements and the product;
* The error metrics enable for tracing and serve as entry point for the analysis and detection of abnormal behavior in the managed element and in the system;
* Response latencies on the product queries or protocol errors on incoming data from managed elements enable to detect defective or compromised managed elements.
**Objective:** Bad availability can be a indication of compromise.<br/>
**Preparation:**
1. Have the product initialised and available with the default configuration and required credentials.
2. Have the managed element initialised and available with the default configuration and required credentials.
1. Have the product initialised and available with the default configuration and required credentials;
2. Have at least one managed element as part of the system the product operates;
3. Have the managed element initialised and available with the required credentials;
4. Study the deployment instrcutions about sizing of the operative environment;
5. Study the technical documentation how to interact with the collected metrics data;
**Activities:**
1. Study the provided documentation to interpret the monitoring data.
2. Intensionally disconnect the NMS from a provided service, for example the timestamp server.
**Verdict:**
1. Pass, if the monitoring data the expected operations and their statuses per connected element and relevant system processes
2. and if the monitoring data and notifies the disconnection from the provided service
3. and if the disconnected service can be identified in the logging records.
4. Fail otherwise.
**Supporting Evidence:**
1. The technical documentation.
1. Screenshot of the GUI displaying how the data is displayed.
#### 6.14.2.7 REQ-METRICS-7
**Objective:** Out of the normal metrics indicate failures or compromise of both, the managed elements and the NMS.<br/>
**Preparation:**
1. Study the provided technical documentation how to interpret the monitoring data;
2. Simulate abnormal behaviour by intensionally cutting the connection with a managed element and observe the monitoring data;
3. Restart the managed element;
4. Purposefully crash a process or restart a connected element;
5. Purposefully crash a product process;
6. Generate large amount of traffic in a random location;
7. Send a correct API request from the product towards each managed element API endpoint;
8. Send a correct API request form the managed element towards the product API endpoint;
9. Send an incorrect API request towards each system API endpoint;
10. Send a momentary spike of requests that exceeds the reasonable and expected load of incoming requests.
1. Have the NMS initialised and available with the default configuration and required credentials.
2. Have the managed element initialised and available with the default configuration and required credentials.
Outcomes:
**Activities:**
1. Study the provided product documentation on metrics data.
2. Study the provided product documentation on metrics data that are received from managed elements.
1. Study the product behaviour from the collected metrics in terms of:
1. process recovery
2. how the listed test activities are displayed in the collected metrics.
**Verdict:**
1. Pass if the NMS monitoring data shows the expected operation statuses for both, the NMS and for the managed devices:
- success rate
- operation failure rate
- query execution time
- response size where applicable.
2. Fail otherwise.
1. Pass if basic administrative metrics are tracked and displayed
2. and if the product process or the product recovers to normal operation
3. and if the connection loss to the managed element is recognized
4. and if by observing the metrics a baseline of the system operation can be established
5. and if anomalities like load spikes after a restart can be observed
6. and if the product detects and displays or reports the test event on the managed device
7. and if the product monitoring data shows the expected operation statuses where applicable for both, the product and for the managed devices:
* success rate
* operation failure rate
* query execution time
* response size
8. and if the monitoring data the expected operations and their statuses per connected element and relevant system processes
9. and if the monitoring data and notifies the disconnection from the provided service
10. and if the disconnected service can be identified in the metrics
11. and if corresponding metrics matches the generated traffic
12. and if all API endpoints responds as expected
13. and if the response latency is recorded in defined accuracy, where reasonably available
14. and if system tracks the number of successfull requests, where reasonably available
15. and if the incorrect API requests are identified and accounted
16. and if the spike of high volume requests is handled gracefully.
17. Fail otherwise.
**Supporting Evidence:**
1. The technical documentation on metrics data of NMS and managed elements.
2. Sample or a screenshot how the data is displayed for both, the NMS and the managed elements.
#### 6.14.2.8 REQ-METRICS-8
**Objective:** The error logging records enable for tracing back and serve as entry point for the analysis and detection of abnormal behavior in the managed element and in the system.<br/>
**Preparation:**
1. Have the product initialised and available with the default configuration and required credentials.
2. Have the managed element initialised and available with the default configuration and required credentials.
**Activities:**
1. Study the provided documentation to interpret the monitoring data GUI with respect to errors.
1. Study the provided documentation.
1. Generate large amount of traffic in random location.
**Verdict:**
1. Pass if corresponding metrics matches the generated traffic.
1. Fail otherwise.
**Supporting Evidence:**
1. The technical documentation.
1. Screenshot of the GUI displaying how the data is displayed.
#### 6.14.2.9 REQ-METRICS-9
**Objective:** Response latencies on the product queries or protocol errors on incoming data from managed elements enable to detect defective or compromised managed elements.<br/>
**Preparation:**
1. Have the product initialised and available with the default configuration and required credentials.
2. Have the managed element initialised and available with the default configuration and required credentials.
3. Study the deployment instrcutions about sizing of the operative environment.
**Activities:**<br/>
1. Send a correct API request from the product towards each managed element API endpoint.
2. Send an incorrect API request towards each system API endpoint.
3. Send a momentary spike of request that exceeds the reasonable and expected load of incoming requests.
**Verdict:**<br/>
1. Pass if all API endpoints responds as expected
2. and the response latency is recorded in defined accuracy
3. and tracks the number of successfull requests
4. and the incorrect API requests are identified and logged
5. and the spike of high volume requests is handled gracefully.
6. Fail otherwise.
**Supporting Evidence:**
1. Relevant metrics described in the technical documentation.
* Metrics output showing detected system or managed element crash or restart with the reported cause;
* Relevant vendor or design documentation describing the applied measures;
* Test reports showing the steps performed and results obtained;
* Screenshots, captures, or console outputs confirming the correct execution or protection behaviour;
* Logs, configuration files, or audit traces demonstrating the implementation of the requirement;
