- Sep 25, 2024
-
-
Konstantinos Poulakakis authored
-
Waleed Akbar authored
- Analytics backend Request listener will terminated incase topic doesn't exists. - The GetAlrams() in KpiValueApi is updated to process the new resposne from daskstreamer.
-
- Sep 24, 2024
-
-
Waleed Akbar authored
- Replaced `SparkStreamer` class with `DaskStreamer` class. - Updated methods to call the `DaskStreamer` class. - Improved messages and test files. - Added new package requirements. - Made minor changes to the frontend `requirements` file.
-
- Sep 22, 2024
-
-
Waleed Akbar authored
- Added new tests in the `scripts` folder. - Improved messages and test files across all services. Telemetry and Analytics Backend: - Added a `try` statement to handle Kafka message decoding errors. Telemetry Frontend and KPI Value API: - Downgraded `apscheduler` from `3.10.4` to `3.10.1` in the requirements file. Kafka: - Added a method to retrieve `KAFKA_SERVER_ADDRESS` from environment variables. Telemetry Backend: - Added a condition to allow the collector backend to run indefinitely. - Changed the extracted `KpiValue` timestamp format to `"%Y-%m-%dT%H:%M:%SZ"` to meet Apache Spark requirements. - Removed some unused packages from the `requirements.in` file. Deployment Script: - Added `"telemetry"` and `"analytics"` component names to the `TFS_COMPONENTS` variable in the `my_deploy.sh` file.
-
- Sep 20, 2024
-
-
Waleed Akbar authored
- "FAIL" is changed to "FALL" in Spark streamer. - start() was missing in RequestListener call. - updated test file in KpiValueAPI
-
- Sep 19, 2024
-
-
Waleed Akbar authored
- Corrected the name of `install_servicers()`. - Removed `stop_event` from the `RequestedListener` call. - Spark Streamer now writes to the `ALARMS` topic. - Improved messages and test files.
-
- Sep 17, 2024
-
-
Waleed Akbar authored
- The `split` method has been added to correctly extract the aggregation method name from the dictionary.
-
- Sep 13, 2024
-
-
Waleed Akbar authored
Deployment Script - Analytics is added in new montioring component to be deployed with TFS. - condition is added in analytics component TFS deployment script. - Analytics show logs files are added. Manifest - modifed KAFKA_ADVERTISED_LISTENERS with internal kafka service address in kafka.yml - analyticsservice.yml added. Analytics - grpc service is added in backend - Docker file for backend and frontend is added. - main file is updaed - small changes in requirements.in files Kafka Variables - get_kafka_address method logic is updated.
-
Waleed Akbar authored
Backend: - Updated the `StartSparkStreamer` function call to send `Analyzer_uuid` as the key for messages produced on the Kafka topic. - Updated the `SparkStream` definition to receive the key and added the key to the `streamwriter` object. Frontend: - Integrated APScheduler to manage `StreamListener`. - Added `ResponseListener` to consume messages from the `analytics_response_topic` and process them. - Added APScheduler to manage StreamListener. - Added "ResponseListener" to consumer messages from analytics response_topic and process it.
-
- Sep 09, 2024
-
-
Waleed Akbar authored
Proto: - Added `Duration_s` field to the proto file. Frontend: - Added `SelectAnalyzer` logic. - Improved message formatting in the `create_analyzer_filter()` function. - Added a test case: `test_SelectAnalytics`. Backend: - Renamed the `RunSparkStreamer` method to `StartSparkStreamer`. - Updated the `StartRequestListener` method to return `(thread, stop_event)`. - Added a `StopRequestListener` method to stop the listener. Database: - Added the `select_with_filter` method with actual logic implementation. - Updated the `ConvertRowToAnalyzer` method to correctly read the `operation_mode` ENUM value.
-
- Sep 07, 2024
-
-
Waleed Akbar authored
- Updated the position of the `stop_event` parameter. - Added confirmation for pySpark checkpoint deletion. - Added a PySpark termination script to handle the `StopCollector` event.
-
Waleed Akbar authored
**Backend:** - Added a dictionary to manage running analyzers. - Implemented logic to manage running analyzers. - Added the `TerminateAnalyzerBackend` method to handle analyzer termination. **Frontend:** - Modified and invoked the `PublishStopRequestOnKafka` method.
-
Waleed Akbar authored
- Thresholds, window_size, window_slider is added in Frontend and Backend.
-
- Sep 05, 2024
-
-
Waleed Akbar authored
- In the BackendService.py: + Updated SparkStreamer call with new parameters. - In SparkStreaming.py: + Added 'GetAggregerations' and 'ApplyThresholds' methods + Added 'window_size', 'win_slide_duration', 'time_stamp_col' and 'thresholds' parameters. - Added new messages. - Updated the 'RunSparkStreamer' call wth new parameters.
-
- Sep 03, 2024
-
-
Waleed Akbar authored
- Added Kafka Request Listener to listen for requests from the Analytics Frontend. - Updated `SparkStreamer.py` to consume and process streams from VALUE topics, and further filter rows based on KPIs in the `input_kpis` list. - Updated the frontend TOPIC from `VALUE` to `ANALYTICS_REQUEST`. - Updated messages to keep the KPI UUID consistent with the one generated by the Telemetry Backend service.
-
Waleed Akbar authored
- Added CRDB secret. - Added test script. - Added `main` and `__init__.py` files. - Added Backend Service file. - Added Spark Streamer file (successfully consuming streams from Kafka Topics). - Added Tests folder and files. - Added requirements file. - Added ANALYTICS Kafka topics in `common.tools.kafka.variables` file.
-
- Apr 18, 2024
-
-
Lluis Gifre Renom authored
-
- Feb 01, 2023
-
-
Lluis Gifre Renom authored
- Restored execute permissions
-
Lluis Gifre Renom authored
- Added/Updated headers - Added script to update copyright headers
-
- Jul 26, 2022
-
-
Lluis Gifre Renom authored
Common: - added helper scripts FrontEnd: - corrected import paths - added missing parameters in test - refined creation of backend request
-
- May 20, 2022
-
-
Lluis Gifre Renom authored
- initial skeleton (not functional)
-
- Mar 08, 2022
-
-
Lluis Gifre Renom authored
-
- Feb 17, 2022
-
-
Lluis Gifre Renom authored
-
Lluis Gifre Renom authored
-
- Feb 16, 2022
-
-
Lluis Gifre Renom authored
-
- Nov 04, 2021
-
-
Lluis Gifre Renom authored
-
- Sep 23, 2021
-
-
Javi Moreno authored
-
- Jul 16, 2021
-
-
Lluis Gifre Renom authored
- monitoring/test_monitoring.py: provides a basic testing skeleton - monitoring/requirements.in/.txt: updated with pytest - monitoring/genproto.py: added instructions to import generated Python modules using "from . import ..." instead of "import ..."; otherwise, pytest and other possible modules doing imports might complain. - monitoring/monitoring_server.py: packed start_server and stop_server functionaity to simplify test definition, minor formattings, bug resolutions and improvements in package import. - monitoring/monitoring_client.py: minor formattings, bug resolutions, and improvements in package import.
-
- Jun 14, 2021
-
-
Ricard Vilalta authored
-
- Jun 11, 2021
-
-
Ricard Vilalta authored
-