- Sep 13, 2024
-
-
Waleed Akbar authored
Backend: - Updated the `StartSparkStreamer` function call to send `Analyzer_uuid` as the key for messages produced on the Kafka topic. - Updated the `SparkStream` definition to receive the key and added the key to the `streamwriter` object. Frontend: - Integrated APScheduler to manage `StreamListener`. - Added `ResponseListener` to consume messages from the `analytics_response_topic` and process them. - Added APScheduler to manage StreamListener. - Added "ResponseListener" to consumer messages from analytics response_topic and process it.
-
- Sep 09, 2024
-
-
Waleed Akbar authored
Proto: - Added `Duration_s` field to the proto file. Frontend: - Added `SelectAnalyzer` logic. - Improved message formatting in the `create_analyzer_filter()` function. - Added a test case: `test_SelectAnalytics`. Backend: - Renamed the `RunSparkStreamer` method to `StartSparkStreamer`. - Updated the `StartRequestListener` method to return `(thread, stop_event)`. - Added a `StopRequestListener` method to stop the listener. Database: - Added the `select_with_filter` method with actual logic implementation. - Updated the `ConvertRowToAnalyzer` method to correctly read the `operation_mode` ENUM value.
-
- Sep 07, 2024
-
-
Waleed Akbar authored
- Updated the position of the `stop_event` parameter. - Added confirmation for pySpark checkpoint deletion. - Added a PySpark termination script to handle the `StopCollector` event.
-
Waleed Akbar authored
- Thresholds, window_size, window_slider is added in Frontend and Backend.
-
- Sep 06, 2024
-
-
Waleed Akbar authored
- UNSPECIFIED option added in the "AnalyzerOperationMode" enum as a best practice. - In SparkStreamer, changed the thresholds parameter from optional to compulsory.
-
- Sep 05, 2024
-
-
Waleed Akbar authored
- Updated function names.
-
Waleed Akbar authored
- In the BackendService.py: + Updated SparkStreamer call with new parameters. - In SparkStreaming.py: + Added 'GetAggregerations' and 'ApplyThresholds' methods + Added 'window_size', 'win_slide_duration', 'time_stamp_col' and 'thresholds' parameters. - Added new messages. - Updated the 'RunSparkStreamer' call wth new parameters.
-
- Sep 03, 2024
-
-
Waleed Akbar authored
- SparkStreamer write Stream to Kafka topic ANALYTICS_RESPONSE.
-
Waleed Akbar authored
- Added Kafka Request Listener to listen for requests from the Analytics Frontend. - Updated `SparkStreamer.py` to consume and process streams from VALUE topics, and further filter rows based on KPIs in the `input_kpis` list. - Updated the frontend TOPIC from `VALUE` to `ANALYTICS_REQUEST`. - Updated messages to keep the KPI UUID consistent with the one generated by the Telemetry Backend service.
-
Waleed Akbar authored
- Added CRDB secret. - Added test script. - Added `main` and `__init__.py` files. - Added Backend Service file. - Added Spark Streamer file (successfully consuming streams from Kafka Topics). - Added Tests folder and files. - Added requirements file. - Added ANALYTICS Kafka topics in `common.tools.kafka.variables` file.
-