Skip to content
Snippets Groups Projects
  1. Sep 13, 2024
    • Waleed Akbar's avatar
      Changes in Analytics · c1ee28db
      Waleed Akbar authored
      Backend:
      - Updated the `StartSparkStreamer` function call to send `Analyzer_uuid` as the key for messages produced on the Kafka topic.
      - Updated the `SparkStream` definition to receive the key and added the key to the `streamwriter` object.
      
      Frontend:
      - Integrated APScheduler to manage `StreamListener`.
      - Added `ResponseListener` to consume messages from the `analytics_response_topic` and process them.
      - Added APScheduler to manage StreamListener.
      - Added "ResponseListener" to consumer messages from analytics response_topic and process it.
      c1ee28db
  2. Sep 09, 2024
    • Waleed Akbar's avatar
      Changes in Analytics · 8ac130c1
      Waleed Akbar authored
      Proto:
      - Added `Duration_s` field to the proto file.
      
      Frontend:
      - Added `SelectAnalyzer` logic.
      - Improved message formatting in the `create_analyzer_filter()` function.
      - Added a test case: `test_SelectAnalytics`.
      
      Backend:
      - Renamed the `RunSparkStreamer` method to `StartSparkStreamer`.
      - Updated the `StartRequestListener` method to return `(thread, stop_event)`.
      - Added a `StopRequestListener` method to stop the listener.
      
      Database:
      - Added the `select_with_filter` method with actual logic implementation.
      - Updated the `ConvertRowToAnalyzer` method to correctly read the `operation_mode` ENUM value.
      8ac130c1
  3. Sep 07, 2024
    • Waleed Akbar's avatar
      Changes in Analytics Backend · d6dc91c2
      Waleed Akbar authored
      - Updated the position of the `stop_event` parameter.
      - Added confirmation for pySpark checkpoint deletion.
      - Added a PySpark termination script to handle the `StopCollector` event.
      d6dc91c2
    • Waleed Akbar's avatar
      Changes in Analytics Service. · 88d7a200
      Waleed Akbar authored
      - Thresholds, window_size, window_slider is added in Frontend and Backend.
      88d7a200
  4. Sep 06, 2024
    • Waleed Akbar's avatar
      Changes in Analytics. · 5eed4743
      Waleed Akbar authored
      - UNSPECIFIED option added in the "AnalyzerOperationMode" enum as a best practice.
      - In SparkStreamer, changed the thresholds parameter from optional to compulsory.
      5eed4743
  5. Sep 05, 2024
    • Waleed Akbar's avatar
      Changes in Analytics Backend. · faa6c684
      Waleed Akbar authored
      - Updated function names.
      faa6c684
    • Waleed Akbar's avatar
      Changes in Analytics Backend. · 21832189
      Waleed Akbar authored
      - In the BackendService.py:
           + Updated SparkStreamer call with new parameters.
      - In SparkStreaming.py:
         + Added 'GetAggregerations' and 'ApplyThresholds' methods
         + Added 'window_size', 'win_slide_duration', 'time_stamp_col' and 'thresholds' parameters.
      - Added new messages.
      - Updated the 'RunSparkStreamer' call wth new parameters.
      21832189
  6. Sep 03, 2024
    • Waleed Akbar's avatar
      Changes in Analytics Backend · e7e36f7d
      Waleed Akbar authored
      - SparkStreamer write Stream to Kafka topic ANALYTICS_RESPONSE.
      e7e36f7d
    • Waleed Akbar's avatar
      Changes in Telemetry · 067ace16
      Waleed Akbar authored
      - Added Kafka Request Listener to listen for requests from the Analytics Frontend.
      - Updated `SparkStreamer.py` to consume and process streams from VALUE topics, and further filter rows based on KPIs in the `input_kpis` list.
      - Updated the frontend TOPIC from `VALUE` to `ANALYTICS_REQUEST`.
      - Updated messages to keep the KPI UUID consistent with the one generated by the Telemetry Backend service.
      067ace16
    • Waleed Akbar's avatar
      Analytics Backend Initial Version · 0c5d2856
      Waleed Akbar authored
      - Added CRDB secret.
      - Added test script.
      - Added `main` and `__init__.py` files.
      - Added Backend Service file.
      - Added Spark Streamer file (successfully consuming streams from Kafka Topics).
      - Added Tests folder and files.
      - Added requirements file.
      - Added ANALYTICS Kafka topics in `common.tools.kafka.variables` file.
      0c5d2856
Loading