Skip to content
Snippets Groups Projects
  1. Sep 25, 2024
  2. Sep 24, 2024
    • Waleed Akbar's avatar
      Updates to Streamer Classes in Backend Analytics: · dafc8e87
      Waleed Akbar authored
      - Replaced `SparkStreamer` class with `DaskStreamer` class.
      - Updated methods to call the `DaskStreamer` class.
      - Improved messages and test files.
      - Added new package requirements.
      - Made minor changes to the frontend `requirements` file.
      dafc8e87
  3. Sep 22, 2024
    • Waleed Akbar's avatar
      Minor Changes in New Monitoring Services to Improve Overall Functionality · c0c517f0
      Waleed Akbar authored
      - Added new tests in the `scripts` folder.
      - Improved messages and test files across all services.
      Telemetry and Analytics Backend:
      - Added a `try` statement to handle Kafka message decoding errors.
      Telemetry Frontend and KPI Value API:
      - Downgraded `apscheduler` from `3.10.4` to `3.10.1` in the requirements file.
      Kafka:
      - Added a method to retrieve `KAFKA_SERVER_ADDRESS` from environment variables.
      Telemetry Backend:
      - Added a condition to allow the collector backend to run indefinitely.
      - Changed the extracted `KpiValue` timestamp format to `"%Y-%m-%dT%H:%M:%SZ"` to meet Apache Spark requirements.
      - Removed some unused packages from the `requirements.in` file.
      Deployment Script:
      - Added `"telemetry"` and `"analytics"` component names to the `TFS_COMPONENTS` variable in the `my_deploy.sh` file.
      c0c517f0
  4. Sep 20, 2024
  5. Sep 19, 2024
    • Waleed Akbar's avatar
      Changes to Analytics Backend: · 821e14f8
      Waleed Akbar authored
      - Corrected the name of `install_servicers()`.
      - Removed `stop_event` from the `RequestedListener` call.
      - Spark Streamer now writes to the `ALARMS` topic.
      - Improved messages and test files.
      821e14f8
  6. Sep 17, 2024
  7. Sep 13, 2024
    • Waleed Akbar's avatar
      Changes in Analytic, manifest files and deployment script · 2c1fc9c8
      Waleed Akbar authored
      Deployment Script
      - Analytics is added in new montioring component to be deployed with TFS.
      - condition is added in analytics component TFS deployment script.
      - Analytics show logs files are added.
      Manifest
      - modifed KAFKA_ADVERTISED_LISTENERS with internal kafka service address in kafka.yml
      - analyticsservice.yml added.
      Analytics
      - grpc service is added in backend
      - Docker file for backend and frontend is added.
      - main file is updaed
      - small changes in requirements.in files
      Kafka Variables
      - get_kafka_address method logic is updated.
      2c1fc9c8
    • Waleed Akbar's avatar
      Changes in Analytics · c1ee28db
      Waleed Akbar authored
      Backend:
      - Updated the `StartSparkStreamer` function call to send `Analyzer_uuid` as the key for messages produced on the Kafka topic.
      - Updated the `SparkStream` definition to receive the key and added the key to the `streamwriter` object.
      
      Frontend:
      - Integrated APScheduler to manage `StreamListener`.
      - Added `ResponseListener` to consume messages from the `analytics_response_topic` and process them.
      - Added APScheduler to manage StreamListener.
      - Added "ResponseListener" to consumer messages from analytics response_topic and process it.
      c1ee28db
  8. Sep 09, 2024
    • Waleed Akbar's avatar
      Changes in Analytics · 8ac130c1
      Waleed Akbar authored
      Proto:
      - Added `Duration_s` field to the proto file.
      
      Frontend:
      - Added `SelectAnalyzer` logic.
      - Improved message formatting in the `create_analyzer_filter()` function.
      - Added a test case: `test_SelectAnalytics`.
      
      Backend:
      - Renamed the `RunSparkStreamer` method to `StartSparkStreamer`.
      - Updated the `StartRequestListener` method to return `(thread, stop_event)`.
      - Added a `StopRequestListener` method to stop the listener.
      
      Database:
      - Added the `select_with_filter` method with actual logic implementation.
      - Updated the `ConvertRowToAnalyzer` method to correctly read the `operation_mode` ENUM value.
      8ac130c1
  9. Sep 07, 2024
    • Waleed Akbar's avatar
      Changes in Analytics Backend · d6dc91c2
      Waleed Akbar authored
      - Updated the position of the `stop_event` parameter.
      - Added confirmation for pySpark checkpoint deletion.
      - Added a PySpark termination script to handle the `StopCollector` event.
      d6dc91c2
    • Waleed Akbar's avatar
      Changes in Analytics · 88caa1cf
      Waleed Akbar authored
      **Backend:**
      - Added a dictionary to manage running analyzers.
      - Implemented logic to manage running analyzers.
      - Added the `TerminateAnalyzerBackend` method to handle analyzer termination.
      
      **Frontend:**
      - Modified and invoked the `PublishStopRequestOnKafka` method.
      88caa1cf
    • Waleed Akbar's avatar
      Changes in Analytics Service. · 88d7a200
      Waleed Akbar authored
      - Thresholds, window_size, window_slider is added in Frontend and Backend.
      88d7a200
  10. Sep 05, 2024
    • Waleed Akbar's avatar
      Changes in Analytics Backend. · 21832189
      Waleed Akbar authored
      - In the BackendService.py:
           + Updated SparkStreamer call with new parameters.
      - In SparkStreaming.py:
         + Added 'GetAggregerations' and 'ApplyThresholds' methods
         + Added 'window_size', 'win_slide_duration', 'time_stamp_col' and 'thresholds' parameters.
      - Added new messages.
      - Updated the 'RunSparkStreamer' call wth new parameters.
      21832189
  11. Sep 03, 2024
    • Waleed Akbar's avatar
      Changes in Telemetry · 067ace16
      Waleed Akbar authored
      - Added Kafka Request Listener to listen for requests from the Analytics Frontend.
      - Updated `SparkStreamer.py` to consume and process streams from VALUE topics, and further filter rows based on KPIs in the `input_kpis` list.
      - Updated the frontend TOPIC from `VALUE` to `ANALYTICS_REQUEST`.
      - Updated messages to keep the KPI UUID consistent with the one generated by the Telemetry Backend service.
      067ace16
    • Waleed Akbar's avatar
      Analytics Backend Initial Version · 0c5d2856
      Waleed Akbar authored
      - Added CRDB secret.
      - Added test script.
      - Added `main` and `__init__.py` files.
      - Added Backend Service file.
      - Added Spark Streamer file (successfully consuming streams from Kafka Topics).
      - Added Tests folder and files.
      - Added requirements file.
      - Added ANALYTICS Kafka topics in `common.tools.kafka.variables` file.
      0c5d2856
  12. Apr 18, 2024
  13. Feb 01, 2023
  14. Jul 26, 2022
    • Lluis Gifre Renom's avatar
      PathComp component: · ef420f23
      Lluis Gifre Renom authored
      Common:
      - added helper scripts
      
      FrontEnd:
      - corrected import paths
      - added missing parameters in test
      - refined creation of backend request
      ef420f23
  15. May 20, 2022
  16. Mar 08, 2022
  17. Feb 17, 2022
  18. Feb 16, 2022
  19. Nov 04, 2021
  20. Sep 23, 2021
  21. Jul 16, 2021
    • Lluis Gifre Renom's avatar
      Added testunit example: · 98fc013f
      Lluis Gifre Renom authored
      - monitoring/test_monitoring.py: provides a basic testing skeleton
      - monitoring/requirements.in/.txt: updated with pytest
      - monitoring/genproto.py: added instructions to import generated Python modules using "from . import ..." instead of "import ..."; otherwise, pytest and other possible modules doing imports might complain.
      - monitoring/monitoring_server.py: packed start_server and stop_server functionaity to simplify test definition, minor formattings, bug resolutions and improvements in package import.
      - monitoring/monitoring_client.py: minor formattings, bug resolutions, and improvements in package import.
      98fc013f
  22. Jun 14, 2021
  23. Jun 11, 2021
Loading