Merge branch 'docs/update-readme' into 'main'

Web-based data quality validation tool based on the ETSI data quality framework.

## Structure

## Overview

- `etsi_dq/` — Core analysis library (6 metrics)

- `api/` — FastAPI backend (14 REST API endpoints)

- `dashboard.py` — Streamlit frontend

This project provides:

## Run

Backend

- A Python core library for data quality assessment

- A FastAPI backend for dataset upload, analysis, results, and reports

- A React/Vite web dashboard

- Direct Python integration without the frontend or HTTP API

## Supported Metrics

The current implementation supports six data quality metrics:

- Completeness

- Accuracy

- Consistency

- Timeliness

- Reliability

- Uniqueness

## Project Structure

```text

.

├── api/                    # FastAPI backend

├── etsi_dq/                # Core data quality library

└── etsi-dq-frontend/       # React/Vite dashboard

```

The legacy Streamlit entry point has been removed. The current web frontend is implemented with React and Vite.

## Requirements

- Python 3.10 or later

- Node.js

- npm

## Backend Setup

From the repository root, create and activate a Python virtual environment.

### macOS/Linux

```bash

python3 -m venv .venv

source .venv/bin/activate

```

### Windows PowerShell

```powershell

python -m venv .venv

.venv\Scripts\Activate.ps1

```

Install the backend dependencies:

```bash

pip install \

  fastapi \

  uvicorn \

  python-multipart \

  pandas \

  numpy \

  pydantic \

  sqlalchemy \

  openpyxl \

  streamlit

```

> `streamlit` is currently required indirectly by the metric pre-check module. The active frontend is React/Vite; Streamlit is not used as the web frontend.

Start the backend from the repository root:

```bash

uvicorn api.main:app --reload --port 8000

Frontend

streamlit run dashboard.py

```

The backend is available at:

```text

http://localhost:8000

```

### API Documentation

- Swagger UI: http://localhost:8000/docs

- ReDoc: http://localhost:8000/redoc

- Health check: http://localhost:8000/health

## Frontend Setup

Open another terminal and move to the frontend directory:

```bash

cd etsi-dq-frontend

npm install

npm run dev

```

The frontend is available at:

```text

http://localhost:5173

```

For a production build:

```bash

npm run build

```

## Core Library Usage

The quality assessment pipeline can be called directly without the React frontend or HTTP API.

```python

import pandas as pd

from etsi_dq.pipeline import run_analysis

from etsi_dq.schemas import AnalysisRequest

df = pd.read_csv("data.csv")

request = AnalysisRequest(

    df=df,

    selected_metrics=[

        "Completeness",

        "Uniqueness",

    ],

    config={

        "comp_required_cols": [],

        "comp_missing_text": "?, N/A, null",

        "comp_granularity": "Dataset",

        "uniq_cols": [],

    },

)

response = run_analysis(request)

print(response.results)

print(response.messages)

```

Example result:

```python

{

    "Completeness": 95.0,

    "Uniqueness": 98.0,

}

```

The core library returns metric scores on a `0-100` scale.

## Direct Metric Usage

Individual metric functions can also be called directly.

```python

import pandas as pd

from etsi_dq.metrics.completeness import check_completeness

df = pd.read_csv("data.csv")

result = check_completeness(

    df,

    required_cols=["temperature", "humidity"],

    missing_text="?, N/A, null",

    granularity="Dataset",

)

print(result.score_percent)

print(result.missing_cells)

print(result.incomplete_rows)

print(result.col_scores)

```

Direct metric calls are useful when embedding validation into another system, such as an NGSI-LD broker pipeline.

## Basic API Flow

The backend API uses the following stateful flow:

```text

Upload dataset

    ↓

Receive dataset_id

    ↓

Run analysis

    ↓

Receive analysis_id

    ↓

Retrieve summary or metric details

```

Relevant endpoints include:

```text

POST /api/datasets/upload

POST /api/metrics/precheck

POST /api/analysis/run

GET  /api/results/{analysis_id}/summary

GET  /api/results/{analysis_id}/details/{metric}

GET  /api/reports

GET  /api/reports/{analysis_id}/pdf

GET  /api/reports/{analysis_id}/excel

```

For the complete request and response schemas, see the Swagger documentation at:

```text

http://localhost:8000/docs

```

## Supported Dataset Formats

The backend currently supports:

- CSV

- XLSX

- XLS

- ZIP files containing a supported dataset

## Data and Result Storage

Uploaded datasets are held in application memory.

Analysis report history is stored locally using:

- SQLite

- SQLAlchemy

Local databases and generated report files should not be committed to the repository.

## Development Checks

Build the frontend:

```bash

cd etsi-dq-frontend

npm run build

```

Run the frontend linter:

```bash

npm run lint

```

The repository may currently report pre-existing lint issues that are separate from the production build.

Compile-check the Python source:

```bash

python -m compileall api etsi_dq

```

## Tech Stack

- Frontend: Streamlit

- Backend: FastAPI + Uvicorn

- Analysis: Python, pandas, numpy

- API Docs: http://localhost:8000/docs

- Frontend: React, Vite

- Backend: FastAPI, Uvicorn

- Analysis: Python, pandas, NumPy

- Validation schemas: Pydantic

- Persistence: SQLite, SQLAlchemy

- Spreadsheet support: openpyxl

- Reports: PDF and Excel export