Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
# STF584
ETSI STF584 - oneM2M architecture for AI
This repository contains the source code of ETSI STF584 Use Case 3. It uses the OpenMTC middleware compliant with the
OneM2M standard.
It consists of a a Common Service Function (CSF) which was included in a Middle Node Common Service
Entity (CSE) and two applications, one for generating data and another for receiving this data after having been
processed by our CSF.
## Installation and startup
Run the [build.sh](build.sh) shell script, this will download OpenMTC source code from its repository, apply a fix for some files
which have since been affected by dependencies code changes, generate OpenMTC's docker images and, finally, remove the
files previously downloaded as they are no longer necessary.
After having the docker images generated, just run the [docker compose](docker-compose.yml) with `docker-compose up -d` and both the gateway
and the 2 applications will be running in the background, each in separate containers. The logs can be viewed with the
following commands:
* Gateway: `docker-compose logs -f gateway`
* Device Application: `docker-compose logs -f device`
* NLP Application: `docker-compose logs -f nlp`
## Common Service Function (CSF)
The CSF is included in the [clean_text_handler](clean_text_handler) directory, which is mapped in the docker-compose to the correct
location inside the gateway container. The main code is inside the [\_\_init\_\_.py](clean_text_handler/__init__.py)
file, while the [utils.py](clean_text_handler/utils.py) file
contains helper functions for the text clean process.
The configuration file for the gateway is the [config_gateway.json](config-gateway.json) file. Please note the following
data required to enable the new CSF:
```json
{
...,
"plugins": {
...,
"openmtc_cse": [,
...,
{
"name": "CleanTextHandler",
"package": "openmtc_cse.plugins.clean_text_handler",
"disabled": false,
"config": {
"labels_input": [
"to_clean"
],
"labels_output": [
"cleaned"
],
"stopwords_langs": [
"english"
]
}
}
]
}
}
```
## Applications
### Device Application
The device application code is present in the [device_ae.py](Device_AE/src/device_ae/device_ae.py) file. It starts by
attempting to load a [csv file](Device_AE/example.csv) with example messages and proceeds to register the necessary
resources for the application to run. The data source can easily be edited at will inside the `load_data` method
and triggering of publishing data can be edited inside the `get_data` method.
The configuration file for this application is present [here](Device_AE/config.json).
### NLP Application
The NLP application code is present in the [nlp_ae.py](NLP_AE/src/nlp_ae/nlp_ae.py) file. The [models](NLP_AE/models)
and [tokenizers](NLP_AE/tokenizers) folders contain the pre trained model for occurrence detections. The application
starts by loading the model and then registers the necessary resources for it to run. The classification and output
when a message is received is present inside the `handle_processed_text` method.
The configuration file for this application is present [here](NLP_AE/config.json).
## Credits
Ubiwhere \
Dataset: The project was based on the dataset provided in the Kaggle competition [Real or Not? NLP with Disaster Tweets](https://www.kaggle.com/c/nlp-getting-started)