Auto-generation of Annex C Tables
Annex C tables have proved to be exceedingly difficult and time-consuming to create, with a considerable risk of making fundamental errors in correctly following the logic of the requirement when determining exactly what constitutes a pass or a fail. As there should be a fundamental logic connecting the format of the tables with the wording and structure of the requirements, I experimented with seeing if I could auto-generate logically correct tables by using ChatGPT to transform the requirements into tables.
After an hour or two of my first attempt to use ChatGPT, I was astonished how good the results of processing a few (simpler) requirements proved to be. Fed with these plain text requirements:
-
6.2.2.1 Visually distinguishable display
Where ICT has RTT send and receive capabilities, displayed sent text shall be visually differentiated from, and separated from, received text.
-
6.6 Alternatives to video-based services Where ICT provides real-time video-based communication and also provides answering machine, auto attendant or interactive response facilities, the ICT should offer users a means to access the information and carry out the tasks related to these facilities:
- a) for audible information, without the use of hearing;
- b) for spoken commands, without the use of speech;
- c) for visual information, without the use of vision.
-
6.2.4 RTT responsiveness Where ICT utilises RTT input, that RTT input shall be transmitted to the ICT network or platform on which the ICT runs within 500 ms of the time that the smallest reliably composed unit of text entry is available to the ICT for transmission. Delays due to platform or network performance shall not be included in the 500 ms limit.
C.6.2.2.1 Visually distinguishable display
Type of assessment Inspection
Pre-conditions 1. ICT has RTT send and receive capabilities
Procedure 1. Displayed sent text shall be visually differentiated from, and separated from, received text
Result Pass: Check1 is true
Fail: Check1 is false
it produced the following Markdown formatted tables:
C.6.2.2.1 Visually distinguishable display
Type of assessment | Inspection |
---|---|
Pre-conditions | 1. ICT has RTT send and receive capabilities |
Procedure | 1. Displayed sent text shall be visually differentiated from, and separated from, received text |
Result | Pass: Check1 is true Fail: Check1 is false |
C.6.6 Alternatives to video-based services
Type of assessment | Inspection |
---|---|
Pre-conditions | None |
Procedure | 1. ICT provides real-time video-based communication and answering machine, auto attendant, or interactive response facilities 2. The ICT should offer users a means to access the information and carry out the tasks related to these facilities for: - a) audible information, without the use of hearing - b) spoken commands, without the use of speech - c) visual information, without the use of vision |
Result | Pass: Check1 is true Check2a is true Check2b is true Check2c is true Fail: Check1 is false OR Check2a is false OR Check2b is false OR Check2c is false |
C.6.2.4 RTT responsiveness
Type of assessment | Inspection |
---|---|
Pre-conditions | 1. ICT utilises RTT input |
Procedure | 1. RTT input shall be transmitted to the ICT network or platform within 500 ms of the time that the smallest reliably composed unit of text entry is available to the ICT for transmission 2. Delays due to platform or network performance shall not be included in the 500 ms limit |
Result | Pass: Check1 is true Check2 is true Fail: Check1 is false OR Check2 is false |
I have not yet taken the time to check that the logic has been correctly applied, but I would be surprised if there were mistakes (AI should probably be more reliable than error-prone humans :-)).
In the EN we have added some more guidance on how to run the tests, and I believe that this may well be required to be added to these ChatGPT generated tables. At the very least, running requirements through the same ChatGPT to generate tables should be a useful way to cross-chek any more human-intensive efforts that have already been done and will need to be done in the future.