Commit 1deccbf6 authored by Martti Käärik's avatar Martti Käärik
Browse files

Mapping guidelines for importing OpenAPI and ASN.1 data definitions 

Moved from the wiki page UserScenarios (TR 103 119 material). Examples converted
to the current textual syntax and validated; notes where the TOP importers, the
libraries or the language differ from the guidelines.
parent b47a93c1

docs/import-mapping-guidelines.md

0 → 100644
+335 −0
Changes for docs/import-mapping-guidelines.md: 335 added lines, 0 removed lines.
Original line number Diff line number Diff line 
# Mapping guidelines for importing OpenAPI and ASN.1 data definitions



This document gives the conventions for mapping OpenAPI and ASN.1 data definitions to TDL data types. It is meant for developers of the TOP importers and of other tools that derive TDL data models. The conventions follow ETSI TR 103 119. The user documentation of the importers is on the wiki pages [OpenAPI](https://labs.etsi.org/rep/top/ide/-/wikis/OpenAPI) and [ASN.1](https://labs.etsi.org/rep/top/ide/-/wikis/ASN1).



The text was moved here from the wiki page UserScenarios. The examples are converted to the current textual syntax (`.tdltx`) and checked against the constraints of the TDL standard. Notes marked **Note (TOP)** describe where the TOP importers, the TDL libraries or the current language differ from the guidelines.



## OpenAPI



### Overview



The [OpenAPI Specification](https://swagger.io/specification/) (previously known as the Swagger Specification) is a notation for the specification of interfaces for RESTful web services. In addition to data-related information, OpenAPI specifications also include paths to identify resources by means of URLs, along with applicable operations, and corresponding request and response specifications. While these can be used to derive skeletons for test purposes and test descriptions as outlined in [ETSI EG 203 647 Methods for Testing and Specification (MTS); Methodology for RESTful APIs specifications and testing](https://www.etsi.org/deliver/etsi_eg/203600_203699/203647/01.01.01_60/eg_203647v010101p.pdf), the present document focuses solely on data-related information. Further information and guidelines regarding the use of OpenAPI for specification and testing at ETSI can be found in ETSI EG 203 647.



In addition to a set of primitive data types, OpenAPI provides means for defining structured data types. The specification is an extension of the [JSON schema](https://datatracker.ietf.org/doc/html/draft-bhutton-json-schema-01). Data type schemas may be defined inline or in a schemas object which enables reuse of those definitions. In the present document, only the latter is considered. Future editions may add guidelines for inline data definitions as well.



**Note (TOP):** the OpenAPI importer also imports inline schemas used as operation bodies, `allOf` as inheritance, `oneOf` with a discriminator as a subtype hierarchy, and request instances for the operations under `paths`.



### Built-in types



The built-in primitive types in OpenAPI are mapped to TDL according to the conventions in Table 1. The mapping relies on a TDL library of predefined types and constraints. As OpenAPI specifications may include format specifications for types, a generic constraint (`OpenAPIFormat`) with corresponding quantifiers may be used to capture this information in the derived TDL data model. Non-standard formats may be present in an OpenAPI specification as well. The generic constraint can be used for such formats as well.



Table 1: OpenAPI built-in type mapping



| OpenAPI | Type in TDL | Constraints | Formats and patterns |

|---|---|---|---|

| integer | Integer | OpenAPIFormat | int32, int64 |

| number | String | OpenAPIFormat | float, double |

| string | String | OpenAPIFormat | e-mail, password |

| boolean | Boolean | | |



**Note (TOP):** TR 103 119 writes the TDL type for `integer` as `integer`; the predefined TDL type is `Integer`.



### Structured types



A structured type in OpenAPI is either an `array` with member type declaration (`items` object) or an `object` with a set of properties (`properties` object). Consequently, the transformation of OpenAPI data types into TDL data types involves the following conventions:



- If the data type corresponds to one of the primitive data types within the OpenAPI library as indicated in Table 1, the 'Type' is mapped to the corresponding 'SimpleDataType' from Table 1.

- If the data type is an `object`, it is mapped to a 'StructuredDataType' with a 'name' corresponding to the name of the OpenAPI data type.

- If the data type is an `array`, it is mapped to a 'CollectionDataType' with a 'name' corresponding to the name of the OpenAPI data type. The data type indicated in the `items` object is mapped to the corresponding 'DataType' as the 'itemType' of the 'CollectionDataType'. If `minItems` and `maxItems` are specified for the `array`, the corresponding predefined constraints need to be added to the 'CollectionDataType'.

- Each item in the `properties` object of the `object` is mapped to a 'Member' within the corresponding 'StructuredDataType' with a 'name' corresponding to the property. If a 'Member' with the same 'name' exists, no action is taken. All 'Member's are to be marked as optional, except for 'Member's corresponding to properties which are listed in the `required` array of the `object`. The 'dataType' of the 'Member' corresponds to:

  - a new 'DataType' corresponding to the `type` of the property with a 'name' prefixed with the 'name' of the containing 'StructuredDataType', in case a property is of `type` `object`;

  - a 'SimpleDataType' corresponding to the `type` of the property, in case the 'SimpleDataType' is one of the predefined 'DataType's within the OpenAPI library as indicated in Table 1.

- Nested `object`s are transformed according to the conventions above.

- If the property contains an `enum` array, it is mapped to an 'EnumDataType' with a 'name' corresponding to the name of the property. The items contained in the `enum` array are mapped to 'SimpleDataInstance's of the 'EnumDataType' that are contained in the 'EnumDataType'.

- Corresponding 'DataElementMapping's are created for the defined data types. 'DataElementMapping's for 'DataType's derived from anonymous (inline) data types are not created. The 'DataElementMapping's may include target platform mappings in addition to the source mappings to the OpenAPI specifications.



**Note (TOP):** the TDL standard requires a 'DataElementMapping' of a 'StructuredDataType' to map all of its members. The examples below include the member mappings, which TR 103 119 omits.



### Example



Consider the OpenAPI snippet in Figure 1 and the derived TDL data model in Figure 2. Corresponding 'StructuredDataType's are created for both the `Library` and `LibraryBook` data types, as well as for the nested anonymous `object`s and `array`s, which are prefixed with `Library___` and `LibraryBook___` accordingly. This would also apply to additional anonymous `object`s and `array`s nested further within the `object`s. The 'dataType's for the corresponding 'Member's are then set accordingly. Finally, both source and target (for Java in this example) 'DataElementMapping's are provided.



Figure 1: OpenAPI example including nested anonymous data types



```yaml

components:

  schemas:

    LibraryBook:

      type: object

      properties:

        title:

          type: string

        authors:

          type: array

          items:

            type: string

        reviewers:

          type: array

          items:

            type: string

    Library:

      type: object

      properties:

        address:

          type: string

        books:

          type: array

          items:

            $ref: '#/components/schemas/LibraryBook'

```



Figure 2: Corresponding flattened TDL definitions for Figure 1



```

Structure LibraryBook (

    optional String title,

    optional LibraryBook___authors authors,

    optional LibraryBook___reviewers reviewers

)

Collection LibraryBook___authors of String

Collection LibraryBook___reviewers of String



Structure Library (

    optional String address,

    optional Library___books books

)

Collection Library___books of LibraryBook



Use "mapping_conventions.yaml" as SOURCE_MAPPING

Use "generated/java" as TARGET_MAPPING

Map LibraryBook to "#/components/schemas/LibraryBook" in SOURCE_MAPPING as LibraryBook_SOURCE_MAPPING {

    title -> "title",

    authors -> "authors",

    reviewers -> "reviewers"

}

Map LibraryBook to "LibraryBook" in TARGET_MAPPING as LibraryBook_TARGET_MAPPING {

    title -> "title",

    authors -> "authors",

    reviewers -> "reviewers"

}

Map Library to "#/components/schemas/Library" in SOURCE_MAPPING as Library_SOURCE_MAPPING {

    address -> "address",

    books -> "books"

}

Map Library to "Library" in TARGET_MAPPING as Library_TARGET_MAPPING {

    address -> "address",

    books -> "books"

}

```



## ASN.1



### Overview



[ASN.1 (Abstract Syntax Notation One), Recommendation ITU-T X.680](https://www.itu.int/itu-t/recommendations/rec.aspx?rec=x.680) is a standardized language for the specification of data types and data structures. As the name implies, the specifications are abstract and therefore independent of a specific target platform. The specifications provide the information about the structure and encoding of the data, which can be processed by generators or compilers to produce data type implementations for the desired target language and platform, including codecs for encoding and decoding the data for transmission. While TDL is not concerned with the encoding and decoding at the specification level, in many cases the test execution platform needs to include codecs for the operationalisation of the tests.



When ASN.1 specifications are imported in TDL, the level of detail may vary from the bare essentials, including the data types only, to including additional constraints, and even encoding information (where applicable). The additional information can be utilised for early validation of the TDL specifications. While it is also possible to specify constant values in ASN.1, these are not covered in the guidelines at present.



**Note (TOP):** the TTF 045 converter, used through the protocol server, extends these guidelines to information object classes and parameterised types (ITU-T X.681 and X.683). Its requirements are in the `requirements` folder of the `ttf-045` repository.



### Built-in types



ASN.1 includes a set of built-in types, some of which are mapped to TDL according to the conventions in Table 2. The mapping relies on a TDL library of predefined types and constraints. The generic constraints (`ASN1String`, `ASN1DateTime`, `ASN1Real`, `ASN1ObjectIdentifier`) may be used to provide additional patterns for the contents of data instances of the corresponding data types to facilitate validation. Alternatively, a tool may implement implicit validation based on the underlying types.



Table 2: ASN.1 built-in type mapping



| ASN.1 type | Type in TDL | Constraint | Pattern | Examples |

|---|---|---|---|---|

| BIT STRING | BITSTRING | ASN1String | `[0\|1]+'B` | `"1101'B"`, also named bits |

| OCTET STRING | OCTETSTRING | ASN1String | `[A-F\|0-9]+'H` | `"A3B2'H"`, `"10010'B"` |

| BMPString | BMPString | ASN1String | 16-bit characters | |

| IA5String | IA5String | ASN1String | 8-bit ASCII | `"Hallo"` |

| GeneralString | GeneralString | ASN1String | all graphic and character sets, SPACE, DELETE | |

| NumericString | NumericString | ASN1String | `[0-9, SPACE]+` | `"34 8"` |

| PrintableString | PrintableString | ASN1String | `[a-z,A-Z,'()+,-.?:/=,SPACE]` | `"Black, Blue + Brown"` |

| TeletexString | TeletexString | ASN1String | CCITT T.101 | |

| T61String | T61String | ASN1String | CCITT T.101 | |

| UniversalString | UniversalString | ASN1String | ISO 10646 | |

| UTF8String | UTF8String | ASN1String | ASCII and control characters | |

| VideotexString | VideotexString | ASN1String | CCITT T.100, T.101 | |

| VisibleString, ISO646String | VisibleString | ASN1String | printing ASCII | |

| UTCTime | UTCTime | ASN1DateTime | `YYMMDDhhmm[ss]Z` | `"991231235959+0200"` |

| GeneralizedTime | GeneralizedTime | ASN1DateTime | `YYYYMMDDHH[MM[SS[.fff]]]Z` ([ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html)) | `"20200425175522.214+0200"` |

| DATE | DATE | ASN1DateTime | `YYYY-MM-DD` | `"1636-09-18"` |

| TIME-OF-DAY | TimeOfDay | ASN1DateTime | `hh:mm:ss` | `"18:30:23"` |

| DATE-TIME | DateTime | ASN1DateTime | `YYYY-MM-DDThh:mm:ss` | `"2000-11-22T18:30:23"` |

| INTEGER | Integer | | | |

| REAL | String | ASN1Real | | |

| BOOLEAN | Boolean | | | |

| NULL | Null | | | |

| OBJECT IDENTIFIER | ObjectIdentifier | ASN1ObjectIdentifier | | `id-ssp OBJECT IDENTIFIER ::= { itu-t (0) identified-organization (4) etsi (0) smart-secure-platform (3666) part1 (1) }` |

| RELATIVE-OID | ObjectIdentifier | ASN1ObjectIdentifier | | `Relative_id_ssp RELATIVE-OID ::= { smart-secure-platform (3666) part1 (1) }` |



**Notes (TOP):**



- TR 103 119 names the TDL type for DATE `Date`; the `ASN1` library declares it as `DATE`. The table follows the library.

- The `ASN1` library declares `Real {ASN1Real}` for REAL, while TR 103 119 and the TTF 045 converter use `String` with `ASN1Real`. See finding 7 in `docs/code-findings-2026-10.md`.

- The string and time types of the `ASN1` library do not extend `String`. A string literal assigned to them must therefore be typed, for example `"Sophia-Antipolis, France" {UTF8String}`.



### Structured and other types



The transformation of ASN.1 data types into TDL data types involves the following conventions:



- If the data type corresponds to one of the predefined 'DataType's within the ASN.1 library as indicated in Table 2, the ASN.1 'Type' is mapped to the corresponding TDL 'SimpleDataType' from Table 2. For the supported ASN.1 types the following additional restriction applies:

  - The NULL type is only used in the scope of a CHOICE type, where the corresponding alternative is activated if there is no information.

- If the ASN.1 data type is a 'SequenceType', a 'SetType', or a 'ChoiceType', it is mapped to a 'StructuredDataType' with a 'name' corresponding to the name of the ASN.1 data type. In the case of 'ChoiceType', a 'Constraint' with the predefined `union` 'ConstraintType' is applied to the corresponding 'StructuredDataType', and all 'Member's are marked as optional.

- If the ASN.1 data type is a 'SequenceOfType' or a 'SetOfType', it is mapped to a 'CollectionDataType' with a 'name' corresponding to the name of the ASN.1 data type. The 'Type' indicated for the 'SequenceOfType' or 'SetOfType' is mapped to the corresponding 'DataType' as the 'itemType' of the 'CollectionDataType'.

- Each 'ComponentType' in the 'ComponentTypeList' of the 'SequenceType', 'SetType', or 'ChoiceType' is mapped to a 'Member' within the corresponding 'StructuredDataType' with a 'name' corresponding to the 'identifier' of the 'ComponentType'. If a 'Member' with the same 'name' exists, no action is taken. The 'dataType' of the 'Member' corresponds to:

  - a new 'DataType' corresponding to the 'Type' of the 'ComponentType' with a 'name' prefixed with the 'name' of the containing 'StructuredDataType', in case a 'ComponentType' is directly contained within another 'ComponentType';

  - a 'SimpleDataType' corresponding to the 'Type' of the 'ComponentType', in case the 'SimpleDataType' is one of the predefined 'DataType's within the ASN.1 library as indicated in Table 2.

- Nested 'ComponentType's are transformed according to the conventions above.

- If the ASN.1 data type is an 'EnumeratedType', it is mapped to an 'EnumDataType' with a 'name' corresponding to the name of the ASN.1 data type. The contained 'EnumerationItem's are mapped to 'SimpleDataInstance's of the 'EnumDataType' that are contained in the 'EnumDataType'. There are no guidelines for 'NamedNumber's at present.

- Corresponding 'DataElementMapping's are created for the defined data types. 'DataElementMapping's for 'DataType's derived from anonymous (inline) data types are not created. The 'DataElementMapping's may include target platform mappings in addition to the source mappings to the ASN.1 specifications.



For the following built-in types not mentioned in Table 2, the transformations to TDL types are done as follows:



- UnrestrictedCharacterStringType: replace the CHARACTER STRING type with its associated type, obtained by expanding inner subtyping in the associated type of the CHARACTER STRING type (see clause 44.5 of Recommendation ITU-T X.680), and map it to the corresponding TDL type.

- EmbeddedPDVType: replace any EMBEDDED PDV type with its associated type, obtained by expanding inner subtyping in the associated type of the EMBEDDED PDV type (see clause 36.5 of Recommendation ITU-T X.680), and map it to the corresponding TDL type.

- ExternalType: replace the EXTERNAL type with its associated type, obtained by expanding inner subtyping in the associated type of the EXTERNAL type (see clause 37.5 of Recommendation ITU-T X.680), and map it to the corresponding TDL type.

- InstanceOfType: replace the INSTANCE OF type with its associated type, obtained by substituting INSTANCE OF DefinedObjectClass by its associated ASN.1 type (see clause C.7 of [Recommendation ITU-T X.681](https://www.itu.int/itu-t/recommendations/rec.aspx?rec=x.681)), and map all ASN.1 types to their TDL types according to Table 2.



**Notes (TOP):**



- The TTF 045 converter does not expand these types. It maps EXTERNAL, EMBEDDED PDV, ANY and open types to `OCTETSTRING`, and CHARACTER STRING to `VisibleString`.

- The TTF 045 converter names nested anonymous types with a single underscore (`Rocket_speed`) instead of the triple underscore used here and by the built-in importers.



### Examples



The first example, in Figures 3 and 4, uses the `NodeDescriptor` and related type definitions from [ETSI TS 103 666-1, V15.0.0: Smart Secure Platform (SSP); Part 1: General characteristics (Release 15)](https://www.etsi.org/deliver/etsi_ts/103600_103699/10366601/15.00.00_60/ts_10366601v150000p.pdf). A corresponding 'StructuredDataType' is created, including derived 'DataType's for the nested `aNode` anonymous 'ContentType', as well as for the `aLink`, `aFile`, and `aDirectory` anonymous 'ContentType's nested further within the `aNode` 'ContentType'. The 'dataType's for the corresponding 'Member's are then set accordingly. The 'DataType' for the `NodeIdentity` type as well as the derived 'DataType' for the `aNode` 'Member' are assigned a 'Constraint' with the `union` 'ConstraintType'.



Figure 3: ASN.1 example including nested anonymous data types



```asn1

NodeDescriptor ::= SEQUENCE

{

    aNodeName NodeName,  -- Node name

    aShortName UUID,  -- Short node name

    aNode CHOICE

    {

        aLink SEQUENCE

        {

            aLinkedFileIdentity NodeIdentity,  -- Identity of the linked SSP file

            aLinkedFileSize FileSize  -- Size of the linked SSP file

        },

        aFile SEQUENCE

        {

            aFileSize FileSize  -- Size of the SSP file

        },

        aDirectory SEQUENCE

        {

        }

    },

    aMetaData SEQUENCE OF MetaDatum OPTIONAL,  -- Optional meta data

    aACL SET OF AccessControl OPTIONAL  -- Access Control List attribute

}



/* Node identity */

NodeName ::= UTF8String (SIZE(1..16))  -- node name encoded in UTF-8

NodeReference ::= SEQUENCE (SIZE(1..6)) OF NodeName  -- pathname and node name



NodeIdentity ::= CHOICE

{

    aShortName UUID,  -- UUID of file reference using absolute pathname

    aNodeReference NodeReference  -- Node reference

}

```



Figure 4: Corresponding TDL definitions (excerpt) for Figure 3



```

Structure NodeDescriptor (

    NodeName aNodeName,

    UUID aShortName,

    NodeDescriptor___aNode aNode,

    optional NodeDescriptor___aMetaData aMetaData,

    optional NodeDescriptor___aACL aACL

)

Structure NodeDescriptor___aNode {union} (

    optional NodeDescriptor___aNode___aLink aLink,

    optional NodeDescriptor___aNode___aFile aFile,

    optional NodeDescriptor___aNode___aDirectory aDirectory

)

Collection NodeDescriptor___aMetaData of MetaDatum

Collection NodeDescriptor___aACL of AccessControl



Structure NodeIdentity {union} (

    optional UUID aShortName,

    optional NodeReference aNodeReference

)

Collection NodeReference of NodeName

Structure NodeDescriptor___aNode___aLink (

    NodeIdentity aLinkedFileIdentity,

    FileSize aLinkedFileSize

)

Structure NodeDescriptor___aNode___aFile (

    FileSize aFileSize

)

Structure NodeDescriptor___aNode___aDirectory ( )

```



The second example, in Figures 5 and 6, shows the `Library` and `Document` data types. Corresponding 'StructuredDataType's are created for both, as well as for the nested anonymous 'ContentType's, which are prefixed with `Library___` and `Document___` accordingly. This would also apply to additional anonymous 'ContentType's nested further within the 'ContentType's. The 'dataType's for the corresponding 'Member's are then set accordingly. The derived 'DataType' `Document___number` for the `number` 'ContentType' of type CHOICE is assigned a 'Constraint' with the `union` 'ConstraintType'. The ASN.1 DEFAULT value of `address` is carried over as the default value of the corresponding 'Member'. Finally, source 'DataElementMapping's are provided.



**Note (TOP):** TR 103 119 states that default values are not present in the derived TDL data model, as TDL did not support default values for type definitions. The current TDL textual syntax supports default values for members, so the example uses them. A data instance that provides default values, overridden where it is used, remains an alternative.



Figure 5: ASN.1 example including nested anonymous data types



```asn1

Library ::= SEQUENCE {

   address   UTF8String DEFAULT "Sophia-Antipolis, France",

   documents SEQUENCE OF Document

}



Document ::= SEQUENCE {

   title     UTF8String (SIZE(1..128)),

   status    ENUMERATED {draft, published, historical},

   authors   SEQUENCE OF UTF8String,

   number    CHOICE {

      es     INTEGER,

      eg     INTEGER,

      tr     INTEGER

   }  OPTIONAL,

   updated   DATE

}

```



Figure 6: Corresponding flattened TDL definitions (excerpt) for Figure 5



```

Structure Library (

    UTF8String address = "Sophia-Antipolis, France" {UTF8String},

    Library___documents documents

)

Structure Document (

    UTF8String title,

    Document___status status,

    Document___authors authors,

    optional Document___number number,

    DATE updated

)

Collection Library___documents of Document

Collection Document___authors of UTF8String

Structure Document___number {union} (

    optional Integer es,

    optional Integer eg,

    optional Integer tr

)

Enumerated Document___status {

    Document___status draft,

    Document___status published,

    Document___status historical

}

Use "example-1-library.asn" as SOURCE_MAPPING

Map Library to "Library" in SOURCE_MAPPING as Library_MAPPING {

    address -> "address",

    documents -> "documents"

}

Map Document to "Document" in SOURCE_MAPPING as Document_MAPPING {

    title -> "title",

    status -> "status",

    authors -> "authors",

    number -> "number",

    updated -> "updated"

}

```



The examples import the `TDL` and `ASN1` standard libraries. Figures 4 and 6 are excerpts: the types referenced but not shown (such as `UUID`, `FileSize`, `MetaDatum`, `AccessControl` and `NodeName`) are defined elsewhere in the respective specifications.