ETSI's Bug Tracker - Part 01: TTCN-3 Core Language
View Issue Details

ID | Project | Category | View Status | Date Submitted | Last Update
0007448 | Part 01: TTCN-3 Core Language | New Feature | public | 08-07-2016 12:14 | 12-12-2016 18:22

Reporter | Gyorgy Rethy |
Assigned To | Gyorgy Rethy |
Priority | high | Severity | major | Reproducibility | have not tried
Status | closed | Resolution | fixed |
Platform |  | OS |  | OS Version |
Product Version | v4.8.1 (published 2016-07) |
Target Version | v4.9.1 (published 2017-05) | Fixed in Version | v4.9.1 (published 2017-05) |
Clause Reference(s) | Many
Source (company - Author) | L.M.Ericsson

Summary | 0007448: Allowing multiple encodings for TTCN-3 types

Description

Today the type of encoding of the test data is not fixed in the specification in all cases. The simplest example is that IoT common service level data is specified at the abstract level and can be encoded by XML or by JSON, in the discretion of the vendor. The abstract data structure can well be defined in TTCN-3, however the different possible types of encoding is not supported by the language.
Therefore TTCN-3 shall, as well, be able to encode and decode the same abstract data structures in different encodings.
We ask to resolve the CR with urgency, as oneM2M is currently writing the conformance test suite, where this problem has been raised. The project will finish by end of the year, that means they need tool support ASAP.
Proposed solution can be found in the attached file.

Steps To Reproduce |
Additional Information |
Tags | No tags attached.

Relationships

related to | 0007445 | closed | Tomas Urban | Part 01: TTCN-3 Core Language | usage of encode/variant attributes should be enhanced
related to | 0007353 | closed | Gyorgy Rethy | Part 01: TTCN-3 Core Language | attributes should be attachable to their definition only
related to | 0007451 | closed | Gyorgy Rethy | Part 01: TTCN-3 Core Language | Multiple occurrence of an attribute
related to | 0007459 | closed | Gyorgy Rethy | Part 01: TTCN-3 Core Language | the overwriting rules for attributes and the examples should be written more consistent
related to | 0007524 | closed | Tomas Urban | Part 06: TTCN-3 Control Interface | TCI extension for multiple attributes

Attached Files

Allowing multiple encodings of TTCN PA3.docx (30,489) 08-07-2016 12:16
http://oldforge.etsi.org/mantis/file_download.php?file_id=3412&type=bug
CR7448-1.docx (167,050) 16-08-2016 16:21
http://oldforge.etsi.org/mantis/file_download.php?file_id=3464&type=bug
CR7448-V2.docx (118,202) 17-08-2016 15:15
http://oldforge.etsi.org/mantis/file_download.php?file_id=3476&type=bug
CR7448-3.docx (170,626) 18-08-2016 09:43
http://oldforge.etsi.org/mantis/file_download.php?file_id=3486&type=bug
CR7448-4.docx (129,174) 18-08-2016 15:28
http://oldforge.etsi.org/mantis/file_download.php?file_id=3492&type=bug
CR7448-5.docx (129,427) 19-08-2016 13:24
http://oldforge.etsi.org/mantis/file_download.php?file_id=3499&type=bug
CR7448-6.docx (127,120) 19-08-2016 14:35
http://oldforge.etsi.org/mantis/file_download.php?file_id=3503&type=bug
CR7448-7.docx (185,850) 16-11-2016 11:21
http://oldforge.etsi.org/mantis/file_download.php?file_id=3532&type=bug
CR7448-8.docx (132,164) 16-11-2016 15:09
http://oldforge.etsi.org/mantis/file_download.php?file_id=3541&type=bug
CR7448-9.docx (133,532) 17-11-2016 09:50
http://oldforge.etsi.org/mantis/file_download.php?file_id=3547&type=bug

Issue History
Date Modified | Username | Field | Change
08-07-2016 12:14 | Gyorgy Rethy | New Issue |
08-07-2016 12:16 | Gyorgy Rethy | File Added: Allowing multiple encodings of TTCN PA3.docx |
11-07-2016 10:27 | Jacob Wieland - Spirent | Relationship added | related to 0007445
11-07-2016 10:28 | Jacob Wieland - Spirent | Relationship added | related to 0007353
18-07-2016 09:59 | Jens Grabowski | Assigned To | => Tomas Urban
18-07-2016 09:59 | Jens Grabowski | Status | new => assigned
19-07-2016 16:10 | Tomas Urban | Note Added: 0013993 |
19-07-2016 16:11 | Tomas Urban | Assigned To | Tomas Urban => Gyorgy Rethy
19-07-2016 16:11 | Tomas Urban | Status | assigned => feedback
20-07-2016 12:03 | Gyorgy Rethy | Note Added: 0014005 |
20-07-2016 12:03 | Gyorgy Rethy | Status | feedback => assigned
20-07-2016 12:04 | Gyorgy Rethy | Note Edited: 0014005 | bug_revision_view_page.php?bugnote_id=14005#r302
20-07-2016 12:07 | Gyorgy Rethy | Note Edited: 0014005 | bug_revision_view_page.php?bugnote_id=14005#r303
20-07-2016 12:07 | Gyorgy Rethy | Assigned To | Gyorgy Rethy => Tomas Urban
20-07-2016 12:09 | Gyorgy Rethy | Note Edited: 0014005 | bug_revision_view_page.php?bugnote_id=14005#r304
20-07-2016 16:07 | Tomas Urban | Note Added: 0014012 |
20-07-2016 16:08 | Tomas Urban | Assigned To | Tomas Urban => Gyorgy Rethy
20-07-2016 16:08 | Tomas Urban | Status | assigned => feedback
21-07-2016 08:25 | Tomas Urban | Note Edited: 0014012 | bug_revision_view_page.php?bugnote_id=14012#r306
21-07-2016 09:46 | Gyorgy Rethy | Note Added: 0014024 |
21-07-2016 09:46 | Gyorgy Rethy | Status | feedback => assigned
21-07-2016 09:46 | Gyorgy Rethy | Assigned To | Gyorgy Rethy => Tomas Urban
02-08-2016 10:50 | Martin Hauch | Note Added: 0014041 |
02-08-2016 13:03 | Gyorgy Rethy | Note Added: 0014042 |
02-08-2016 13:46 | Gyorgy Rethy | Note Added: 0014043 |
02-08-2016 13:46 | Gyorgy Rethy | Note Edited: 0014042 | bug_revision_view_page.php?bugnote_id=14042#r312
16-08-2016 16:21 | Tomas Urban | File Added: CR7448-1.docx |
16-08-2016 16:24 | Tomas Urban | Note Added: 0014114 |
16-08-2016 16:24 | Tomas Urban | Assigned To | Tomas Urban => Jens Grabowski
16-08-2016 16:24 | Tomas Urban | Status | assigned => confirmed
16-08-2016 16:27 | Tomas Urban | Relationship added | related to 0007451
17-08-2016 11:05 | Jacob Wieland - Spirent | Target Version | => v4.9.1 (published 2017-05)
17-08-2016 15:15 | Jens Grabowski | File Added: CR7448-V2.docx |
17-08-2016 15:17 | Jens Grabowski | Note Added: 0014134 |
17-08-2016 15:17 | Jens Grabowski | Assigned To | Jens Grabowski => Tomas Urban
17-08-2016 15:17 | Jens Grabowski | Status | confirmed => assigned
18-08-2016 09:43 | Tomas Urban | File Added: CR7448-3.docx |
18-08-2016 09:48 | Tomas Urban | Note Added: 0014150 |
18-08-2016 09:48 | Tomas Urban | Assigned To | Tomas Urban => KristÃ³f Szabados
18-08-2016 09:48 | Tomas Urban | Status | assigned => confirmed
18-08-2016 09:49 | Tomas Urban | Relationship added | related to 0007459
18-08-2016 15:28 | Jacob Wieland - Spirent | File Added: CR7448-4.docx |
18-08-2016 15:29 | Jacob Wieland - Spirent | Note Added: 0014158 |
18-08-2016 16:44 | Gyorgy Rethy | Note Added: 0014161 |
19-08-2016 13:24 | KristÃ³f Szabados | File Added: CR7448-5.docx |
19-08-2016 13:26 | KristÃ³f Szabados | Note Added: 0014178 |
19-08-2016 13:26 | KristÃ³f Szabados | Assigned To | KristÃ³f Szabados => Tomas Urban
19-08-2016 13:26 | KristÃ³f Szabados | Status | confirmed => assigned
19-08-2016 14:35 | KristÃ³f Szabados | File Added: CR7448-6.docx |
19-08-2016 14:36 | KristÃ³f Szabados | Note Added: 0014183 |
14-11-2016 15:38 | Jens Grabowski | Note Added: 0014230 |
16-11-2016 11:01 | Tomas Urban | File Added: CR7448-7.docx |
16-11-2016 11:02 | Tomas Urban | Note Added: 0014259 |
16-11-2016 11:02 | Tomas Urban | Assigned To | Tomas Urban => Jacob Wieland - Spirent
16-11-2016 11:02 | Tomas Urban | Status | assigned => confirmed
16-11-2016 11:15 | Tomas Urban | Relationship added | related to 0007524
16-11-2016 11:21 | Tomas Urban | File Deleted: CR7448-7.docx |
16-11-2016 11:21 | Tomas Urban | File Added: CR7448-7.docx |
16-11-2016 15:09 | Jacob Wieland - Spirent | File Added: CR7448-8.docx |
16-11-2016 15:10 | Jacob Wieland - Spirent | Note Added: 0014274 |
16-11-2016 15:10 | Jacob Wieland - Spirent | Status | confirmed => resolved
16-11-2016 15:10 | Jacob Wieland - Spirent | Fixed in Version | => v4.9.1 (published 2017-05)
16-11-2016 15:10 | Jacob Wieland - Spirent | Resolution | open => fixed
16-11-2016 15:10 | Jacob Wieland - Spirent | Assigned To | Jacob Wieland - Spirent => Gyorgy Rethy
17-11-2016 09:50 | KristÃ³f Szabados | Assigned To | Gyorgy Rethy => KristÃ³f Szabados
17-11-2016 09:50 | KristÃ³f Szabados | Status | resolved => feedback
17-11-2016 09:50 | KristÃ³f Szabados | Resolution | fixed => reopened
17-11-2016 09:50 | KristÃ³f Szabados | File Added: CR7448-9.docx |
17-11-2016 09:52 | KristÃ³f Szabados | Note Added: 0014288 |
17-11-2016 09:52 | KristÃ³f Szabados | Assigned To | KristÃ³f Szabados => Jens Grabowski
17-11-2016 09:52 | KristÃ³f Szabados | Status | feedback => assigned
17-11-2016 11:53 | Jens Grabowski | Note Added: 0014300 |
17-11-2016 11:54 | Jens Grabowski | Status | assigned => resolved
17-11-2016 11:54 | Jens Grabowski | Resolution | reopened => fixed
17-11-2016 11:54 | Jens Grabowski | Assigned To | Jens Grabowski => Gyorgy Rethy
12-12-2016 18:22 | Gyorgy Rethy | Note Added: 0014395 |
12-12-2016 18:22 | Gyorgy Rethy | Status | resolved => closed

Notes

(0013993)
Tomas Urban
19-07-2016 16:10

The part of the proposal concerning the way how different encodings and associated variants are specified (multiple encode statements, variants with an encode prefix and a colon) is acceptable without any objections.
In the case of selection the actual encoding the STF proposes to use the static approach. We assume that there's (one or more global PDU) containing potentially complicated structure of elements with multiple encodings and variants. The desired encoding and the associated set of variants will be selected by creating an alias type (or template) with a dedicated "filtering" attribute. Please check the following example:
type Payload {
XSD.Integer foo,
XSD.Float bar
} with {
encode "XML";
encode "Jason";
variant "XML":"name as uncapitalized"
variant "Jason":"name as 'dt'"
}
type record PDU {
Payload payload
} with {
encode "XML";
encode "Jason";
variant "XML":"element"; // example of an XML variant
variant "JSON":"name as 'pd'"; // example of a Jason variant
}
type PDU XmlPDU with { encode @only "XML" }
// This definition removes all Jason related attributes and keeps XML (on all
// levels), so that the XmlPDU type is equal to:
// type record XmlPDU {
// Payload payload
// } with {
// encode "XML";
// variant "element";
// variant(payload) "name as uncapitalized";
// }
type PDU JasonPDU { encode @only "JSON" }
// type record JasonPDU {
// Payload payload
// } with {
// encode "Jason";
// variant "name as 'pd'";
// variant(payload) "name as 'dt'";
// }
Could you please comment if the @only filter would work for you? It should be possible to write a universal test suite for XML and Jason with it. You will still need two different receive branches, but handling can be common for both of them as the type structures are equal.
The main advantage of the @only modifier is that it doesn't require complicated changes in the receiving, sending, encoding and decoding statements and the impact on the TCI will be minimal. It is also easily applicable to protocol stacks.

(0014005)
Gyorgy Rethy
20-07-2016 12:03 (edited on: 20-07-2016 12:09)

Depending on what the question is:
- if @only is proposed *in addition* what the CR requests (i.e. allow types with multiple encodings, where the actual codec is chosen by the TE based on a runtime configuration parameter), it is acceptable, but is useless, superfluous; it doesn't solve dynamic encoding control either;
- if all the multiple encoding problem is wanted to be solved by aliases, it is not acceptable. It is principally wrong approach. It would not need any language extension, just a clarification of overwriting rules for encode and variant attributes for aliases.
Considering the below type definitions;
The requested solution looks like:
template PDU t_PDU := { payload := { foo := 42, bar := 5.0 }
function f() runs on CT {
P.send (t_PDU);
alt {
[] P.receive (PDU:?){...} //note, that receiving a "wrongly" encoded message, i.e. of encoded by the 'another' encoding causes a runtime error here
}
...
}
----------------------------------------------------------------------
The code with the proposed alias solution would look like:
template XmlPDU t_PDUxml := { payload := { foo := 42, bar := 5.0 }
template JasonPDU t_PDUjson := t_PDUxml;
function f() runs on CT {
if (modulepar_encoding=="XML") {P.send (t_PDUxml)};
elseif (modulepar_encoding=="JSON") {P.send (t_PDUjson);
alt {
[modulepar_encoding=="XML"] P.receive (XmlPDU :?){...} //note, that if receiving receiving a PDU encoded with "Jason" it will cause a deadlock of the component here
[modulepar_encoding=="Jason"] P.receive (JasonPDU:?){...}// the same for receiving an "XML" encoded message
//Pls. note, if "XML" encoding is expected from the SUT, but it uses "Jason" instead, this shall be a fail verdict as well!
}
...
}
Drawbacks of the proposed alias solution:
- huge amount of superfluous code, to be written for nothing, with all its readability and other consequences (the TE could choose the needed encoding itself, instead of this, the user shall explicitly control it from the TTCN-3 code);
- horrible maintain-ability: if a new, 3rd (or the 2nd) encoding is introduced, the "whole" TTCN-3 code has to be updated: new type aliases, new template aliases, new if~ and alt branches everywhere... if adding a new if~ or alt branch is forgotten at a single place, or a wrong template alias is used (a typical copy-paste error), the error cannot be discovered @ compile time, only @ runtime, by a wrong test case result (if, at all);
for example, in standard test suites, where the code is typically written by the standard body, but is not executed, this would cause a nightmare.

(0014012)
Tomas Urban
20-07-2016 16:07 (edited on: 21-07-2016 08:25)

We propose the following solution to the dynamic problem:
a) Keep the @only modifier for static filtering (for the users that might find some use of it).
b) Introduce a dynamic solution using a new encode statement:
( PortReference | "all" "port" | "self") "." encode "(" (Type | all) "," Expression ")"
The encode statement would activate an encoding filter for a particular port, all port or a whole component. The filter would apply to a particular type, its part (useful for protocol stacks) or to all messages passed to the codec. It would basically do the same thing as the @only modifier, but dynamicly.
Example for our XML/Jason case:
modulepar universal charstring ENCODING; // set to "XML" by TCI-TM
self.encode(PDU, ENCODING); // Disables all encodings that are not "XML"
// and variants related to them
p.send(PDU:{...}}; // When the message is sent out, only the "XML" encode
// attribute and related "XML" variants are visible to the codec
// in order to go through all encodings:
type record of universal charstring TEnc;
var TEnc v_enc := {...} // dynamically specify acceptable encodings
var integer i := 0;
p.encode(PDU, v_enc[i]);
alt {
[] p.receive(PDU:?)
// [] ... other alternatives
[else] {
i := i + 1;
if (i < lengthof(v_enc)) {
p.encode(PDU, v_enc[i]);
repeat;
}
}
}
The advantage of this approach is that we don't have to change the syntax of the sending/receiving operations and of the enc_value and dec_value functions. It contains support of protocol stacks as an extended type reference can be used as the first parameter.
Gyorgy, could you please say if it is acceptable for you?

(0014024)
Gyorgy Rethy
21-07-2016 09:46

If I understand correctly:
- several encode attributes can be attached to a type (group module ...) and variant attributes can be scoped with the syntax proposed by the CR;
- P.encode(PDU,modPar_encoding); activates a concrete encoding to a given port
- all port.encode(PDU,modPar_encoding); activates a concrete encoding to all ports of the given component, but NOT for encvalue/decvalue-s
- self.encode(PDU,modPar_encoding); activates a concrete encoding to all ports AND encvalue/decvalue-s of the given component
- in addition, a new optional parameter will allow select/change the default encoding for individual encvalue/decvalue/encvalue_unichar/decvalue_unichar calls
This solution, as a whole, is fine with me.

(0014041)
Martin Hauch
02-08-2016 10:50

Questions:
Where are the types defined, which should be used for "JSON" and "XML" coding?
Definitions from XSD:
"Part 9: Using XML schema with TTCN-3" does not define variant "XML":"element"; (or other "XML" renamings for elements or attributes). Current "Part 11: Using JSON with TTCN-3" does not define variant "JSON":name as 'pd'"
Should the syntax for variant attributes be changed to bind the variant-atrribute information to an encoding-rule in case of more than 1 encoding-rule is assigned?
How should JSON-variants be added to these from XSD converted types?
How should xsd-defined types be encoded in JSON?
Definitions from TTCN:
I think this should work. The encode- and variant-attributes could be defined for all types (as shown in your example), respecting the possible attribute-values defined in the documents "Part 9: Using XML schema with TTCN-3" and "Part 11: Using JSON with TTCN-3".

(0014042)
Gyorgy Rethy
02-08-2016 13:03 (edited on: 02-08-2016 13:46)

Thanks Martin, good points.
I think both Part-9 and draft Part-11 shall be updated to allow generating the variant scopings automatically (for backward compatibility and readibility reasons this should be a user-choice converter configuration parameter).
As JSON doesn't have a schema, no JSON encoding instructions can be generated automatically, they always need to be added manually to TTCN-3 files. Therefore, the output of an XML converter need to be decorated with JSON variant attributes manually.

(0014043)
Gyorgy Rethy
02-08-2016 13:46

During the internal discussions we were also discussing a possible compact syntax to assign an encoding instruction to multiple encode-s, something like
variant (resourceName){"XML","JSON"}:"name as 'rn'";
at the end this feature wasn't included into the initial proposal, to keep it simple.
However, now ETSI has raised the same issue, as the main attribute to be used in the IoT conformance tests is name as, which is the same in both XML and JSON. Another ETSI's point is that e.g. if a new, 3rd encoding is added, it is easier to add the name of the new encoding to the list of encodes, than copy-paste-change all the variants for the new encoding, which would make the code long, nasty and hardly readable.
Could you please also consider a compact syntax, to add an encoding instruction to multiple encodings by a single variant?

(0014114)
Tomas Urban
16-08-2016 16:24

Proposal uploaded. The document also contains a resolution for all related CRs. Please check.

(0014134)
Jens Grabowski
17-08-2016 15:17

Please check again. Ericsson should also proofread before resolving the issue.

(0014150)
Tomas Urban
18-08-2016 09:48

I am fine with the corrections. I only made a couple of minor formating changes in the document. Kristóf, could you please look at the document too and discuss the matter with György?

(0014158)
Jacob Wieland - Spirent
18-08-2016 15:29

I have added/changed multiple places according to our discussions. Please review.

(0014161)
Gyorgy Rethy
18-08-2016 16:44

Hi, in general the proposal looks fine, but a few comments/questions:
1) Regarding the @local modifier I’m not sure if it’s really needed: while it complicates the language and the tools, I can see little practical use. Let see the example from the standard:
module M {
type record MyRec {
integer field1,
integer field1,
} with { encode @local "CodecB" }
// the record type MyRec will be encoded with CodecB, but its fields with CodecA,
// because local attribute CodecB doesn’t affect elements of the MyRec type.
} with { encode "CodecA" }
In reality, typically protocol messages and their fields have encode attributes, in which case fields are practically always some constrained types, having their own type definitions (and thus inheriting the attribute associated to the group/module). Maybe boolean fields are exceptions, but I haven’t seen any of them being encoded differently than the enclosing type; and if, it is more intuitive to add their encoding to the field directly (than to understand what @local means):
module M {
type integer (0..255) Intfield;
type record MyRec {
Intfield field1,
Intfield field2,
boolean field3
} with { encode "CodecB";//this does not apply to fields field1 and field2 acc. to the previous rule
encode (field3) “CodecA”}
} with { encode "CodecA" }
2) I think it would be more secure if variant attribute without encode reference is allowed only, if there is just a single encode applied to the scope (i.e. not to have "global variant"s). Once a new encoding is added to a module, all variants has to be re-checked anyway, during which it is easy to copy-paste the encoding scope in front of the already existing variant. But with the current "global variant" rule, variants forgotten to be cross-checked and scoped, will automatically start effecting on the new encoding as well.
3) If considering the example:
// Modifying list of allowed encodings
type Int Int2 with {
encode "CodecA"; // variants "CodecA"."Rule1" and "GlobalRule" are kept
encode "CodecC"; variant "CodecC"."Rule6"; // new encoding and related variant
// "CodecB" encoding and related variant are discarded as "CodecB" is not
// explicitly referenced
}
It is unclear why the @only modifier is needed.
In the example
// Selecting encoding with an @only modifier
type MyRecord MyRecord2 with {
encode @only (field1) "CodecB"; // field1 will be encoded with CodecB
encode @only "CodecA" // the record and field2 will be encoded with CodecA
}
Applying encode (field1) "CodecB" and encode "CodecA" would have exactly the same effect: whatever other encodings applied to MyRecord and MyRecord.field1 are simply discarded.
4) Has ETSI’s request to allow associating a variant to several encodings with a single statement been considered? Example (syntax below is appropriate, any suitable syntax will do it):
module M {
type record MyRecX {
Intfield field1,
Intfield field2,
boolean field3
} with { variant(field1) {"CodecA”,”CodecB”}.”name as ‘f1’”;
variant(field2) {"CodecA”,”CodecB”}.”name as ‘f2’”;
variant(field3) {"CodecA”,”CodecB”}.”name as ‘f3’”}
} with { encode "CodecA" }

(0014178)
KristÃ³f Szabados
19-08-2016 13:26

Added clarification, that variants without explicit encoding relation are implicitly related to all encodings in effect.

(0014183)
KristÃ³f Szabados
19-08-2016 14:36

Added example to show the difference between encodes with and without @only.

(0014230)
Jens Grabowski
14-11-2016 15:38

STF discussion: 1) @local will be kept 2) no global variants for multiple encodings 4) one variant for multiple encodings by explicit naming 3) @only will be skipped (no different scope level, same scope level) - Precedence rules for 3) will be checked for tools represented in STF.

(0014259)
Tomas Urban
16-11-2016 11:02

Specification updated according to STF discussion conclusion. Please review.

(0014274)
Jacob Wieland - Spirent
16-11-2016 15:10

added a missing closing parenthesis in BNF, otherwise ok, ready for implementation

(0014288)
KristÃ³f Szabados
17-11-2016 09:52

Corrected some typos. Added a note before example 4. Corrected example in section 27.9 (in the testcase ContainerType needed to be changed to PDU as that is the name of the type)

(0014300)
Jens Grabowski
17-11-2016 11:53

OK, ready for implementation.

(0014395)
Gyorgy Rethy
12-12-2016 18:22

Added to draft V4.8.2