ETSI's Bug Tracker - Ext Pack: Advanced Matching (ES 203 022)
View Issue Details

ID | Project | Category | View Status | Date Submitted | Last Update
0007174 | Ext Pack: Advanced Matching (ES 203 022) | New Feature | public | 11-09-2015 08:57 | 24-12-2016 12:01

Reporter | Tomas Urban |
Assigned To | Jens Grabowski |
Priority | normal | Severity | minor | Reproducibility | have not tried
Status | closed | Resolution | fixed |
Platform |  | OS |  | OS Version |
Product Version |  |
Target Version | v1.1.1 (published 2017-07) | Fixed in Version |  |
Clause Reference(s) | 15.7, B.1
Source (company - Author) | Elvior
TS version |

Summary | 0007174: New matching mechanism: matching by function

Description

Current TTCN-3 matching mechanisms are very efficient in dealing with quite many standard situations. However, there's always possibility that matching conditions are too complicated and none of the matching mechanisms can be used. The typical solution in this case is performing additional checking in the code block attached to receive statements.
This approach has unfortunately quite many drawbacks that are mostly related to the jump out of the snapshot mechanism. For that reason I propose a new matching mechanism that would allow the users to define their own custom matching procedure.
Syntax:
function match FunctionIdentifier
// NOTE: there might be more elegant syntax options. The current proposal
// advantage is that it uses existing keywords and pretty well describes
// the functionality
Semantics:
FunctionIdentifier shall be a reference to a function that accepts a single in value parameter and returns boolean. The matching mechanism matches values of the same type or compatible type as the type of the parameter.
When matching, the referenced function is called with the matched value passed to it as an actual parameter. Succesful match is achieved only if the function returns true.
Restrictions:
a. When used in receiving communication operations, rules from the section 16.1.4 apply.
b. An error shall be produced when the referenced function returns something else than a boolean value.
c. An error shall be produced when the referenced function contains a different number of parameter than 1.
d. An error shall be produced when the parameter of the referenced function is not an in value paremeter.
Required modifications in other rules:
15.6: the matching mechanism doesn't allow to reference subfields and it should be therefore listed in 15.6.2.a, 15.6.3.b, 15.6.4.a, 15.6.5.a
Example:
type record R {
charstring startDate,
charstring endDate
}
function f_extractMonth(charstring p_date) return integer {
// unspecified algorithm to extract month from a string-based date
}
function f_isMonthTheSame(in R p_rec) return boolean {
return f_extractMonth(p_rec.startDate) == f_extractMonth(p_rec.endDate);
}
template R mw_monthMatch := function match f_isMonthTheSame;

Steps To Reproduce |
Additional Information |
Tags | No tags attached.

Relationships

related to | 0007494 | closed | Jens Grabowski | TRI and TCI extensions for advanced matching
related to | 0007175 | closed | Jens Grabowski | No easy way to describe conjunction matching mechanism.
related to | 0007167 | closed | Jens Grabowski | Matching mechanism to match multiple items in arrays, records and sets of single types

Attached Files

Dynamic-Matching-Templates-Ideas-160722.pptx (31,688) 22-07-2016 13:36
http://oldforge.etsi.org/mantis/file_download.php?file_id=3434&type=bug
CR7174.docx (147,228) 16-08-2016 09:21
http://oldforge.etsi.org/mantis/file_download.php?file_id=3450&type=bug
CR7174-v2.docx (148,644) 16-08-2016 11:44
http://oldforge.etsi.org/mantis/file_download.php?file_id=3459&type=bug
CR7174-3.docx (167,385) 18-08-2016 15:23
http://oldforge.etsi.org/mantis/file_download.php?file_id=3491&type=bug
CR7174-4.docx (170,169) 19-08-2016 15:08
http://oldforge.etsi.org/mantis/file_download.php?file_id=3505&type=bug

Issue History
Date Modified | Username | Field | Change
11-09-2015 08:57 | Tomas Urban | New Issue |
11-09-2015 10:07 | Jacob Wieland - Spirent | Note Added: 0013196 |
11-09-2015 10:08 | Jacob Wieland - Spirent | Note Edited: 0013196 | bug_revision_view_page.php?bugnote_id=13196#r162
11-09-2015 10:09 | Jacob Wieland - Spirent | Note Edited: 0013196 | bug_revision_view_page.php?bugnote_id=13196#r163
11-09-2015 10:36 | Tomas Urban | Note Added: 0013197 |
21-09-2015 10:07 | Gyorgy Rethy | Relationship added | related to 0007175
21-09-2015 10:25 | Gyorgy Rethy | Target Version | => v4.8.1 (ongoing)
21-09-2015 11:11 | Gyorgy Rethy | Relationship added | related to 0007167
22-09-2015 11:35 | Gyorgy Rethy | Note Added: 0013237 |
22-09-2015 11:40 | Gyorgy Rethy | Note Edited: 0013237 | bug_revision_view_page.php?bugnote_id=13237#r171
24-09-2015 13:19 | Jacob Wieland - Spirent | Note Added: 0013289 |
25-09-2015 10:19 | Jacob Wieland - Spirent | Note Added: 0013301 |
30-09-2015 15:00 | Tomas Urban | Note Added: 0013342 |
01-10-2015 08:48 | Jacob Wieland - Spirent | Note Added: 0013343 |
01-10-2015 09:18 | Jacob Wieland - Spirent | Note Added: 0013344 |
01-10-2015 10:36 | Tomas Urban | Note Added: 0013345 |
01-10-2015 11:06 | Jacob Wieland - Spirent | Note Added: 0013346 |
01-10-2015 11:08 | Jacob Wieland - Spirent | Note Edited: 0013346 | bug_revision_view_page.php?bugnote_id=13346#r175
03-11-2015 11:55 | Gyorgy Rethy | Target Version | v4.8.1 (ongoing) => Next version (to be defined)
03-11-2015 11:56 | Gyorgy Rethy | Note Added: 0013465 |
03-11-2015 11:59 | Gyorgy Rethy | Note Edited: 0013465 | bug_revision_view_page.php?bugnote_id=13465#r200
05-11-2015 11:01 | Gyorgy Rethy | Project | Part 01: TTCN-3 Core Language => Ext Pack: Advanced Matching (ES 203 022)
05-11-2015 11:08 | Gyorgy Rethy | Product Version | v4.7.1 (published 2015-06) =>
05-11-2015 11:08 | Gyorgy Rethy | Target Version | Next version (to be defined) => v1.1.1 (published 2017-07)
18-07-2016 11:24 | Jens Grabowski | Assigned To | => Jens Grabowski
18-07-2016 11:24 | Jens Grabowski | Status | new => assigned
22-07-2016 13:36 | Jens Grabowski | File Added: Dynamic-Matching-Templates-Ideas-160722.pptx |
15-08-2016 13:10 | Jens Grabowski | Assigned To | Jens Grabowski => Jacob Wieland - Spirent
15-08-2016 16:00 | Jacob Wieland - Spirent | File Added: CR7174.docx |
15-08-2016 16:02 | Jacob Wieland - Spirent | Note Added: 0014081 |
15-08-2016 16:02 | Jacob Wieland - Spirent | Assigned To | Jacob Wieland - Spirent => Jens Grabowski
15-08-2016 16:02 | Jacob Wieland - Spirent | Status | assigned => confirmed
16-08-2016 09:21 | Jacob Wieland - Spirent | File Deleted: CR7174.docx |
16-08-2016 09:21 | Jacob Wieland - Spirent | File Added: CR7174.docx |
16-08-2016 09:21 | Jacob Wieland - Spirent | Note Added: 0014087 |
16-08-2016 11:44 | Jens Grabowski | File Added: CR7174-v2.docx |
16-08-2016 11:44 | Jens Grabowski | Note Added: 0014103 |
16-08-2016 11:45 | Jens Grabowski | Assigned To | Jens Grabowski => Tomas Urban
16-08-2016 11:45 | Jens Grabowski | Status | confirmed => assigned
18-08-2016 15:23 | Tomas Urban | File Added: CR7174-3.docx |
18-08-2016 15:24 | Tomas Urban | Relationship added | related to 0007494
18-08-2016 15:26 | Tomas Urban | Note Added: 0014157 |
18-08-2016 15:26 | Tomas Urban | Assigned To | Tomas Urban => Axel Rennoch
18-08-2016 15:26 | Tomas Urban | Status | assigned => confirmed
19-08-2016 15:08 | Axel Rennoch | File Added: CR7174-4.docx |
19-08-2016 15:11 | Axel Rennoch | Note Added: 0014188 |
19-08-2016 15:11 | Axel Rennoch | Assigned To | Axel Rennoch => Jens Grabowski
19-08-2016 15:11 | Axel Rennoch | Status | confirmed => assigned
14-11-2016 10:38 | Jens Grabowski | Status | assigned => confirmed
17-11-2016 12:37 | Jens Grabowski | Status | confirmed => resolved
17-11-2016 12:37 | Jens Grabowski | Resolution | open => fixed
24-12-2016 12:01 | Jens Grabowski | Status | resolved => closed

Notes

(0013196)
Jacob Wieland - Spirent
11-09-2015 10:07 (edited on: 11-09-2015 10:09)

I second the proposal. Since other templates can also use functions for initialization, there is no new problem concerning the Halting Problem with these new functions.
Also, since no object can be both of type T and of type T => boolean, it would probably be sufficient to just allow any deterministic function-reference of type function(T x) return boolean as matching mechanism for a template of type T.
I.e.
I would also simply allow:
template R x := f
if f is of type function(R y) return boolean.
Since this at the moment is a semantic error, there is no backward compatibility issue with that.
If used in modifies, it should be replaced fully (if it is part of the base template) or replace fully if it is in the modifying template.
I'm a bit against using 'function' as a keyword for that matching mechanism as this will make error-recovery for parsers harder which rely on declaration-keywords. In that case, I would prefer a modifier @matching or simple @is, i.e.
template R x := @matching f
or
template R x := @is f
This @is is motivated by the fact that boolean functions are normally considered semantically equivalent to predicates and for a predicate P 'P(v)' is the same as 'v is P'.

(0013197)
Tomas Urban
11-09-2015 10:36

I have no problem with the proposed modifiers. However, I would prefer not to use function identifiers directly because of a negative impact on readability of the code. The syntax looks the same as a variable/template reference and if naming convensions are not strictly followed, the real meaning of the matching mechanism won't be immediately obvious. It is probably also more complicated to implement it than the solution with a modifier.

(0013237)
Gyorgy Rethy
22-09-2015 11:35 (edited on: 22-09-2015 11:40)

STF discussion:
- the feature is useful
- the body shall be deterministic and without side effects to snapshots
- its declaration is open yet: it can be a new language "class" user defined matching mechanism or a function with a specific prototype. This to be decided.
- Parameterization to be allowed, but the concrete syntax still need to be agreed.
- Alternative idea defining dynamic templates with a function-like body (see restrictions above) and returning boolean (without a return clause in the header).
- A new package for advanced matching is to be considered.

(0013289)
Jacob Wieland - Spirent
24-09-2015 13:19

Syntactically, I would propose the following:
@dynamic template ReferencedType Identifier[FormalParList] StatementBlock
In the body part, I would refer to the argument per default as _ (single underscore).
In easy expressions, this looks like this:
return _ > 5;
If a name should be given to the parameter, it can be done by simple variable/constant declaration:
var T x := _;
This way, there is no need for an additional syntactical way of declaring the template-instance-parameter name and it is still possible to give it a name, if so desired.
The underscore does not clash with the normal identifiers as they need to start with a letter.
Alternatively, we could allow the following lambda-closure-like syntax:
@dynamic template ReferencedType Identifier[FormalParList] ":=" "(" Identifier ")" "=>" StatementBlock
where the Identifier before the "=>" is the name of the value to be matched
This would actually also work great with inline-templates:
T:(x) => { ... }
because
T:{ ... } needs possibly indefinite parser lookahead (you need to find out if the block begins with a statement of with a field assignment or with an element) to distinguish from non-dynamic templates.
If not going with the lambda-notation, inline dynamic templates thus need also be prefixed with the @dynamic modifier:
p.receive(@dynamic T:{...})
Additionally, as a shorthand notation for statement blocks that only contain a return statement, we could use the declarative notation:
@dynamic template ReferencedType Identifier[FormalParList] := [(Identifier) =>] BooleanExpression
as in (if isOdd is a @deterministic boolean function with an integer formal parameter):
@dynamic template integer odd := isOdd(_);
or inline
@dynamic integer:isOdd(_)
or
@dynamic template integer odd := (x) => isOdd(x);
or inline
@dynamic integer:(x) => isOdd(x)
For the very common pattern: (x) => f(x), we could also allow just f, which would shorten the above to:
@dynamic template integer odd := isOdd;
or inline
@dynamic integer:isOdd
Finally, for places where the type of the template is already known, i.e. for template fields or template parameters, we could allow both
(Identifier) => BooleanExpression,
(Identifier) => StatementBlock and
BooleanFunctionReference
(maybe the last thing should be (optionally or mandatory) prefixed with @dynamic)

(0013301)
Jacob Wieland - Spirent
25-09-2015 10:19

Thinking about this some more, I think it is probably best to not allow the underscore-notation. It is too abstract and also not compositional (i.e. if in the body of the function, another inline template is used, the inner _ would shadow the outer _ and that is normally not allowed in TTCN-3.
The notation
(x) => BooleanExpression
also has the drawback that you potentially need indefinite look-ahead in the parser (if using LL-approach) to determine that it is not the (x) => StatementBlock notation (in the special, though probably pathological case that the boolean expression starts with a compound value, like in { ... } == x. Limiting the syntax of the BooleanExpression to a subset that does not clash in prefix with the StatementBlock is not good language design, either.
So, the remaining syntaxes would be:
(x) => StatementBlock
FunctionReference (for the special case (x) => { return f(x) }), possibly with a modifier in front like @dynamic or @match or @is.

(0013342)
Tomas Urban
30-09-2015 15:00

I think it is better to keep things as simple as possible. Parameter passing can be resolved by using lambda closure in a similar way as in the case of @lazy and @fuzzy variables.
From the syntactical point of view, I would skip the (x) part completely and use a modifier as a prefix followed always by a statement block. This kind of syntax should be easy to parse and won't cause any grammar conflicts. The syntax would be then:
modifier StatementBlock
For referencing the matched value, I would use a universal identifier such as "value". The value keyword is already present in TTCN-3 and it is commonly used as a context-specific variable identifier (e.g. in C#).
Examples:
// @dynamic modifier is used, but any of Jacob's proposed alternatives is fine
// simple dynamic template
template integer mw_greaterThan2 := @dynamic { return value > 2; }
// parameterized dynamic template
template integer mw_greaterThanParam(integer p_par) := @dynamic { return value > p_par; }
// calling a function
template integer mw_matchByFunction := @dynamic { return f_myFunc(value); }
// when used inside a block
function f_createTemplate() return template integer
{
var integer v_var1, v_var2;
// ... compute v_var1 and v_var2
// lambda closure required to keep the values of v_var1 and v_var2 after exiting the function
return @dynamic { return value * v_var1 > v_var2; };
}

(0013343)
Jacob Wieland - Spirent
01-10-2015 08:48

I am fine with this proposal (although it complicates the implementation of the compilation frontend quite a bit).
It is also compositional (if a dynamic template uses another dynamic template inside its body, it must simply declare a variable/constant referring to the value so that the inner template can access it (as the inner 'value' shadows that outer 'value').
It cannot clash with the normal use of the value-keyword as inside such bodies, no blocking or otherwise state-changing statements are allowed.
It would still like to have a shorthand version for the @dynamic { return f(value) } template. Either
@dynamic <FunctionReference> (which is clearly distinguished syntactically as <FunctionReference> must start with an identifier and the other starts with opening bracket '{')
or using a different modifier like @is or @match (which is simply more readable but introduces another modifier)
For the last example you gave (which produced the closure), I would add a restriction that a dynamic template shall not reference their context's inout or out parameters or their sub-structures as the target of an assignment (or not at all, to avoid confusion). That way, there is no need to explain what the semantics of such an assignment would be.

(0013344)
Jacob Wieland - Spirent
01-10-2015 09:18

Thinking some more on this, I would actually disallow any reference to any variable or parameter (except read-access to the template-parameters if we are contextually inside a parameterized template because they cannot change) inside a @dynamic body.
consider the following example:
function g(template integer p) {
}
function f(integer p) return template integer {
var template integer t := @dynamic { value > p };
while (...) {
ptc.start(g(t));
p := ...;
ptc.done;
}
}
Here, it would be unclear that the reference p inside the lambda-template when used by g would be and even worse, there is suddenly some data-sharing between the component running f and the one running g. Similar things are true if write-access to outside-variables by the dynamic template were allowed.
If we restrict that the only things from outside the dynamic block that can be referenced are const and non-var template and formal parameters of the containing template, this cannot happen.
Of course, in your lambda-example, this would lead to the inconvenience that v_var1 and v_var2 need to be copied to const c_1 and c_2 and referenced as these from the dynamic block.
If this restriction is not added, then a different semantics needs to be given to references of outside-variables inside a dynamic block, i.e. they are implicit formal in parameters whose initial value is the one their outside-namesake has when the @dynamic block is instantiated. While this approach is a little bit more convenient for the user, it has a level of implicitly that might be confusing (it could also be added later if the restrictive approach is deemed to cumbersome).

(0013345)
Tomas Urban
01-10-2015 10:36

Well written, Jacob! I didn't notice the component issue at all.
1. I like the explanation you provided for the direct function call notation, i.e. that @dynamic FunctionIdentifier is a shorthand for @dynamic { return FunctionIdentifier(value); }
This so far the best explanation and completely in-line with similar shortcuts that are already present in the TTCN-3 specification.
2. For the closure issues I propose two semantic restrictions:
a. In the statement block of the @dynamic matching mechanism, only variables defined in this block can appear on left hand side of an assignment and as an actual parameter of formal out and inout parameters. Using any other variable in these place shall produce an error.
-> This restriction should prevent the assignment issue already in the compilation stage. We don't have to use copying which might be actually undesirable, because in some cases the user might want to compare the matched value against the actual variable value.
b. Templates containing a @dynamic matching mechanism created in functions, test cases or altsteps running on a main or parallel test component cannot be used as a parameter of a function referenced in a component.start operation. Using such a template shall produce an error.
-> The rule should prevent possibility of dynamic access to other component variables.
I agree that compilation might be complicated, because the mechanism itself is not strongly typed, but the type should be always visible from the the context:
template MyType mw_t1 := @dynamic{...} // type from the template definition
template MyRec mw_t2 := {
field1 := ?
field2 := @dynamic {...} // typed according to MyRec.field2 definition
}
p.receive(MyType:@dynamic {...}); // using inline type
f_func(@dynamic {...}); // using function parameter type

(0013346)
Jacob Wieland - Spirent
01-10-2015 11:06 (edited on: 01-10-2015 11:08)

I don't think that your restriction b. is a good idea as in general it can only be checked at runtime (can only be statically checked if the @dynamic template is actually part of the start-statement). Runtime-checks always introduce inefficiency during execution and also have the drawback that often standardization-bodies do not execute their tests and thus errors come to light only much later and it takes a lot of time until they are officially fixed.
The restrictions I proposed are easily checked statically (at compile-time), even though they introduce some inconvenience to the usage (though not much, you could easily declare all the variables-to-be-used as const from the beginning and then simply initialize with the computed value directly).
Otherwise, I would rather opt for the copy-semantics, i.e.
if the variables/parameters (not parameters of the enclosing parameterized template) V1 ... Vn which are not declared but used in the @dynamic B body B have the types T1 ... Tn respectively, the semantics of @dynamic B is the same as t(V1, ..., Vn) where t is an implicit generic template t(T1 V1, ..., Tn Vn) := @dynamic B (with this semantic trick a copy of each variable at the time of template instantiation is used in B).
The frontend-issue I mentioned is more the problem that suddenly the value-keyword can be used as an identifier in a certain context and it is context-dependent whether it shall be interpreted as a keyword or as an identifier. It's not an unsolvable problem, but it is tricky.
The type of the 'value' keyword, of course, is always the type of the whole dynamic template, so that type always must be known, either by the inline-template-type-prefix or because the template is used in a context whose type is known.

(0013465)
Gyorgy Rethy
03-11-2015 11:56 (edited on: 03-11-2015 11:59)

To be part of the new extension on advanced matching mechanisms.

(0014081)
Jacob Wieland - Spirent
15-08-2016 16:02

Added main section and BNF changes. Please review. Still missing: TCI changes.

(0014087)
Jacob Wieland - Spirent
16-08-2016 09:21

refined the restriction on in parameters to be more airtight

(0014103)
Jens Grabowski
16-08-2016 11:44

Tomas: Can you proofread the text and add a section on TCI changes if needed.

(0014157)
Tomas Urban
18-08-2016 15:26

Minor changes made (shorted name of the matching mechanism: dynamic matching function -> dynamic matching, qualified function name), assigned to Axel for proofreading. TRI and TCI will be handled separately in its own CR 0007494.

(0014188)
Axel Rennoch
19-08-2016 15:11

modified (naming conventions, font corrections, additional comment in example) proposal can be used for the final draft