ETSI's Bug Tracker - Part 01: TTCN-3 Core Language
View Issue Details

ID | Project | Category | View Status | Date Submitted | Last Update
0006690 | Part 01: TTCN-3 Core Language | New Feature | public | 17-02-2014 16:01 | 28-07-2014 16:43

Reporter | Wolfgang Seka |
Assigned To | Gyorgy Rethy |
Priority | urgent | Severity | minor | Reproducibility | have not tried
Status | closed | Resolution | fixed |
Platform |  | OS |  | OS Version |
Product Version | v4.6.1 (published 2014-06) |
Target Version | v4.6.2 (interim 2014) | Fixed in Version | v4.6.2 (interim 2014) |
Clause Reference(s) | B.1.5, C.4.1
Source (company - Author) | MCC160 - Wolfgang

Summary | 0006690: Case insensitive pattern matching

Description

there is an issue with text based protocols:
E.g. SIP is widely case insensitive. But when looking at existing implementation this is not handled in TTCN.
Of course it could be handled by codec (e.g. by converting all case insensitive strings to lower case) but that seems to be not the best solution as there are still case sensitve strings and it may be hard to find generic criteria for the codec implementation. Furthermore it seems to be not a good design decision to mandate manipulation of received strings by codec.
=> it seems to be better to add means for case insensitive pattern matching to the TTCN language:
This may be e.g. by enhancement of the "pattern" keyword:
template (present) charstring v_MyPattern := pattern caseinsensitive "*AB*"; /* shall match any string containing "AB", "Ab", "aB", "ab" */
(Of course this is just an example for how it could be expressed).
In addition the regexp function may be enhanced by a new parameter as well to allow case insensitive regular expressions.
My questions to you:
1. Can you acknowledge the issue as such ??
2. Is it feasable to add support of case insensitive pattern matching to the TTCN-3 language ??
3. Assuming the answer to 1./2. is "yes" what can be the time schedule ??

Steps To Reproduce |
Additional Information |
Tags | No tags attached.
Relationships |

Attached Files

CaseInsensitivePattern.ttcn (1,054) 10-04-2014 08:58
http://oldforge.etsi.org/mantis/file_download.php?file_id=2988&type=bug
CR6690_caseInsensitivePattern_resolution_v1.doc (762,880) 11-04-2014 11:21
http://oldforge.etsi.org/mantis/file_download.php?file_id=3014&type=bug
CR6690_caseInsensitivePattern_resolution_v2.doc (763,904) 11-04-2014 12:35
http://oldforge.etsi.org/mantis/file_download.php?file_id=3015&type=bug
CR6690_caseInsensitivePattern_resolution_v3.doc (764,416) 16-06-2014 09:13
http://oldforge.etsi.org/mantis/file_download.php?file_id=3028&type=bug
CR6690_caseInsensitivePattern_resolution_v4.doc (764,416) 17-06-2014 15:42
http://oldforge.etsi.org/mantis/file_download.php?file_id=3038&type=bug

Issue History
Date Modified | Username | Field | Change
17-02-2014 16:01 | Wolfgang Seka | New Issue |
20-02-2014 09:25 | Jacob Wieland - Spirent | Note Added: 0011925 |
07-04-2014 13:26 | Gyorgy Rethy | Project | TTCN-3 Change Requests => Part 01: TTCN-3 Core Language
07-04-2014 14:05 | Jens Grabowski | Note Added: 0011939 |
07-04-2014 14:06 | Jens Grabowski | Assigned To | => Tomas Urban
07-04-2014 14:06 | Jens Grabowski | Status | new => assigned
08-04-2014 13:14 | Tomas Urban | Note Added: 0011958 |
08-04-2014 13:22 | Tomas Urban | Assigned To | Tomas Urban => Gyorgy Rethy
08-04-2014 17:01 | Gyorgy Rethy | Product Version | => v4.6.1 (published 2014-06)
08-04-2014 17:01 | Gyorgy Rethy | Target Version | => v4.7.1 (published 2015-06)
09-04-2014 12:37 | Gyorgy Rethy | Priority | high => urgent
09-04-2014 13:19 | Gyorgy Rethy | Note Added: 0011972 |
09-04-2014 13:33 | Gyorgy Rethy | Note Edited: 0011972 | bug_revision_view_page.php?bugnote_id=11972#r6
10-04-2014 08:01 | Gyorgy Rethy | Note Added: 0011980 |
10-04-2014 08:42 | Gyorgy Rethy | Note Added: 0011981 |
10-04-2014 08:50 | Gyorgy Rethy | Note Edited: 0011981 | bug_revision_view_page.php?bugnote_id=11981#r8
10-04-2014 08:55 | Gyorgy Rethy | Note Added: 0011982 |
10-04-2014 08:56 | Gyorgy Rethy | File Added: CaseInsensitivePattern.ttcn |
10-04-2014 08:57 | Gyorgy Rethy | File Deleted: CaseInsensitivePattern.ttcn |
10-04-2014 08:58 | Gyorgy Rethy | File Added: CaseInsensitivePattern.ttcn |
10-04-2014 10:28 | Gyorgy Rethy | Note Edited: 0011958 | bug_revision_view_page.php?bugnote_id=11958#r10
10-04-2014 10:35 | Gyorgy Rethy | Note Added: 0011988 |
10-04-2014 11:13 | Gyorgy Rethy | Note Added: 0011989 |
10-04-2014 11:18 | Gyorgy Rethy | Note Added: 0011990 |
11-04-2014 11:20 | Gyorgy Rethy | Note Edited: 0011990 | bug_revision_view_page.php?bugnote_id=11990#r12
11-04-2014 11:20 | Gyorgy Rethy | Note Added: 0012022 |
11-04-2014 11:21 | Gyorgy Rethy | File Added: CR6690_caseInsensitivePattern_resolution_v1.doc |
11-04-2014 11:22 | Gyorgy Rethy | Note Added: 0012023 |
11-04-2014 11:22 | Gyorgy Rethy | Assigned To | Gyorgy Rethy => Tomas Urban
11-04-2014 11:22 | Gyorgy Rethy | Status | assigned => confirmed
11-04-2014 12:35 | Tomas Urban | File Added: CR6690_caseInsensitivePattern_resolution_v2.doc |
11-04-2014 12:37 | Tomas Urban | Note Added: 0012024 |
11-04-2014 12:37 | Tomas Urban | Status | confirmed => resolved
11-04-2014 12:37 | Tomas Urban | Resolution | open => fixed
11-04-2014 12:37 | Tomas Urban | Assigned To | Tomas Urban => Gyorgy Rethy
15-04-2014 14:07 | Jacob Wieland - Spirent | Note Added: 0012040 |
15-04-2014 14:07 | Jacob Wieland - Spirent | Status | resolved => feedback
15-04-2014 14:07 | Jacob Wieland - Spirent | Resolution | fixed => reopened
30-04-2014 08:39 | Jacob Wieland - Spirent | Assigned To | Gyorgy Rethy => Tomas Urban
30-04-2014 08:39 | Jacob Wieland - Spirent | Status | feedback => confirmed
16-06-2014 09:13 | Tomas Urban | File Added: CR6690_caseInsensitivePattern_resolution_v3.doc |
16-06-2014 09:13 | Tomas Urban | Assigned To | Tomas Urban => Gyorgy Rethy
16-06-2014 09:13 | Tomas Urban | Status | confirmed => assigned
16-06-2014 09:14 | Tomas Urban | Note Added: 0012074 |
17-06-2014 09:36 | Gyorgy Rethy | Status | assigned => confirmed
17-06-2014 15:42 | Gyorgy Rethy | File Added: CR6690_caseInsensitivePattern_resolution_v4.doc |
17-06-2014 15:55 | Gyorgy Rethy | Target Version | v4.7.1 (published 2015-06) => v4.6.2 (interim 2014)
17-06-2014 15:59 | Gyorgy Rethy | Note Added: 0012109 |
17-06-2014 15:59 | Gyorgy Rethy | Status | confirmed => resolved
17-06-2014 15:59 | Gyorgy Rethy | Fixed in Version | => v4.6.2 (interim 2014)
17-06-2014 15:59 | Gyorgy Rethy | Resolution | reopened => fixed
20-06-2014 11:55 | Gyorgy Rethy | Fixed in Version | v4.6.2 (interim 2014) =>
28-07-2014 16:43 | Gyorgy Rethy | Note Added: 0012207 |
28-07-2014 16:43 | Gyorgy Rethy | Status | resolved => closed
28-07-2014 16:43 | Gyorgy Rethy | Fixed in Version | => v4.6.2 (interim 2014)

Notes

(0011925)
Jacob Wieland - Spirent
20-02-2014 09:25

Testing Tech agrees that such a feature is necessary. We have similar problems in other text-based protocols.

(0011939)
Jens Grabowski
07-04-2014 14:05

STF agreed, syntactical alternatives will be investigated by Tomas.

(0011958)
Tomas Urban
08-04-2014 13:14 (edited on: 10-04-2014 10:28)

Possible solutions:
1. Using an additional modifier after "pattern". The solution is similar to what was proposed by the reported, but a modifier is used instead of a keyword in order to avoid problems with name clashes in the existing test suites. The proposed keyword (caseinsensitive) seems to be too long, maybe something shorter would be more feasible, e.g. insensitive or nocase:
pattern @caseinsensitive "*AB*"
pattern @insensitive "*AB*"
pattern @nocase "*AB*"
//need to be decided if this reference inherits the case-insensitive-ness or not? If no inheritance, the same template can be matched both in a case-sensitive and case-insensitive ways:
pattern "{t_pattern}";
pattern @nocase "{t_pattern}";
Advantages:
- short syntax
- it is obvious from the code what should be achieved
Disadvantages:
- adds a new modifier
- static construction, not possible to insert the modifier dynamically
2. Using a dedicated escape sequence in the beginning of the pattern string, e.g. "\I":
pattern "\I*AB*"
Advantages:
- the shortest alternative to write
- no changes in the general TTCN-3 syntax
- the escape sequence can be added dynamically to the pattern
Disadvantages:
- meaning is not obvious without detailed knowledge of TTCN-3 regular expression syntax
- possibility of backwards compatibility issues (although \I doesn't represent any metacharacter, it might still be present in existing patterns being interpreted as "I")
3. Using additional parameter of a chosen type (e.g. of a bitstring type restricted to certain length) following the pattern string:
pattern "*AB*" variant CASE_INSENSITIVE; // where CASE_INSENSITIVE is defined as const bitstring CASE_INSENSITIVE := '10000000'B;
Advantages:
- the content of the parameter part can be dynamic
- allows future extensions and combinations of parameters using bitwise logic (similar to regular expressions in java, C#): pattern "*AB*" variant CASE_INSENSITIVE or4b FUTURE_PARAMETER
- with parameters defined as useful values (similar to port status strings defined in E.2.2.4), the meaning is obvious to unskilled reader
- no new keyword/modifier
Disadvantages:
- long syntax
- adding new useful values poses a risk of name clashes

(0011972)
Gyorgy Rethy
09-04-2014 13:19 (edited on: 09-04-2014 13:33)

Thanks for the good analysis. I agree with it, just some additions:
Option 1, Disadvantages
in all cases it is possible to reference a case insensitive pattern from a case sensitive one, and vice versa; in option 1 its semantic is a matter of decision, but because of e.g. email addresses, URN-s embedded in a case sensitive environment, probably we will need to preserve the case sensitive-ness of the referenced patterns. (Btw. it may become a problem, how to identify case-insensitive fragments of patterns in logs)
Do we need a more dynamic construction mechanism than this?
Please note, that in POSIX(like) regex implementations case sensitive-ness causes additional complications, therefore dynamic construction should be kept at the minimum necessary level.
Option 3, +Disadvantage
Would overload the meaning of "variant" that is a sub-attribute of the encode attribute currently.
Or, using the current syntax, it would be
pattern "*AB*" with { variant "CASE_INSENSITIVE" }
that looks less nice and would mix the rather special variant overwriting rules into the picture.

(0011980)
Gyorgy Rethy
10-04-2014 08:01

Mail from Wolfgang (2014-04-09):
Hi György,
I don't have a strong preference from stf160 point of view:
- in general I'll try to encapsulate issues like case-sensitivity
- I would not expect any name clashes with our test suite (and if so we can solve them - we don't have frozen code)
- I general I don't have problems with long expressions
but what is important for me is that we can e.g. define a string as base to be used in an expression for case sensitive pattern as well as for case insensitive.
A typical use case for stf160 would be the GenericParam of ETSI's "LibSip" which represents the generic-param definition of RFC 3261:
generic-param = token [ EQUAL gen-value ]
gen-value = token / host / quoted-string
and in TTCN:
type record GenericParam
{
charstring id,
charstring paramValue optional
}
For this in TTCN we need some generic receive template which currently is:
template (present) GenericParam cr_GenericParam(template (present) charstring p_Id,
template charstring p_ParamValue := omit) :=
{
id := p_Id, // needs to be declared as case-insensitive
paramValue := p_ParamValue // needs to be declared as case-insensitive or quoted string (case-sensitive)
};
When looking at the above example it seems to be useful that a given pattern can be further declared as case-insensitive i.e. that it is not necessary to declare a pattern as case-insensitive from the beginning.
So for the above example we would need to "convert" p_Id to be case-insensitive and to assign to paramValue something like (<case insensitive variant of p_ParamValue>, """" & p_ParamValue & """") => another aspect could be whether we will have an "inline" solution or whether we need to go for an explicit definition in which case we may need to define wrapper functions in TTCN (as e.g. we cannot define variables in templates)
Please note that regarding definition of GenericParam we have on-going discussions to change paramValue to a union of token and quotedString to avoid dealing with double quotes in TTCN (but that's a different story)
Best regards
Wolfgang

(0011981)
Gyorgy Rethy
10-04-2014 08:42 (edited on: 10-04-2014 08:50)

Gyorgy's answer (2014-04-10):
Hi Wolfgang,
Thanks for the explanation. It seems that you may possibly need something to switch the case-insensitive-ness. One problem with this that, as we discussed up to know, embedding case insensitive fragments into case sensitive patterns (e.g. email-addresses, URN-s into a case sensitive string environment) may be needed. But control of case-sensitiveness could be achieved when referencing values from a pattern.
If taking your example, it could be changed to:
template (present) GenericParam cr_GenericParam( charstring p_Id,
template charstring p_ParamValue := omit) :=
{
id := pattern @nocase "{p_Id}", // "converted to needs to be declared as case-insensitive
paramValue := f_processParamValue(p_ParamValue) // needs to be declared as case-insensitive or quoted string (case-sensitive)
// choice between case sensitive and insensitive matching would need an additional parameter
};
function f_processParamValue(template charstring p_ParamValue) return template charstring {
if (ispresent(p_ParamValue)) {
var charstring t_tmp := valueof(p_ParamValue);
return pattern @nocase "{t_tmp}"
}
else {return omit}
}
Would something like this be OK with you?

(0011982)
Gyorgy Rethy
10-04-2014 08:55

Wolfgang's example - extended w. switching between case sensitive and insensitive matching is attached in CaseInsensitivePattern.ttcn

(0011988)
Gyorgy Rethy
10-04-2014 10:35

STF discussion: at this stage go for the pattern @nocase solution without inhering case-sensitive-ness property of referenced patterns.

(0011989)
Gyorgy Rethy
10-04-2014 11:13

Mail from Wolfgang (2014-04-10): Hi György, yes - would be ok for me. Best regards Wolfgang

(0011990)
Gyorgy Rethy
10-04-2014 11:18 (edited on: 11-04-2014 11:20)

STF discussion: The @nocase has to be added to regexp as well, with the same semantics!

(0012022)
Gyorgy Rethy
11-04-2014 11:20

Fists draft resolution text is in CR6690_caseInsensitivePattern_resolution_v1.doc.

(0012023)
Gyorgy Rethy
11-04-2014 11:22

Pls. review proposed resolution.

(0012024)
Tomas Urban
11-04-2014 12:37

The proposed resolution checked. I have found no major issues, only two small text corrections were made (updated draft uploaded). The resolution is ready to be included in the next version of the language standard.

(0012040)
Jacob Wieland - Spirent
15-04-2014 14:07

there are two typos: derefencing (should be dereferencing) and 'the whole resulted pattern' (must be 'the whole resulting pattern')

(0012074)
Tomas Urban
16-06-2014 09:14

Updated according to Jacob's comments.

(0012109)
Gyorgy Rethy
17-06-2014 15:59

Final check OK

(0012207)
Gyorgy Rethy
28-07-2014 16:43

Added to master copy of V4.6.2 (interim 2014)