Commit bffcf4a8 authored by Martti Käärik's avatar Martti Käärik
Browse files

Constraints test suite.

parent 19745cbf
Loading
Loading
Loading
Loading
+156 −0
Changes for plugins/org.etsi.mts.tdl.tx.tests/src/constraints/AUTHORING.md: 156 added lines, 0 removed lines.
Original line number Diff line number Diff line
# Writing constraint fixtures

A fixture is a small TDLtx model that violates exactly one constraint of the standard, so that the
validator (interpreted OCL today, generated Java later) must report it. The suite
(`ConstraintSuiteTest`) parses the fixture, links it against `TDL.tdltx`, validates it, and compares
the reported constraints with the fixture's header.

## Header

    //! expect NamedElement::DistinguishableName x2   exactly two reports
    //! expect Package::CyclicImports                 at least one report
    //! expect none                                   conformant model, nothing may fire
    //! allow  Some::OtherConstraint                  may fire, not counted either way

Ids are `Context::Name` exactly as written in `org.etsi.mts.tdl.model/model/*.ocl`.
A file without a header is a helper (loaded, not validated).

## Rules

1. **One package per file.** The grammar allows a single `Package` per `.tdltx` file. A fixture that
   needs several packages is a **folder**: every `.tdltx` in the folder is loaded together, and each
   file with a header is validated on its own.
2. **Violate one constraint.** Everything else in the file must be well formed: no syntax errors, no
   unresolved references, no other constraint violations. An unexpected report fails the test; use
   `allow` only when a second violation is unavoidable, and say why in a comment.
3. **Minimal.** Only the elements the constraint needs. Fewer elements means fewer accidental
   violations.
4. **Name it after the constraint.** `<Context>_<Name>.tdltx` for the violating case,
   `<Context>_<Name>.<variant>.tdltx` (or a folder of that name) for variants. Keep them in the
   subfolder of their document: `core/`, `structured/`, `configurations/`.
5. **Cover the branches.** Write one violating fixture per constraint first. Where the OCL has
   branches (`if`, `implies`, `oclIsKindOf`, a `forAll` over a collection that can be empty), add a
   conformant `expect none` variant that takes the other branch, so that a Java implementation which
   collapses the branch is caught.
6. **Comment the intent.** One line saying what is wrong and why the constraint must fire.
7. **Do not let a neighbour block the constraint under test.** All constraints of an element are
   evaluated together, and one that cannot be evaluated stops the others on that element. So keep the
   rest of the element as undemanding as possible: prefer `?` to a literal when the value's type does
   not matter to what you are testing, since a literal has to be resolved and that resolution can
   fail. A fixture that needs an `allow` for a neighbour is often a fixture that can be simplified
   until the neighbour stays quiet.

## If a constraint turns out to be wrong

Do not adjust the fixture to the implementation. If the expression itself is at fault, correct it in
`org.etsi.mts.tdl.model/model/*.ocl` (or the operation body in `tdl.ecore`) **and add the correction
to `org.etsi.mts.tdl.model/model/STANDARD-CHANGES.md`**: those files are generated from the
specification document, so a fix that is not carried into the document is reverted by the next export.

## Syntax

The authority is the grammar, `plugins/org.etsi.mts.tdl.tx/src/org/etsi/mts/tdl/TDLtx.xtext`;
worked examples are in `plugins/org.etsi.mts.tdl.tx.tests/src/models/*.tdltx` and
`src/fixtures/f1/`. The predefined library is `plugins/org.etsi.mts.tdl.library/src/org/etsi/mts/tdl/library/TDL.tdltx`
(`Integer`, `String`, `Boolean`, `Verdict`, `pass`/`fail`/`inconclusive`, `Time second`, constraint
types `length`, `range`, `union`, …, annotations `Master`, `MappingName`, …).

```tdltx
Package Example {
    Import all from TDL                 // or: Import Integer, String from TDL

    Type Simple                          // SimpleDataType
    Type Bounded { range: low, high }    // with a constraint (quantifiers are DataUses)
    Type Derived extends Simple

    Structure Struct ( Simple a, optional Simple b )
    Collection Coll of Simple
    Enumerated Colour { red, green, blue }

    Simple inst                          // SimpleDataInstance
    Simple valued = 42                   // with a value
    Struct sInst ( a = inst )            // StructuredDataInstance
    Coll cInst [ inst, inst ]            // CollectionDataInstance

    Signature Sig ( in Simple p1, out Simple p2 )
    Function f ( Simple p ) returns Simple: "description"
    Action act ( Simple p ) : "do something"
    Annotation Anno
    Objective Obj { Description: "..." }
    Constraint myConstraintType          // ConstraintType
    Time secondLocal

    Use "schema.json" as Res             // DataResourceMapping
    Map Struct to "path" in Res as M { a -> "a_uri" }   // DataElementMapping

    Message Gate mg accepts Simple, Struct
    Procedure Gate pg accepts Sig

    Component Node {
        gate mg g
        variable Simple v
        timer t
    }

    Configuration cfg {
        Node tester as Tester,
        Node sut as SUT,
        connect tester::g to sut::g
    }

    Test Description td uses cfg {
        tester::g sends inst to sut::g
        sut::g sends inst to tester::g
        tester::g receives inst from sut::g
        v = inst                                  // Assignment (variable of a component)
        tester::v = inst                          // qualified
        set verdict to pass
        assert inst == inst on tester otherwise fail
        perform act ( p = inst ) on tester
        perform action : "inline" on tester
        execute other ( p = inst )                // TestDescriptionReference
        start tester::t for one
        stop tester::t
        timeout on tester::t
        wait for one on tester
        quiet for one on tester
        break
        terminate
        if [ inst == inst on tester ] { ... } else { ... }
        alternatively { ... } or { ... }
        run { ... } in parallel to { ... }
        repeat 3 times { ... }
        while { ... }
        optionally { ... }
        pc: tester::g calls Sig ( p1 = inst ) on sut::g
        pc: sut::g responds with Sig ( p2 = inst ) to tester::g
        with { default { ... } }                  // exceptional behaviour of a combined behaviour
        with { every one { ... } }                // periodic behaviour
    }
}
```

Data uses inside behaviour: `inst`, `42`, `"text"`, `"text" {Type}` (typed literal), `?` (any),
`*` (any or omit), `omit`, `inst.member`, `coll.get(0)`, `inst ( member = value )` (override),
`instance returned from f ( p = inst )`, `inst as Type` (cast), `a + b`, `not flag`, `size(coll)`,
`@label`, `@label.last`. Time labels and constraints hang off an interaction with
`with { t = now }` and `with { { (@t2 - @t1) <= 5 {second} } }`.

## Pitfalls

- `last`, `first`, `previous`, `now`, `gate`, `on`, `to`, `from`, `with` are keywords; do not use
  them as identifiers.
- A single-letter name `a` collides with the `a`/`an` instance prefix of the grammar.
- Every `NamedElement` needs a name, and qualified names must be unique inside the model, so reusing
  a name in one package raises `NamedElement::DistinguishableName` as a side effect.
- `Import all from TDL` is needed whenever a predefined type, verdict or constraint type is used.
- Enumeration values may omit the type (`Enumerated Colour { red, green }`); an explicit type inside
  an enumeration is also allowed (`Colour red`).

## Running

From `org.etsi.mts.tdl.parent`, whole reactor:

    mvn verify -Dtest=ConstraintSuiteTest -DfailIfNoTests=false

The coverage report is written to `plugins/org.etsi.mts.tdl.tx.tests/target/constraints-coverage.md`.
+55 −0
Changes for plugins/org.etsi.mts.tdl.tx.tests/src/constraints/EXCEPTIONS.md: 55 added lines, 0 removed lines.
Original line number Diff line number Diff line
# Constraints without a fixture

Lines of the form `- Context::Name — reason` are counted as documented exclusions in the coverage
report instead of gaps. Two kinds only, both verified against the grammar and the scope provider.

**These constraints still have to be implemented.** A model that violates them can be built through
the graphical editor, an importer or an XMI file; it just cannot be written in the `.tdltx` syntax,
so this suite cannot test them.

## A. Not expressible in the textual syntax

The grammar or the scope provider makes the violation unwritable: what would violate the constraint
is either a parse error or an unresolved reference, never a constraint report.

- NamedElement::MandatoryName — every grammar rule that produces a NamedElement assigns a mandatory, non-empty identifier.
- ElementImport::ConsistentImports — the scope of `importedElement` is exactly `importedPackage.packagedElement`, so a foreign element does not link.
- Comment::CommentNestedComments — no rule returning a Comment assigns nested comments.
- Comment::CommentNestedAnnotations — no rule returning a Comment assigns annotations.
- Annotation::AnnotationNestedAnnotations — no rule returning an Annotation assigns nested annotations.
- Annotation::AnnotationNestedComments — no rule returning an Annotation assigns comments.
- StructuredDataInstance::ExistingMemberOfDataType — a member assignment's member is scoped to `allMembers()` of the instance's type, the same set the constraint checks.
- EnumDataType::EnumDataTypeExtensions — the `Enumerated` rule has no `extends` clause.
- EnumDataType::NoValueInEnumInstance — the enumeration value rule has no `= value` slot.
- MemberReference::MemberOrIndex — `member` and `collectionIndex` are exclusive grammar alternatives, each mandatory in its branch, so exactly one is always set.
- DataInstanceUse::NoItemWithDataInstance — the alternative that sets `dataInstance` admits no collection items.
- VariableUse::VariableUseComponent — without an effective component the variable itself does not link.
- TimeLabelUse::TimeLabelLocallyOrdered — time labels are scoped to their own test description, so a cross-description reference does not link.
- TimerOperation::TimerComponent — a timer only resolves through its effective component; without one the reference does not link.
- ProcedureCall::ProcedureCallTargetCount — both call rules have exactly one mandatory target.
- Target::TargetComponent — a value assignment's variable is scoped to the target component's own variables.
- ValueAssignment::AssignedProcedureParameter — the parameter is scoped to the called signature's parameters.
- GateReference::GateInstanceReference — a gate reference's gate is scoped to `component.type.allGates()`.
- TestConfiguration::OwnedComponents — a gate reference's component is scoped to the enclosing configuration.
- Block::AllowedBehaviourTypes — periodic and exceptional behaviours are reachable only through the `with { … }` fragments, which fill `periodic`/`exceptional`, never `Block.behaviour`.
- AlternativeBehaviour::AlternativeBlockCount — the grammar requires at least two blocks.
- ParallelBehaviour::ParallelBlockCount — the grammar requires at least two blocks.
- Message::DataTypeResolvable — the argument is mandatory in both message rules.
- LiteralValueUse::SpecifiedLiteralValue — the grammar admits exactly one of value, intValue and boolValue.
- PICSReference::MultiplePICS — the grammar forces an `and`/`or` comment on every reference after the first.
- RepeatedEventSequence::RepetitionOrInterval — repetitions and interval are exclusive grammar alternatives.
- EntityReference::EntityOrComponentInstance — entity and component instance are exclusive alternatives.
- Content::ContentOrValue — nested content and value are exclusive alternatives.
- VariantBinding::VariantBindingValues — the referenced value is scoped to the binding's own objective.
- ComponentAlias::AliasMandatoryName — the alias name is mandatory in the grammar.

## B. Cannot fail by construction — defects in the standard's OCL

These expressions can never evaluate to false, so the constraint text does not actually constrain
anything. They are candidates for correction in the standard rather than for a fixture.

- PredefinedFunctionCall::PredefinedFunctionCallParameters — the body is literally `true`.
- CollectionDataInstance::CollectionDataInstanceLengthConstraint — the body is literally `true`.
- RepeatedEventSequence::RepetitionCount — the body is literally `true`.
- RepeatedEventSequence::RepetitionInterval — the body is literally `true`.
- DataElementUse::DataInstanceOrArgumentsOrItemsInDataElementUse — falsifying the second disjunct requires a StructuredDataType and the third a CollectionDataType, so they cannot both be false.
+19 −0
Changes for plugins/org.etsi.mts.tdl.tx.tests/src/constraints/README.md: 19 added lines, 0 removed lines.
Original line number Diff line number Diff line
# Constraint test suite fixtures

One `.tdltx` file per expectation, any folder depth. The grammar allows one package per file, so a fixture that needs several packages is a folder whose files are loaded together (files without a header are helpers). The header declares what a file violates:

    //! expect NamedElement::DistinguishableName x2   -- exactly two reports
    //! expect Package::CyclicImports                 -- at least one report
    //! expect none                                   -- conformant model, nothing may fire
    //! allow  SomeContext::SomeName                  -- may fire, ignored

Constraint ids are `Context::Name` as written in `org.etsi.mts.tdl.model/model/*.ocl`.
Each fixture is parsed, linked against `TDL.tdltx` and validated through Xtext's resource validator, i.e. through
whichever constraint implementation the runtime configures. Syntax or linking errors fail the fixture: keep them
syntactically and referentially clean, violate only the constraint under test.

Run: `mvn verify -Dtest=ConstraintSuiteTest -DfailIfNoTests=false` from `org.etsi.mts.tdl.parent` (whole reactor).
The coverage report (`target/constraints-coverage.md`) lists every constraint of the standard with the number of
fixtures that expect it and the OCL operations of its expression; aim for one fixture per branch of the expression
(each `if`, `forAll`/`exists` with empty and non-empty collections, `oclIsKindOf` alternatives).
Naming: `<Context>_<Name>[.<variant>].tdltx`, grouped by document (`core`, `structured`, `configurations`).
+31 −0
Changes for plugins/org.etsi.mts.tdl.tx.tests/src/constraints/configurations/ComponentMerge_ComponentMergeType.conforming.tdltx: 31 added lines, 0 removed lines.
Original line number Diff line number Diff line
//! expect none
// The other branch: the component type of the target extends the component type of the source,
// so it conforms to it.
Package ComponentMergeTypesOk {
	Import all from TDL

	Type Msg
	Message Gate MsgGate accepts Msg

	Component NodeA {
		gate MsgGate ga
	}

	Component NodeB extends NodeA {
		gate MsgGate gb
	}

	Configuration Base {
		NodeA tester as Tester,
		NodeB sut as SUT,
		connect tester::ga to sut::gb
	}

	Configuration Ext with {
		Base as base,
		base::tester merged into base::sut,
		NodeA extTester as Tester,
		NodeB extSut as SUT,
		connect extTester::ga to extSut::gb
	}
}
+31 −0
Changes for plugins/org.etsi.mts.tdl.tx.tests/src/constraints/configurations/ComponentMerge_ComponentMergeType.tdltx: 31 added lines, 0 removed lines.
Original line number Diff line number Diff line
//! expect ComponentMerge::ComponentMergeType
// The target 'sut' has component type 'NodeB', which does not conform to the component type
// 'NodeA' of the merge source 'tester' - the two types are unrelated.
Package ComponentMergeTypes {
	Import all from TDL

	Type Msg
	Message Gate MsgGate accepts Msg

	Component NodeA {
		gate MsgGate ga
	}

	Component NodeB {
		gate MsgGate gb
	}

	Configuration Base {
		NodeA tester as Tester,
		NodeB sut as SUT,
		connect tester::ga to sut::gb
	}

	Configuration Ext with {
		Base as base,
		base::tester merged into base::sut,
		NodeA extTester as Tester,
		NodeB extSut as SUT,
		connect extTester::ga to extSut::gb
	}
}
Loading