Add a t3s command for extracting TTCN-3 examples and importing DOCX.

# Handover: document model, Markdown reader, TTCN-3 extract

This document describes the first implementation slice of the tool in

[README.md](README.md). It records the decisions taken so far, so the work

can be picked up later without the original discussion.

## Goal of this slice

Read the Markdown sources in [`../src`](../src) into a format-neutral,

ETSI-aware document model, and build TTCN-3 extraction on top of that model.

Out of scope for this slice:

- No formatting. Extracted code is copied verbatim.

- No dependency on `github.com/nokia/ntt` or `github.com/5nord/ebnf` yet.

- No DOCX, HTML, or Markdown writer.

## Current state

`tools/` contains only a CommonMark parser in

[`internal/markdown`](internal/markdown) (a copy of `rsc.io/markdown`). None

of the README features exist yet.

The old CLI in [`../old/tools/etsi`](../old/tools/etsi) is a reference for

walking headings, lists, and quotes. Do not copy its design: its DOCX builder

walks `markdown.*` types directly, which is the coupling this slice avoids.

## Architecture

Every task reads one model of the whole deliverable. Each document format gets

a sibling package with `Read` and `Write`.

```mermaid

flowchart LR

  mdSrc[Markdown files] --> mdPkg["doc/md Read"]

  docxIn["doc/docx Read"] --> model

  mdPkg --> model["doc.Document"]

  model --> extract["extract.TTCN3"]

  model -.-> grammar["extract.Grammar later"]

  model -.-> lint["lint later"]

  model -.-> docxOut["doc/docx Write later"]

  model -.-> htmlOut["doc/html Write later"]

  model --> mdOut["doc/md Write"]

```

| Package | Done | Later |

|---|---|---|

| `internal/doc` | model | |

| `internal/doc/md` | `Read`, `Write` | |

| `internal/doc/docx` | `Read` | `Write` |

| `internal/doc/html` | | `Read`, `Write` |

| `internal/extract` | `TTCN3` | `Grammar`, keywords, modifiers |

| `cmd/t3s` | `extract examples`, `import` | |

| `internal/lint` | | yes |

Package names avoid the word "format", because the README uses it for

reformatting code.

## Model: `internal/doc`

The model imports no format package. Its node set is what `../src` uses today,

plus the ETSI elements that writers map to styles.

```go

type Pos struct{ File string; Line int } // 1-based line in the source file

type Document struct { // the whole deliverable

	Sections []*Section

}

type Section struct {

	Pos

	Number   string // "5", "5.3.1", "19.1", "A", "A.2"

	Level    int

	Title    []Inline // without the number

	Blocks   []Block

	Sections []*Section

}

// Blocks

type Paragraph struct{ Pos; Content []Inline }

type List struct{ Pos; Ordered bool; Start int; Items [][]Block }

type Quote struct{ Pos; Blocks []Block }

type Note struct{ Pos; Number string; Blocks []Block } // "NOTE:" or "NOTE 2:", label stripped

type Code struct {

	Pos             // first body line

	EndLine int     // last body line

	Lang    string  // "ttcn3", "ebnf", "syntax", "math", ""

	Meta    []Meta  // info-string markers, in source order

	Lines   []string

}

type Meta struct{ Key, Value string }

type Comment struct{ Pos; Text string } // <!-- TODO ... -->

// Inlines

type Text string

type Emph []Inline

type Strong []Inline

type InlineCode string

type Link struct{ URL string; Content []Inline }

type Ref struct{ Term string } // undefined [term]

type SoftBreak struct{}

type HardBreak struct{}

```

- `Block` and `Inline` are marker interfaces.

- One `Document` is the whole deliverable, not one file. Grammar extraction,

  DOCX export, reference resolution, and duplicate detection span files.

- `EndLine` lets a later command format a fence in place by replacing only

  its lines in the Markdown source.

- Bold-only paragraphs such as `**Semantic Description**` stay paragraphs.

- Tables and images are added when a source needs them. DOCX import will.

### Code meta

The info string of a fence carries markers that later writers act on. The

drafts use:

| Info string | Meta |

|---|---|

| ` ```ttcn3 example = "multiple assignments"` | `{example, multiple assignments}` |

| ` ```ttcn3 example` | `{example, ""}` |

| ` ```ttcn3 test="rem"` | `{test, rem}` |

| ` ```ttcn3 test` | `{test, ""}` |

| ` ```ttcn3 signature` | `{signature, ""}` |

The model keeps every block and every marker. Writers decide what to emit:

- An `example` is included in the DOCX export. When it has a value, the

  writer adds the caption in the `EX` style first, for example

  `EXAMPLE 1: assignment notation`.

- Code marked `test` is omitted from the DOCX export.

- All other code, including `ebnf`, `signature`, and unmarked `ttcn3`, is

  included.

## Reader: `internal/doc/md`

```go

func Read(d *doc.Document, path string, src []byte) error

```

It parses one file with `markdown.Parser` and appends that file's sections to

`d`.

### Numbering

- The filename prefix numbers the first heading of the file. Leading zeros are

  stripped and an annex letter is allowed:

  | File | Number |

  |---|---|

  | `05-basic-language-elements.md` | `5` |

  | `8.2.1-declaring-module-parameters.md` | `8.2.1` |

  | `A-lexical-structure-and-grammar.md` | `A` |

  | `00-type-expressions.md` | `0` |

- A heading that starts with a number, such as `## 19.1 Assignment`, uses

  that number, and the number is removed from `Title`.

- Deeper headings count up under their parent. The headings of

  `05-basic-language-elements.md` become `5.1 General`,

  `5.2 Identifiers and keywords`, `5.3 Scope rules`, `5.3.1 General`, and so

  on.

- These are errors:

  - content before the first heading;

  - a later heading at or above the level of the first heading;

  - a file with neither a filename prefix nor a numbered first heading.

### Conversion

- A quote whose first paragraph starts with `NOTE:` or `NOTE <n>:` becomes

  `Note`. Every other quote stays `Quote`.

- Each fence becomes `Code`:

  - `Lang` is the first word of the info string.

  - Every following word is one `Meta`, either a bare flag or

    `key="value"`, with optional spaces around `=`.

  - `Pos.Line` and `EndLine` cover the body lines, not the fences.

- Indented code becomes `Code` with an empty `Lang`.

- An HTML block that is only a comment becomes `Comment`. Any other HTML block

  is an error.

- Text such as `[variables]` with no link definition becomes `Ref`. Escaped

  text becomes `Text`.

- Any other construct (table, image, thematic break, strikethrough, emoji) is

  an error at `file:line`, so content is never dropped silently.

### Tests

- A conversion fixture covers every rule above: an annex file, an explicit

  heading number, a fence inside a list item, `NOTE 2:` with two paragraphs,

  `example = "..."`, a bare `example`, and `test="..."`. Assert section

  numbers, titles, node types, `Meta` keys and values, `Pos.Line`, and

  `EndLine`.

- Read every `../src/*.md` into a fresh `Document` without error. This test

  fails when a draft starts using something the model does not cover. Use a

  fresh `Document` per file, because the drafts currently share clause

  numbers (see below).

## Extract: `internal/extract`

```go

type File struct {

	Name    string

	Content []byte

}

func TTCN3(d *doc.Document) ([]File, error)

```

It returns files in memory and does not touch the filesystem.

- Walk sections in document order. Descend into blocks, lists, quotes, and

  notes. Collect every `Code` with `Lang == "ttcn3"`: examples, tests, and

  unmarked blocks.

- Name each file `<section number>-<kind><n>.ttcn3`. `kind` is the first of

  `example` or `test` in `Meta` order, and `code` for any other block. `n`

  counts per section and kind, starting at 1. Clause 10 with two examples,

  one test, and one unmarked block yields `10-example1.ttcn3`,

  `10-example2.ttcn3`, `10-test1.ttcn3`, and `10-code1.ttcn3`.

- The header copies the meta as TTCN-3 comments, then the clause:

  ```ttcn3

  // @title: multiple assignments

  // @clause: 10

  ```

  - An `example` value is written as `@title`, the name used in the README.

  - Every other marker with a value is written as `@<key>`, for example

    `// @test: rem`.

  - A bare marker adds no line.

  - `// @clause: <section number>` is always the last header line.

- The fence body follows, copied verbatim with its whitespace, then a trailing

  newline. Do not parse or format it.

- Two blocks that produce the same name are an error naming both `file:line`

  positions.

Tests cover names, headers, a byte-exact body, skipped `ebnf` code,

per-section and per-kind counting, and the duplicate error.

## Known issues in the drafts

These will surface once the slice runs on all of `../src`:

- `16-declaring-dynamic-behaviour.md` and `16-parameterization.md` both

  claim clause 16, so their extracted names collide.

- `05.4.2-actual-parameters.md` claims 5.4.2, but in

  `05-basic-language-elements.md` 5.4 is "Ordering of language elements".

- `19.11-log-statement.md` line 44 has ` ```ttcne `, a typo for `ttcn3`, so

  that example is not extracted.

## Later slices on the same model

- **Command line.** [`cmd/t3s`](cmd/t3s) implements

  `t3s extract examples -o dir [path...]` and

  `t3s import -o dir [-f] file.docx...`.

  `extract examples` writes `extract.TTCN3` results into the directory.

  Further commands wait on formatting, grammar, lint, and DOCX export.

- **TTCN-3 parse and format** use `github.com/nokia/ntt`:

  - Parse `Code.Lines` with `ttcn3/syntax.Parse`, which accepts declaration

    and statement fragments.

  - Format with `ttcn3/format.Fprint`, which refuses input with syntax errors.

  - Format in place by replacing lines `Pos.Line` to `EndLine` in the

    Markdown source. Never rewrite the whole file through the model.

  - Report errors at `Code.Pos.Line` plus the line inside the fence.

  - Extracted files stay verbatim.

- **Grammar** uses `github.com/5nord/ebnf`:

  - `extract.Grammar` joins `ebnf` code in document order and keeps a map

    from every grammar line back to its Markdown `Pos`.

  - It then calls `ebnf.Parse` and `ebnf.Verify(grammar, start)`. `Verify`

    checks that every production is defined and reachable, which is the

    README's lint rule. The start symbol is the first non-lexical

    production.

  - Keywords and modifiers come from the grammar's token literals.

- **DOCX and HTML writers** map `Section.Number`, `Note` (`NO`), titled

  examples (`EX`), and code (`PL`) to ETSI styles, and apply the meta rules

  above. The ETSI DOCX template is still needed; [`../assets`](../assets)

  holds screenshots only. EBNF production names inside code become links to

  their definitions.

- **DOCX import** is implemented. See the slice below.

## DOCX import slice

`docx.Read` appends the sections of one DOCX file to a `doc.Document`.

`md.Write` writes one section tree as canonical Markdown.

`t3s import` writes one file per top-level section.

`Pos.Line` for DOCX input is the 1-based paragraph index in document order,

including skipped paragraphs and paragraphs inside tables.

### What is read

- Everything before the first `heading 1` is skipped (the cover page).

  TOC styles (`toc N`, `VerzeichnisN`) are skipped anywhere.

- Style names are used, not style IDs. `berschrift1` and `Heading1` are both

  "heading 1". `B1` in one file is the style named "B1+".

- The first heading 1 is numbered from the file name (`06 - Data types.docx`

  is clause 6, `09 Port types.docx` is clause 9). Deeper headings are

  counted. A number typed into a heading, including one glued to the title

  (`19.1Assignments`), is removed from the title and is not used. A later

  heading at the same level starts a new top-level section. One with no

  number of its own is numbered from 1000.

- Tracked insertions and moves are kept. Deletions are dropped. Field codes

  are dropped and the cached result is kept, so an unresolved reference stays

  "Error: Reference source not found". Review comments are ignored.

- `PL` paragraphs become one code block. The language is `syntax` after a

  "Syntactical Structure" or "Syntactic(al) Description" paragraph, `ebnf`

  when a line matches `Name ::=`, and `ttcn3` otherwise. An `EX` paragraph

  `EXAMPLE n: title` becomes `example="title"` on the following code.

  `NOTE n:` becomes a note. List styles and numbered paragraphs become

  lists; a typed `a)` or `1)` marker is removed.

- Tables become `doc.Table` and are written as GFM tables. The Markdown

  reader accepts tables, and `<br>` in a cell is a hard break.

- Bold is strong, italic is emphasis, Courier New is code, and a hyperlink

  with a URL is a link. An image becomes a `<!-- TODO: image -->` comment.

### Emphasis repair

Runs are merged, then repaired before they are stored:

- A bold or italic `*` that sits against a letter or another `*` is marker

  debris and is removed. A spaced `*` (`omit, * or ifpresent`) stays.

- A formatting change inside a word is joined when the shorter side is at

  most 3 letters and is not a whole stopword (`the`, `for`, `a`, ...). The

  longer side wins, and a tie keeps the first side. So `****bool**ean**` and

  bold `bool` plus plain `ean` both become `**boolean**`. Italic `me` plus

  plain `chanism` becomes plain `mechanism`. Letters are not joined to digits.

- A missing space between words is not repaired (`theAnyElementsOrNone`,

  `forelse`).

### Markdown writer

`md.Write` writes the section as a file whose name carries the clause number.

Headings omit that number unless a child was numbered explicitly (for example

`19.1.7`), so a round trip through `md.Read` restores it. Paragraphs are one

sentence per line, with abbreviations such as `e.g.` and `i.e.` left intact.

Lines longer than 80 columns wrap at `,` `;` `:` first, then at a space.

Lists have no blank line between items. Code fences carry the language and

the meta (`example="title"`).

### Known gaps

- Word's paragraph-mark revisions are not interpreted.

- A formatting change inside a word is joined only for a short fragment.

  A missing space is left as it is.

- Images are comments, not figures. The model still has no image node.

- A second unnumbered top-level heading, such as "History" at the end of a

  chapter file, is numbered from 1000.

The markdown documents are the single source of truth. They rely on custom

conventions to make automation and maintenance simpler to achieve.

## Commands

The command is `t3s` (`cmd/t3s`).

```text

t3s extract examples -o dir [path...]

t3s import -o dir [-f] file.docx...

t3s help [extract|import]

t3s version

```

`extract examples` reads Markdown in clause order and writes one `.ttcn3` file

per TTCN-3 code block. With no paths it reads `./src`. `import` writes one

canonical Markdown file per top-level section of each DOCX file. It refuses

to overwrite an existing file unless `-f` is given.

## Features

- [ ] Format TTCN-3 code blocks inplace.

- [ ] Extract TTCN-3 code blocks

- [x] Extract TTCN-3 code blocks

- [ ] Extract TTCN-3 grammar

- [ ] Code Linting

- [ ] DOCX export

- [ ] DOCX import

- [x] DOCX import

### Extract TTCN-3 code blocks

package main

import (

	"flag"

	"fmt"

	"io"

	"os"

	"path/filepath"

	"tools/internal/extract"

)

func cmdExtract(args []string, stdout, stderr io.Writer) int {

	if len(args) == 0 {

		fmt.Fprint(stderr, extractHelp)

		return 2

	}

	switch args[0] {

	case "-h", "--help", "help":

		fmt.Fprint(stdout, extractHelp)

		return 0

	case "examples":

		return cmdExtractExamples(args[1:], stdout, stderr)

	default:

		if len(args[0]) > 0 && args[0][0] == '-' {

			fmt.Fprintln(stderr, "t3s: extract target required")

		} else {

			fmt.Fprintf(stderr, "t3s: unknown extract target %q\n", args[0])

		}

		fmt.Fprint(stderr, extractHelp)

		return 2

	}

}

func cmdExtractExamples(args []string, stdout, stderr io.Writer) int {

	var out string

	fs := flag.NewFlagSet("t3s extract examples", flag.ContinueOnError)

	fs.SetOutput(io.Discard)

	fs.StringVar(&out, "o", "", "directory to write the .ttcn3 files into")

	if code, ok := parseFlags(fs, args, stdout, stderr, extractHelp); !ok {

		return code

	}

	if out == "" {

		fmt.Fprintln(stderr, "t3s: -o is required")

		fmt.Fprint(stderr, extractHelp)

		return 2

	}

	d, err := readMarkdown(fs.Args())

	if err != nil {

		return report(stderr, err, extractHelp)

	}

	files, err := extract.TTCN3(d)

	if err != nil {

		fmt.Fprintln(stderr, err)

		return 1

	}

	outFiles := make([]outFile, len(files))

	for i, f := range files {

		outFiles[i] = outFile{Name: f.Name, Content: f.Content}

	}

	if err := writeFiles(out, outFiles, true); err != nil {

		fmt.Fprintln(stderr, err)

		return 1

	}

	return 0

}

func parseFlags(fs *flag.FlagSet, args []string, stdout, stderr io.Writer, usage string) (int, bool) {

	if err := fs.Parse(args); err != nil {

		if err == flag.ErrHelp {

			fmt.Fprint(stdout, usage)

			return 0, false

		}

		fmt.Fprintln(stderr, "t3s:", err)

		fmt.Fprint(stderr, usage)

		return 2, false

	}

	return 0, true

}

type outFile struct {

	Name    string

	Source  string

	Content []byte

}

func writeFiles(dir string, files []outFile, force bool) error {

	seen := map[string]string{}

	for _, f := range files {

		if !validName(f.Name) {

			return fmt.Errorf("t3s: invalid output name %q", f.Name)

		}

		if prev, ok := seen[f.Name]; ok {

			return fmt.Errorf("%s: duplicate output from %s and %s", f.Name, prev, f.Source)

		}

		seen[f.Name] = f.Source

	}

	if !force {

		for _, f := range files {

			path := filepath.Join(dir, f.Name)

			if _, err := os.Stat(path); err == nil {

				return fmt.Errorf("t3s: %s exists (use -f to overwrite)", path)

			} else if !os.IsNotExist(err) {

				return err

			}

		}

	}

	if err := os.MkdirAll(dir, 0o755); err != nil {

		return err

	}

	for _, f := range files {

		if err := os.WriteFile(filepath.Join(dir, f.Name), f.Content, 0o644); err != nil {

			return err

		}

	}

	return nil

}

func validName(name string) bool {

	return name != "" && name != "." && name != ".." && name == filepath.Base(name)

}

package main

import (

	"bytes"

	"flag"

	"fmt"

	"io"

	"os"

	"path/filepath"

	"strconv"

	"strings"

	"unicode"

	"tools/internal/doc"

	"tools/internal/doc/docx"

	"tools/internal/doc/md"

)

func cmdImport(args []string, stdout, stderr io.Writer) int {

	var (

		out   string

		force bool

	)

	fs := flag.NewFlagSet("t3s import", flag.ContinueOnError)

	fs.SetOutput(io.Discard)

	fs.StringVar(&out, "o", "", "directory to write the Markdown files into")

	fs.BoolVar(&force, "f", false, "overwrite existing files")

	if code, ok := parseFlags(fs, args, stdout, stderr, importHelp); !ok {

		return code

	}

	if out == "" {

		fmt.Fprintln(stderr, "t3s: -o is required")

		fmt.Fprint(stderr, importHelp)

		return 2

	}

	paths := fs.Args()

	if len(paths) == 0 {

		fmt.Fprintln(stderr, "t3s: at least one .docx file is required")

		fmt.Fprint(stderr, importHelp)

		return 2

	}

	for _, p := range paths {

		fi, err := os.Stat(p)

		if err != nil {

			fmt.Fprintln(stderr, err)

			return 1

		}

		if fi.IsDir() || !strings.EqualFold(filepath.Ext(p), ".docx") {

			fmt.Fprintf(stderr, "t3s: %s: want a .docx file\n", p)

			fmt.Fprint(stderr, importHelp)

			return 2

		}

	}

	files, err := importDocs(paths)

	if err != nil {

		fmt.Fprintln(stderr, err)

		return 1

	}

	if err := writeFiles(out, files, force); err != nil {

		fmt.Fprintln(stderr, err)

		return 1

	}

	return 0

}

func importDocs(paths []string) ([]outFile, error) {

	var all []outFile

	for _, path := range paths {

		b, err := os.ReadFile(path)

		if err != nil {

			return nil, err

		}

		d := &doc.Document{}

		if err := docx.Read(d, path, b); err != nil {

			return nil, err

		}

		for _, s := range d.Sections {

			var buf bytes.Buffer

			if err := md.Write(&buf, s); err != nil {

				return nil, err

			}

			all = append(all, outFile{

				Name:    markdownName(s.Number, inlineText(s.Title)),

				Source:  fmt.Sprintf("%s:%d", s.File, s.Line),

				Content: buf.Bytes(),

			})

		}

	}

	return all, nil

}

// markdownName is the source file name for a clause. A single integer below

// 100 is padded to two digits, so clause 6 is "06-data-types.md".

func markdownName(number, title string) string {

	num := number

	if n, err := strconv.Atoi(number); err == nil && n < 100 {

		num = fmt.Sprintf("%02d", n)

	}

	slug := slug(title)

	if slug == "" {

		return num + ".md"

	}

	return num + "-" + slug + ".md"

}

func slug(title string) string {

	var b strings.Builder

	dash := false

	for _, r := range strings.ToLower(title) {

		if unicode.IsLetter(r) || unicode.IsDigit(r) {

			b.WriteRune(r)

			dash = false

			continue

		}

		if !dash && b.Len() > 0 {

			b.WriteByte('-')

			dash = true

		}

	}

	return strings.TrimRight(b.String(), "-")

}

func inlineText(ins []doc.Inline) string {

	var b strings.Builder

	var walk func([]doc.Inline)

	walk = func(ins []doc.Inline) {

		for _, in := range ins {

			switch in := in.(type) {

			case doc.Text:

				b.WriteString(string(in))

			case doc.Emph:

				walk([]doc.Inline(in))

			case doc.Strong:

				walk([]doc.Inline(in))

			case doc.InlineCode:

				b.WriteString(string(in))

			case doc.Link:

				walk(in.Content)

			case doc.Ref:

				b.WriteString(in.Term)

			case doc.SoftBreak, doc.HardBreak: