Commit cb4cfa52 authored by Lluis Gifre Renom's avatar Lluis Gifre Renom
Browse files

test: document and replay ECOC optical Agentic demo

parent 77c3522f
Loading
Loading
Loading
Loading
+132 −0
Changes for src/tests/ecoc26-hyr-agentic-optical/README.md: 132 added lines, 0 removed lines.
Original line number Diff line number Diff line
# ECOC Optical Agentic Demo

Single-domain demo with two ROADM node agents, two transponder node agents,
TeraFlowSDN Optical Controller, MCP Server, and Agentic using ADK Web.
The descriptor exposes six logical transponders, T1.1-T1.3 and T2.1-T2.3.
The line endpoints are named `1` and `6`, respectively.

## Deploy

From the controller repository, launch the node agents before onboarding:

```bash
bash src/tests/ecoc26-hyr-agentic-optical/deploy-node-agents.sh
source my_deploy.sh
./deploy/all.sh
./src/tests/tools/load_scenario/run.sh \
  src/tests/ecoc26-hyr-agentic-optical/descriptors/topology.json
```

Review database reset settings in `my_deploy.sh` before redeploying an existing
controller. The node-agent launcher recreates its four named containers.
Run the scenario loader with the TFS Python dependencies and runtime variables.

Set `ADK_MODEL` to `openai/gpt-4.1-mini` in the Agentic
ConfigMap, leave `ADK_PEERS` empty, and populate `OPENAI_API_KEY` in
`agentic-secrets` from a private environment file. Anthropic is also supported
with `anthropic/claude-haiku-4-5-20251001` and `ANTHROPIC_API_KEY`.
The current validation uses OpenAI.
Never put a real API key
in the manifest, topology, transcript, or this directory.
Reapplying the example manifest resets its empty secret placeholders; inject
the key afterwards and restart `deployment/agenticservice`.

Open `/agentic/dev-ui/` on the controller and choose the `agentic` application.
Use a fresh session when validating prompt changes to avoid stale replies.

## Exercise The Demo

The `PROMPTS` list in `check_prompts.py` covers device types, explicit endpoint
names and UUIDs, optical links, services and connections, preview, creation,
inspection, and deletion. It deliberately keeps one session for follow-ups.

```bash
python src/tests/ecoc26-hyr-agentic-optical/check_prompts.py \
  --inventory-only --output /tmp/ecoc-inventory.json
python src/tests/ecoc26-hyr-agentic-optical/check_prompts.py \
  --output /tmp/ecoc-lifecycle.json
```

The second command performs real mutations. It uses unique `ecoc-check-*`
service names. Inspect the saved transcript and controller state; HTTP success
alone does not establish correct LLM behavior. On interruption, remove any
remaining services with those names through MCP or Agentic.

Expected behavior:

- R1 and R2 are optical ROADMs, not packet routers.
- `T1.1/1` identifies device T1.1 and its existing endpoint 1.
- An 800 Gbps request preserves the capacity constraint. The Optical Controller
  selects a supported transceiver mode; it is not an 800 GHz request.
- Preview retrieves inventory but submits no service.
- Unidirectional creation uses CreateService followed by UpdateService and
  confirms ACTIVE before claiming success.
- Bidirectional creation produces `<name>-forward` and `<name>-reverse`, each
  ACTIVE. Teardown must remove both and their connections.
- A failed local request is reported with its actual failure; it does not
  trigger a cross-domain workaround.

Offline regressions are in `src/agentic/tests/test_local_optical.py`.

## Validation Notes (2026-09-15)

- Local deployment uses the supplied node agents and topology: 8 devices,
  including 2 ROADMs and 6 transponders, and 14 optical links.
- Corrected executable permissions on the mounted node-agent startup scripts;
  otherwise the containers exit with status 126.
- The supplied unidirectional controller example reached ACTIVE and was
  removed. Applying a service-only descriptor with `load_scenario` subsequently
  fails its whole-scenario count assertions because the topology already
  exists; this is separate from service creation success.
- Live deterministic Agentic workflows over the official MCP SDK validated
  preview, an 800 Gbps unidirectional service, a bidirectional pair of one-way
  services, and teardown. All three services reached ACTIVE. Final MCP service
  and connection inventories were empty.
- Live Anthropic ADK Web replay completed the first seven prompts: inventory,
  ROADM classification, transponder ports, optical links, empty services,
  payload preview, and follow-up unidirectional creation. TFS independently
  reported ACTIVE with an 800 Gbps constraint and canonical endpoint IDs.
- Prompt eight (service/path inspection) exposed a wrapper passing `scope`
  to a neighbor-query function without that argument. The function now honors
  local/remote/all scope; a regression covers the actual model-facing wrapper.
  All 24 Agentic tests pass, and the fixed image is deployed locally.
- The full Anthropic post-fix conversation, including bidirectional creation
  and teardown, remains pending. Earlier deterministic MCP lifecycle tests are
  not a substitute for that conversation test.
- Removed the interrupted replay's test service through TFS-API and verified
  the service inventory was empty. Replay transcripts are local temporary
  files `/tmp/ecoc-live-prompts.json` and `/tmp/ecoc-live-prompts.log`;
  do not commit potentially sensitive live transcripts.
- Detailed optical-link queries omit spectrum maps unless requested, keeping
  port inspection small after the Optical Controller expands slot maps.

### OpenAI Replay

Validation uses `openai/gpt-4.1-mini` with its API key supplied through the
Kubernetes Secret. No real keys are stored here.

The first OpenAI replay exposed missing `context_uuid` arguments in service
and connection inventory calls. MCP validation failures were incorrectly
converted into empty inventories. The calls now supply the configured context,
and inventory failures are propagated instead of reported as successful empty
results. Regression coverage now passes 28 Agentic tests.

After rebuilding and restarting Agentic, all 13 prompts completed successfully
in one fresh session. The unidirectional service reached ACTIVE and inspection
returned `T1.1 -> R1 -> R2 -> T2.1` with real endpoint IDs. Bidirectional
creation produced two ACTIVE services with opposite paths between T1.2 and
T2.2. Agentic removed all three services; an independent MCP check confirmed
zero services and zero connections. The local transcript is
`/tmp/ecoc-openai-retest.json` with readable output in
`/tmp/ecoc-openai-retest.log`.

Observed request times were 6.84 s for unidirectional creation, 8.60 s for its
deletion, 10.41 s for bidirectional creation, and 13.37 s for deleting both
directions. These are single-run functional checks, not performance results.

Controller caveat: after baseline teardown, Context still reported occupied
slots on the removed service's links while Optical Controller deletion logs
reported those slots as available. Subsequent provisioning succeeded, but that
does not validate Context spectrum synchronization. Investigate that controller
behavior separately before demonstrating authoritative spectrum release.
No controller-spectrum fix is included in these Agentic changes.
+101 −0
Changes for src/tests/ecoc26-hyr-agentic-optical/check_prompts.py: 101 added lines, 0 removed lines.
Original line number Diff line number Diff line
# Copyright 2022-2026 ETSI SDG TeraFlowSDN (TFS) (https://tfs.etsi.org/)
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Repeat demo prompts through ADK Web with a real configured provider.

No API key is needed here: the deployed Agentic service owns provider access.
This is an explicit live test, not a CI test. Mutation prompts create services.
"""

import argparse
import json
import logging
import time
import urllib.request
import uuid
from pathlib import Path

LOGGER = logging.getLogger(__name__)

PROMPTS = [
    "Which devices do I have in my topology? Include device types.",
    "Are R1 and R2 routers or ROADMs? Check the controller.",
    "List ports in the transponders with their names and UUIDs.",
    "Which optical links connect ROADMs and transponders? "
    "Include the transponder line ports.",
    "List services and connections.",
    "Preview, but do not execute, an 800 Gbps unidirectional flexgrid "
    "optical service from T1.1/1 to T2.1/6 named {name}-uni. "
    "Show the exact service payloads.",
    "Go ahead and create that service as previewed.",
    "Inspect that service and its connection path and endpoints.",
    "Delete that service and verify it no longer exists.",
    "Create an 800 Gbps bidirectional flexgrid optical service from "
    "T1.2/1 to T2.2/6 named {name}-bi.",
    "Show both directions and their connections.",
    "Delete both directions of that bidirectional service and verify "
    "they no longer exist.",
    "List remaining services and connections.",
]


def post(url, payload):
    request = urllib.request.Request(
        url, json.dumps(payload).encode(),
        {"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(request, timeout=300) as reply:
        return json.load(reply)


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--url", default="http://127.0.0.1/agentic")
    parser.add_argument("--output", type=Path, required=True)
    parser.add_argument("--inventory-only", action="store_true")
    parser.add_argument("--prompt", action="append",
                        help="Override the sequence; repeat for follow-ups")
    args = parser.parse_args()
    logging.basicConfig(level=logging.INFO)
    name = f"ecoc-check-{uuid.uuid4().hex[:8]}"
    session = post(
        f"{args.url}/apps/agentic/users/ecoc-check/sessions", {}
    )
    records = []
    args.output.parent.mkdir(parents=True, exist_ok=True)
    prompts = args.prompt or (PROMPTS[:5] if args.inventory_only else PROMPTS)
    for prompt in prompts:
        prompt = prompt.format(name=name)
        LOGGER.info("Request %d: %s", len(records) + 1, prompt)
        start = time.monotonic()
        events = post(f"{args.url}/run", {
            "appName": "agentic", "userId": "ecoc-check",
            "sessionId": session["id"],
            "newMessage": {"role": "user", "parts": [{"text": prompt}]},
        })
        records.append({
            "session_id": session["id"], "user_id": "ecoc-check",
            "prompt": prompt, "elapsed_s": time.monotonic() - start,
            "events": events,
        })
        args.output.write_text(json.dumps(records, indent=2))
        for event in events:
            for part in (event.get("content") or {}).get("parts", []):
                if "text" in part:
                    LOGGER.info("Reply: %s", part["text"])
        LOGGER.info("Elapsed: %.3fs", records[-1]["elapsed_s"])


if __name__ == "__main__":
    main()