diff --git a/docs/virtual_switches.md b/docs/virtual_switches.md index 9fada36c..14b7743c 100644 --- a/docs/virtual_switches.md +++ b/docs/virtual_switches.md @@ -1,5 +1,7 @@ # Using virtual switches with Hotstack +## Creating a switch image + ```bash openstack image create hotstack-switch \ --disk-format qcow2 \ @@ -7,3 +9,199 @@ openstack image create hotstack-switch \ --property hw_firmware_type=uefi \ --property hw_machine_type=q35 --public ``` + +## Network wiring: OpenStack networks as bridges + +Hotstack uses a special network architecture to connect VMs (like baremetal +nodes managed by Ironic) to ports on the virtual switch. This approach uses +**OpenStack Neutron networks as L2 bridges** between VMs and switch ports. + +### Architecture overview + +The architecture consists of three key components: + +1. **Trunk port on the switch**: A Neutron trunk port that carries multiple + VLANs to the switch +2. **Bridge networks**: Small point-to-point Neutron networks connecting each + VM to the switch +3. **VLAN configuration inside the switch**: The switch's internal + configuration determines which VLAN each port belongs to + +### How it works + +The key insight is that **the switch's internal configuration determines +VLAN membership**, not OpenStack: + +- The bridge network (e.g. `ironic0-br-net`) provides L2 connectivity + between the VM and the switch +- Inside the switch, you configure which VLAN the port (connected to the + bridge network) belongs to +- When you change the VLAN configuration of the switch port, the VM + effectively "moves" to a different VLAN + +**Example flow:** + +1. VM `ironic0` is connected to `ironic0-br-net` +2. The switch port `ethernet1/2` is also connected to `ironic0-br-net` +3. Inside the switch, configure `ethernet1/2` to be an access port on + VLAN 101 +4. Traffic from `ironic0` now flows through the bridge network to the + switch, where it enters VLAN 101 +5. VLAN 101 traffic exits the switch through the trunk port (tagged with + VLAN 101) +6. OpenStack Neutron receives this as traffic on the `ironic-net` network + (which is VLAN 101 on the trunk) + +**To move the VM to a different VLAN:** + +1. Reconfigure switch port `ethernet1/2` to be on VLAN 103 instead +2. Now traffic from `ironic0` enters VLAN 103 and exits as traffic on the + `tenant-vlan103` network +3. **No changes needed in OpenStack** - the bridge network stays the same + +### Benefits of this approach + +- **Flexibility**: VMs can be moved between VLANs by reconfiguring the + switch, not by reconfiguring OpenStack +- **Realistic testing**: Mimics how physical networks work with switches + controlling VLAN membership +- **Simplified OpenStack config**: Each VM just needs one static connection + (the bridge network) +- **Switch-driven networking**: The switch becomes the central point of + control for network topology, just like in a physical datacenter + +### Implementation in Heat templates + +#### 1. Switch trunk port setup + +The switch has a trunk port that connects to multiple VLAN networks as +subports: + +```yaml +switch-trunk-parent-port: + type: OS::Neutron::Port + properties: + network: {get_resource: trunk-net} + port_security_enabled: false + +switch-trunk-ironic-port: + type: OS::Neutron::Port + properties: + network: {get_resource: ironic-net} + port_security_enabled: false + +switch-trunk-tenant-vlan103-port: + type: OS::Neutron::Port + properties: + network: {get_resource: tenant-vlan103} + port_security_enabled: false + +switch-trunk-tenant-vlan104-port: + type: OS::Neutron::Port + properties: + network: {get_resource: tenant-vlan104} + port_security_enabled: false + +switch-trunk-tenant-vlan105-port: + type: OS::Neutron::Port + properties: + network: {get_resource: tenant-vlan105} + port_security_enabled: false + +switch-trunk: + type: OS::Neutron::Trunk + properties: + port: {get_resource: switch-trunk-parent-port} + sub_ports: + # Ironic VLAN + - port: {get_resource: switch-trunk-ironic-port} + segmentation_id: 101 + segmentation_type: vlan + # Tenant VLANs + - port: {get_resource: switch-trunk-tenant-vlan103-port} + segmentation_id: 103 + segmentation_type: vlan + - port: {get_resource: switch-trunk-tenant-vlan104-port} + segmentation_id: 104 + segmentation_type: vlan + - port: {get_resource: switch-trunk-tenant-vlan105-port} + segmentation_id: 105 + segmentation_type: vlan +``` + +This trunk port presents multiple VLANs to the switch VM, just like a +physical trunk port would. + +#### 2. Bridge networks connecting VMs to the switch + +Each VM that needs to connect to the switch gets a dedicated "bridge +network" - a small Neutron network that acts as a point-to-point L2 +connection: + +```yaml +ironic0-br-net: + type: OS::Neutron::Net + properties: + port_security_enabled: false + +ironic1-br-net: + type: OS::Neutron::Net + properties: + port_security_enabled: false +``` + +Both the VM and the switch get a port on this bridge network: + +```yaml +switch-ironic0-br-port: + type: OS::Neutron::Port + properties: + network: {get_resource: ironic0-br-net} + port_security_enabled: false + +switch-ironic1-br-port: + type: OS::Neutron::Port + properties: + network: {get_resource: ironic1-br-net} + port_security_enabled: false + +switch: + type: OS::Nova::Server + properties: + # ... image, flavor, and disk configuration ... + networks: + - port: {get_resource: switch-machine-port} + - port: {get_attr: [switch-trunk, port_id]} + - port: {get_resource: switch-ironic0-br-port} + - port: {get_resource: switch-ironic1-br-port} +``` + +The VMs also connect to their respective bridge networks: + +```yaml +ironic0-port: + type: OS::Neutron::Port + properties: + network: {get_resource: ironic0-br-net} + port_security_enabled: false + +ironic0: + type: OS::Nova::Server + properties: + # ... image, flavor, and disk configuration ... + networks: + - port: {get_resource: ironic0-port} + +ironic1-port: + type: OS::Neutron::Port + properties: + network: {get_resource: ironic1-br-net} + port_security_enabled: false + +ironic1: + type: OS::Nova::Server + properties: + # ... image, flavor, and disk configuration ... + networks: + - port: {get_resource: ironic1-port} +``` diff --git a/images/.gitignore b/images/.gitignore index 8752f8fd..f3edd30a 100644 --- a/images/.gitignore +++ b/images/.gitignore @@ -1,7 +1,11 @@ +# Built images controller.qcow2 blank-image.qcow2 nat64-appliance.qcow2 -switch-host.qcow2 -switch-host-base.qcow2 + +# Temporary files +*.tmp + +# NAT64 build artifacts .ci-framework/ .nat64-build/ diff --git a/images/Makefile b/images/Makefile index dfe57e36..4d1401d9 100644 --- a/images/Makefile +++ b/images/Makefile @@ -9,18 +9,6 @@ NAT64_IMAGE_NAME ?= nat64-appliance.qcow2 NAT64_IMAGE_FORMAT ?= raw NAT64_CIFMW_DIR ?= $(CURDIR)/.ci-framework NAT64_BASEDIR ?= $(CURDIR)/.nat64-build -SWITCH_HOST_IMAGE_URL ?= https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-x86_64-9-latest.x86_64.qcow2 -SWITCH_HOST_BASE_IMAGE ?= switch-host-base.qcow2 -SWITCH_HOST_IMAGE_NAME ?= switch-host.qcow2 -SWITCH_HOST_IMAGE_FORMAT ?= raw -SWITCH_HOST_INSTALL_PACKAGES ?= libvirt,qemu-kvm,qemu-img,expect,unzip,jq,iproute,nmap-ncat,telnet,git,vim-enhanced,tmux,bind-utils,bash-completion,nmstate,tcpdump,python3-jinja2 - -# Switch vendor image locations (optional, leave empty if not available) -# These will be copied into the image at /opt// during build -FORCE10_10_IMAGE ?= -FORCE10_9_IMAGE ?= -NXOS_IMAGE ?= -SONIC_IMAGE ?= all: controller blank nat64 @@ -86,85 +74,3 @@ nat64_clean: rm -rf $(NAT64_CIFMW_DIR) sudo rm -rf $(NAT64_BASEDIR) rm -rf $(HOME)/test-python - -switch-host: switch-host_download switch-host_copy switch-host_customize switch-host_convert - -switch-host_download: - @if [ ! -f $(SWITCH_HOST_BASE_IMAGE) ]; then \ - echo "Downloading base image to $(SWITCH_HOST_BASE_IMAGE)..."; \ - curl -L -o $(SWITCH_HOST_BASE_IMAGE) $(SWITCH_HOST_IMAGE_URL); \ - else \ - echo "Base image $(SWITCH_HOST_BASE_IMAGE) already exists, skipping download."; \ - fi - -switch-host_copy: - @echo "Creating working copy: $(SWITCH_HOST_BASE_IMAGE) -> $(SWITCH_HOST_IMAGE_NAME)" - @cp $(SWITCH_HOST_BASE_IMAGE) $(SWITCH_HOST_IMAGE_NAME) - -switch-host_customize: - @echo "Customizing switch-host image..." -ifneq ($(FORCE10_10_IMAGE),) - @test -f $(FORCE10_10_IMAGE) || (echo "ERROR: Force10 OS10 image not found: $(FORCE10_10_IMAGE)" && exit 1) -endif -ifneq ($(FORCE10_9_IMAGE),) - @test -f $(FORCE10_9_IMAGE) || (echo "ERROR: Force10 OS9 image not found: $(FORCE10_9_IMAGE)" && exit 1) -endif -ifneq ($(NXOS_IMAGE),) - @test -f $(NXOS_IMAGE) || (echo "ERROR: NXOS image not found: $(NXOS_IMAGE)" && exit 1) -endif -ifneq ($(SONIC_IMAGE),) - @test -f $(SONIC_IMAGE) || (echo "ERROR: SONiC image not found: $(SONIC_IMAGE)" && exit 1) -endif - virt-customize -a $(SWITCH_HOST_IMAGE_NAME) \ - --install $(SWITCH_HOST_INSTALL_PACKAGES) \ - --timezone UTC \ - --copy-in switch-host-scripts/start-switch-vm.sh:/usr/local/bin \ - --chmod 0755:/usr/local/bin/start-switch-vm.sh \ - --run-command 'mkdir -p /usr/local/lib/hotstack-switch-vm' \ - --copy-in switch-host-scripts/common.sh:/usr/local/lib/hotstack-switch-vm \ - --copy-in switch-host-scripts/bridges.nmstate.yaml.j2:/usr/local/lib/hotstack-switch-vm \ - --chmod 0644:/usr/local/lib/hotstack-switch-vm/common.sh \ - --chmod 0644:/usr/local/lib/hotstack-switch-vm/bridges.nmstate.yaml.j2 \ - --run-command 'mkdir -p /etc/hotstack-switch-vm' \ - --run-command 'mkdir -p /var/lib/hotstack-switch-vm' \ - $(if $(FORCE10_10_IMAGE), \ - --copy-in switch-host-scripts/force10_10:/usr/local/lib/hotstack-switch-vm \ - --chmod 0755:/usr/local/lib/hotstack-switch-vm/force10_10/setup.sh \ - --chmod 0755:/usr/local/lib/hotstack-switch-vm/force10_10/wait.sh \ - --chmod 0755:/usr/local/lib/hotstack-switch-vm/force10_10/configure.sh \ - --chmod 0644:/usr/local/lib/hotstack-switch-vm/force10_10/domain.xml.j2 \ - --run-command 'mkdir -p /opt/force10_10' \ - --copy-in $(FORCE10_10_IMAGE):/opt/force10_10 \ - --run-command 'basename $(FORCE10_10_IMAGE) > /opt/force10_10/image-info.txt',) \ - $(if $(FORCE10_9_IMAGE), \ - --run-command 'mkdir -p /opt/force10_9' \ - --copy-in $(FORCE10_9_IMAGE):/opt/force10_9,) \ - $(if $(NXOS_IMAGE), \ - --copy-in switch-host-scripts/nxos:/usr/local/lib/hotstack-switch-vm \ - --chmod 0755:/usr/local/lib/hotstack-switch-vm/nxos/setup.sh \ - --chmod 0755:/usr/local/lib/hotstack-switch-vm/nxos/wait.sh \ - --chmod 0755:/usr/local/lib/hotstack-switch-vm/nxos/configure.sh \ - --chmod 0644:/usr/local/lib/hotstack-switch-vm/nxos/domain.xml.j2 \ - --run-command 'mkdir -p /opt/nxos' \ - --copy-in $(NXOS_IMAGE):/opt/nxos \ - --run-command 'basename $(NXOS_IMAGE) > /opt/nxos/image-info.txt',) \ - $(if $(SONIC_IMAGE), \ - --run-command 'mkdir -p /opt/sonic' \ - --copy-in $(SONIC_IMAGE):/opt/sonic,) \ - --selinux-relabel - @echo "Switch-host image customization complete" - -switch-host_convert: -ifeq ($(SWITCH_HOST_IMAGE_FORMAT),raw) - @echo "Converting switch-host image to raw format (in-place)..." - qemu-img convert -p -f qcow2 -O raw $(SWITCH_HOST_IMAGE_NAME) $(SWITCH_HOST_IMAGE_NAME).tmp - mv $(SWITCH_HOST_IMAGE_NAME).tmp $(SWITCH_HOST_IMAGE_NAME) - @echo "Switch-host image converted to raw format: $(SWITCH_HOST_IMAGE_NAME)" -endif - -switch-host_clean: - rm -f $(SWITCH_HOST_IMAGE_NAME) - rm -f $(SWITCH_HOST_IMAGE_NAME).tmp - -switch-host_clean_all: switch-host_clean - rm -f $(SWITCH_HOST_BASE_IMAGE) diff --git a/images/README.md b/images/README.md index e96a67cd..9e6f8733 100644 --- a/images/README.md +++ b/images/README.md @@ -16,8 +16,9 @@ deployments. The tasks are executed using the `make` utility. Redfish virtual BMC - **nat64**: A NAT64 appliance image built using ci-framework for IPv6-only environments -- **switch-host** (experimental): A CentOS 9 Stream image with libvirt/qemu for - running virtual network switches using nested virtualization + +**Note**: Switch host images have been moved to `../switch-images/`. See +`../switch-images/README.md` for building virtual network switch images. ## Variables @@ -39,16 +40,6 @@ deployments. The tasks are executed using the `make` utility. `.ci-framework`). - `NAT64_BASEDIR`: Build directory for NAT64 appliance artifacts (default: `.nat64-build`). -- `SWITCH_HOST_IMAGE_URL`: The URL to download the CentOS 9 Stream image for - switch host. -- `SWITCH_HOST_IMAGE_NAME`: The name of the switch host image file. -- `SWITCH_HOST_IMAGE_FORMAT`: The desired format for the switch host image - (default: `raw`). Set to `qcow2` to keep the original format. -- `SWITCH_HOST_INSTALL_PACKAGES`: A list of packages to install on the switch - host image (includes libvirt, qemu-kvm, networking tools). -- `FORCE10_10_IMAGE`: Path to Force10 OS10 image file (zip archive like - `OS10_Virtualization_10.6.0.2.74V.zip`). Will be copied to `/opt/force10_10/` - in the image. (Experimental support only) **Note**: Raw format is required for cloud backends using Ceph, as Ceph cannot directly use qcow2 images for VM disks. @@ -80,18 +71,6 @@ directly use qcow2 images for VM disks. - `nat64_convert`: A target that converts the image to the format specified by `NAT64_IMAGE_FORMAT` (in-place conversion if `raw`). - `nat64_clean`: A target that removes the NAT64 image and build artifacts. -- `switch-host` (experimental): A standalone target that depends on - `switch-host_download`, `switch-host_customize`, and `switch-host_convert`. - Not included in `all` or `clean` targets. - - `switch-host_download`: A target that downloads the base image from the - specified URL. - - `switch-host_customize`: A target that customizes the downloaded image by - installing packages (libvirt, qemu, networking tools), copying helper - scripts to `/usr/local/bin/`, installing the systemd service, and creating - necessary directories. - - `switch-host_convert`: A target that converts the image to the format - specified by `SWITCH_HOST_IMAGE_FORMAT` (in-place conversion if `raw`). - - `switch-host_clean`: A target that removes the switch-host image file. ## Examples @@ -167,38 +146,8 @@ make clean environment is created at `~/test-python`. All of these can be cleaned up with `make nat64_clean`. -### Building and uploading the switch-host image to glance - -1. Build the switch-host image: - - ```shell - make switch-host - ``` - - Or with vendor switch images pre-installed: - - ```shell - make switch-host \ - FORCE10_10_IMAGE=/path/to/OS10_Virtualization_10.6.0.2.74V.zip - ``` - - This will: - - Download CentOS 9 Stream image - - Install libvirt, qemu-kvm, and networking tools - - Copy helper scripts to `/usr/local/bin/` - - Install systemd service for managing nested switch VMs - - Create directories for each switch model under `/opt/` - - Copy any provided vendor switch images to their respective directories - -2. Upload the switch-host image to Glance: - - ```shell - openstack image create hotstack-switch-host \ - --disk-format raw \ - --file switch-host.qcow2 \ - --property hw_firmware_type=uefi \ - --property hw_machine_type=q35 - ``` +## Switch Images - See `switch-host-scripts/README.md` for details on switch image requirements - and configuration. +Switch host images have been moved to a separate directory for better +organization and modularity. See `../switch-images/README.md` for complete +documentation on building virtual network switch images. diff --git a/images/switch-host-scripts/bridges.nmstate.yaml.j2 b/images/switch-host-scripts/bridges.nmstate.yaml.j2 deleted file mode 100644 index 13e84cdf..00000000 --- a/images/switch-host-scripts/bridges.nmstate.yaml.j2 +++ /dev/null @@ -1,20 +0,0 @@ -interfaces: -{% for bridge in bridges %} - - name: {{ bridge.name }} - type: linux-bridge - state: up - ipv4: - enabled: false - ipv6: - enabled: false - bridge: - port: - - name: {{ bridge.port }} - - name: {{ bridge.port }} - type: ethernet - state: up - ipv4: - enabled: false - ipv6: - enabled: false -{% endfor %} diff --git a/images/switch-host-scripts/nxos/domain.xml.j2 b/images/switch-host-scripts/nxos/domain.xml.j2 deleted file mode 100644 index ed08559a..00000000 --- a/images/switch-host-scripts/nxos/domain.xml.j2 +++ /dev/null @@ -1,63 +0,0 @@ - - {{ vm_name }} - 8 - 2 - - hvm - - - /usr/share/edk2/ovmf/OVMF_CODE.fd - /var/lib/libvirt/qemu/nvram/{{ vm_name }}_VARS.fd - - - - - - - - destroy - restart - destroy - - - - - - - -
- - - - -
- - - - - - - - - - - - - - - - - - - - - {% for i in range(num_bridges) %} - - - - - {% endfor %} - - diff --git a/images/switch-host-scripts/nxos/setup.sh b/images/switch-host-scripts/nxos/setup.sh deleted file mode 100644 index 9bc44c2f..00000000 --- a/images/switch-host-scripts/nxos/setup.sh +++ /dev/null @@ -1,119 +0,0 @@ -#!/bin/bash -# Setup and start Cisco NXOS virtual switch using libvirt -# NXOS uses POAP (Power-On Auto Provisioning) for automatic configuration - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -LIB_DIR="${LIB_DIR:-/usr/local/lib/hotstack-switch-vm}" - -# Source common functions -# shellcheck disable=SC1091 -source "$LIB_DIR/common.sh" - -WORK_DIR="${WORK_DIR:-/var/lib/hotstack-switch-vm}" -CONSOLE_PORT="${CONSOLE_PORT:-55001}" -VM_NAME="${VM_NAME:-cisco-nxos}" - -# Look for NXOS image in default location or use BASE_IMAGE if set -if [ -z "${BASE_IMAGE:-}" ]; then - # Search for qcow2 file in /opt/nxos/ - if [ -d /opt/nxos ]; then - BASE_IMAGE=$(find /opt/nxos -name "*.qcow2" -type f | head -n1) - fi -fi - -if [ -z "$BASE_IMAGE" ] || [ ! -f "$BASE_IMAGE" ]; then - die "No NXOS image found in /opt/nxos/" -fi - -log "Using NXOS image: $BASE_IMAGE" - -# Load configuration -if [ -f /etc/hotstack-switch-vm/config ]; then - # shellcheck source=/dev/null - source /etc/hotstack-switch-vm/config -fi - -MGMT_INTERFACE="${MGMT_INTERFACE:-eth0}" # VM management (unbridged) -SWITCH_MGMT_INTERFACE="${SWITCH_MGMT_INTERFACE:-eth1}" # Switch management port (bridged) -TRUNK_INTERFACE="${TRUNK_INTERFACE:-eth2}" # Switch trunk port (bridged) -BM_INTERFACE_START="${BM_INTERFACE_START:-eth3}" # Baremetal ports start (bridged) -BM_INTERFACE_COUNT="${BM_INTERFACE_COUNT:-8}" # Number of baremetal ports - -# Ensure work directory exists -mkdir -p "$WORK_DIR" -cd "$WORK_DIR" - -# Libvirt image directory for proper SELinux context and permissions -LIBVIRT_IMAGE_DIR="/var/lib/libvirt/images" -mkdir -p "$LIBVIRT_IMAGE_DIR" - -# Create a working copy of the NXOS image for the VM -NXOS_DISK="$LIBVIRT_IMAGE_DIR/nxos-disk.qcow2" - -if [ ! -f "$NXOS_DISK" ]; then - log "Creating working copy of NXOS image..." - qemu-img create -f qcow2 -F qcow2 -b "$BASE_IMAGE" "$NXOS_DISK" || die "Failed to create backing image" -else - log "Using existing NXOS disk: $NXOS_DISK" -fi - -log "Using NXOS disk: $NXOS_DISK" - -# Build bridge configuration and create bridges -log "Collecting network interface configuration..." -BRIDGES_JSON=$(build_bridge_config "$MGMT_INTERFACE" "$SWITCH_MGMT_INTERFACE" "$TRUNK_INTERFACE" "$BM_INTERFACE_START" "$BM_INTERFACE_COUNT") || die "Failed to build bridge configuration" - -# Create all bridges in one atomic nmstate operation -create_bridges "$BRIDGES_JSON" - -# Calculate number of bridges: 1 switch mgmt + 1 trunk + N baremetal interfaces -NUM_BRIDGES=$((1 + 1 + BM_INTERFACE_COUNT)) - -log "Successfully created $NUM_BRIDGES bridge interfaces (1 switch mgmt + 1 trunk + $BM_INTERFACE_COUNT baremetal)" - -# Render libvirt domain XML from Jinja2 template -log "Rendering libvirt domain XML from template..." - -if ! python3 << EOF -from jinja2 import Template - -# Load template -with open("$SCRIPT_DIR/domain.xml.j2", "r") as f: - template = Template(f.read()) - -# Template variables -context = { - "vm_name": "$VM_NAME", - "nxos_disk": "$NXOS_DISK", - "console_port": "$CONSOLE_PORT", - "num_bridges": $NUM_BRIDGES -} - -# Render and write -with open("$WORK_DIR/domain.xml", "w") as f: - f.write(template.render(**context)) - -print("Domain XML generated successfully") -EOF -then - die "Failed to render domain XML template" -fi - -log "Libvirt domain XML written to: $WORK_DIR/domain.xml" - -# Define and start the VM with libvirt -log "Defining libvirt domain..." -virsh define "$WORK_DIR/domain.xml" || die "Failed to define libvirt domain" - -log "Starting Cisco NXOS VM..." -virsh start "$VM_NAME" || die "Failed to start VM" - -log "Cisco NXOS VM started successfully" -log "Console available at: telnet localhost $CONSOLE_PORT" -log "Or use: virsh console $VM_NAME" -log "Note: NXOS switch will use POAP for automatic configuration" -log " POAP files (poap.py, poap.cfg) should be available via TFTP/HTTP" - -exit 0 diff --git a/scenarios/sno-2nics-force10-10/README.md b/scenarios/sno-2nics-force10-10/README.md index f4454f51..0198cd2f 100644 --- a/scenarios/sno-2nics-force10-10/README.md +++ b/scenarios/sno-2nics-force10-10/README.md @@ -77,7 +77,7 @@ The scenario includes Tempest tests for baremetal instance lifecycle: The scenario uses the `hotstack-switch-host` image which runs a nested Force10 OS10 switch VM inside a CentOS 9 Stream host with KVM/libvirt. -See [images/README.md](../../images/README.md) and [switch-host-scripts/README.md](../../images/switch-host-scripts/README.md) for details on building the switch host image and how the nested switch setup works. +See [switch-images/README.md](../../switch-images/README.md) and [runtime-scripts/README.md](../../switch-images/runtime-scripts/README.md) for details on building the switch host image and how the nested switch setup works. ## Switch Access @@ -97,5 +97,5 @@ Force10 OS10 port naming convention: ## References -- [Switch Host Scripts README](../../images/switch-host-scripts/README.md) -- [Building Switch Host Images](../../images/README.md) +- [Switch Runtime Scripts README](../../switch-images/runtime-scripts/README.md) +- [Building Switch Host Images](../../switch-images/README.md) diff --git a/scenarios/sno-nxsw-netconf/README.md b/scenarios/sno-nxsw-netconf/README.md new file mode 100644 index 00000000..0ecf121f --- /dev/null +++ b/scenarios/sno-nxsw-netconf/README.md @@ -0,0 +1,277 @@ +# SNO-NXSW-NETCONF Scenario + +## Overview + +The `sno-nxsw-netconf` scenario is a Single Node OpenShift (SNO) deployment scenario +for HotStack that deploys OpenStack on OpenShift with ironic bare metal +provisioning capabilities and network switch integration using networking-baremetal's +netconf-openconfig ML2 driver. + +## Architecture + +This scenario provisions: + +- **1x Controller Node**: Management and DNS/DHCP services +- **1x OpenShift Master Node**: Single node OpenShift cluster running OpenStack services +- **1x Switch Node**: NXSW switch with trunk ports for tenant VLAN networks +- **2x Ironic Nodes**: Virtual bare metal nodes for testing Ironic provisioning workflows + +## Features + +- **Complete OpenStack Stack**: Full OpenStack deployment with ironic bare + metal service +- **Network Switch Integration**: Automated switch configuration with + POAP (Power-On Auto Provisioning) and networking-baremetal netconf-openconfig driver +- **NETCONF/OpenConfig**: Uses standard NETCONF protocol and vendor-neutral + OpenConfig YANG models for switch configuration +- **Complete Networking**: All OpenStack service networks with dedicated + ironic networks +- **SNO Deployment**: Single node OpenShift optimized for OpenStack services +- **Development Ready**: Ideal for testing and development environments +- **Bare Metal Provisioning**: Ironic service with 2 nodes for testing bare + metal workflows +- **Ironic Neutron Agent**: Includes ironic-neutron-agent for handling port + binding notifications from Ironic to Neutron + +## Networks + +- **machine-net**: 192.168.32.0/24 - External access network +- **ctlplane-net**: 192.168.122.0/24 - Control plane network +- **internal-api-net**: 172.17.0.0/24 - OpenStack internal API network +- **storage-net**: 172.18.0.0/24 - Storage network +- **tenant-net**: 172.19.0.0/24 - Tenant network for OpenStack workloads +- **ironic-net**: 172.20.1.0/24 - Ironic network for bare metal provisioning +- **tenant-vlan103**: 172.20.3.0/24 - Tenant VLAN network (VLAN 103) +- **tenant-vlan104**: 172.20.4.0/24 - Tenant VLAN network (VLAN 104) +- **ironic0-br-net**: 172.20.5.0/29 - Ironic0 bridge network +- **ironic1-br-net**: 172.20.5.8/29 - Ironic1 bridge network + +## Switch Instance Configuration (Nested Virtualization) + +This scenario uses a **nested virtualization** approach where: +1. A Linux host VM (`hotstack-switch-host` image) runs on OpenStack +2. Inside that VM, the NXOS switch runs as a nested KVM guest +3. Network interfaces are bridged between the host VM and the nested switch + +### Host VM Network Interfaces + +```text +Host VM (hotstack-switch-host): +├── eth0: machine-net (192.168.32.6) - Host VM management (SSH access) +├── eth1: machine-net (192.168.32.7, MAC: 22:57:f8:dd:fe:08) - Nested switch mgmt +├── eth2: trunk (MAC: 22:57:f8:dd:fe:09) - Trunk port with VLANs +├── eth3: ironic0-br-net (MAC: 22:57:f8:dd:fe:0c) - Baremetal port 0 +└── eth4: ironic1-br-net (MAC: 22:57:f8:dd:fe:0d) - Baremetal port 1 +``` + +### Nested NXOS Switch Interfaces (Direct Passthrough Mode) + +```text +Nested NXOS Switch (exclusive interface access): +├── eth0: Passthrough host eth1 (inherits MAC: 22:57:f8:dd:fe:08) → Mgmt IP via POAP +├── eth1: Passthrough host eth2 (inherits MAC: 22:57:f8:dd:fe:09) → Trunk (VLANs 101-104) +├── eth2: Passthrough host eth3 (inherits MAC: 22:57:f8:dd:fe:0c) → Access port to ironic0 +└── eth3: Passthrough host eth4 (inherits MAC: 22:57:f8:dd:fe:0d) → Access port to ironic1 +``` + +**Key Feature**: The nested switch uses **direct passthrough mode** (`mode='passthrough'`) +where the VM gets **exclusive access** to the host interfaces. This enables: +- **No MAC conflicts**: Eliminates bridge MAC address collision warnings +- Transparent DHCP: OpenStack's DHCP server responds to the host interface MACs +- POAP to work correctly with DHCP options from the controller +- Best performance (direct hardware-level access) + +**Trade-off**: The host VM cannot access these interfaces while the nested switch is running, +but this is acceptable since the switch needs exclusive control anyway. + +The nested switch boots and is automatically configured via POAP (Power-On Auto Provisioning). + +### VLAN Mapping + +- **VLAN 101**: ironic (172.20.1.0/24) +- **VLAN 102**: Default native VLAN +- **VLAN 103**: tenant-vlan103 (172.20.3.0/24) +- **VLAN 104**: tenant-vlan104 (172.20.4.0/24) + +The switch uses the `nxsw` image and provides dual trunk ports for redundancy +and high availability. + +### POAP (Power-On Auto Provisioning) + +POAP is a Cisco NX-OS feature that automates the initial configuration of +network switches. When the switch boots up, it automatically: + +1. **Downloads Configuration**: Fetches the switch configuration from a + TFTP/HTTP server +2. **Applies Settings**: Automatically configures interfaces, VLANs, and + network settings +3. **Enables Services**: Activates required network services (NETCONF, LACP, LLDP) +4. **Validates Setup**: Performs integrity checks using MD5 checksums + +In this scenario, POAP enables zero-touch deployment of the NX-OS switch with pre-configured: + +- **Interface Configuration**: Trunk and access ports for tenant VLANs +- **VLAN Setup**: VLANs for network segmentation +- **Management Settings**: IP addressing, DNS, and routing configuration +- **Security**: User accounts and access control + +## Ironic Nodes + +The scenario includes 2 virtual bare metal nodes for testing Ironic provisioning: + +### Ironic Node 0 + +- **Network**: ironic0-br-net (172.20.5.0/29) +- **Purpose**: Bare metal provisioning testing +- **Configuration**: Virtual media boot capable with sushy-tools + +### Ironic Node 1 + +- **Network**: ironic1-br-net (172.20.5.8/29) +- **Purpose**: Bare metal provisioning testing +- **Configuration**: Virtual media boot capable with sushy-tools + +## Networking-Baremetal Integration + +This scenario uses the `networking-baremetal` ML2 mechanism driver with the +`netconf-openconfig` device driver. This provides: + +### NETCONF/OpenConfig Driver Features + +- **Standards-Based**: Uses NETCONF protocol (RFC 6241) and OpenConfig YANG models +- **Vendor Support**: Tested with Cisco NXOS and Arista EOS switches +- **LACP Support**: Can manage Link Aggregation Control Protocol (LACP) port channels +- **VLAN Management**: Automatic VLAN creation and port configuration +- **Port MTU**: Supports configuring MTU on switch ports +- **SSH Key Authentication**: Supports password and SSH key authentication + +### Configuration + +The switch configuration is defined in `manifests/networking-baremetal/config.yaml`: + +- **Driver**: `netconf-openconfig` - Uses NETCONF with OpenConfig YANG models +- **Device Parameters**: `name:nexus` - ncclient device handler for Cisco NXOS +- **Switch ID**: MAC address of the switch for identification +- **Physical Networks**: Maps OpenStack physical networks to the device +- **LACP Management**: Configures automatic management of LACP aggregates +- **Port ID Substitution**: Converts LLDP port names to NETCONF port format + +### Ironic Neutron Agent + +The `ironic-neutron-agent` service handles communication between Ironic and Neutron: + +- Listens for port binding notifications from Neutron +- Triggers switch port configuration when Ironic nodes are provisioned +- Manages VLAN assignment and port activation + +## Usage + +This scenario is ideal for: + +- Testing OpenStack deployments with networking-baremetal ML2 driver +- Validating bare metal provisioning workflows with Ironic +- Network switch integration testing with NETCONF/OpenConfig +- Development and testing of networking-baremetal functionality +- Evaluating vendor-neutral network automation with OpenConfig + +## Files + +- `bootstrap_vars.yml`: Main configuration variables +- `heat_template.yaml`: OpenStack Heat template for infrastructure +- `automation-vars.yml`: Automation pipeline definition +- `manifests/`: OpenShift/Kubernetes manifests +- `test-operator/`: Test automation configuration + +## Switch Host Image + +This scenario uses the `hotstack-switch-host` image, which is a CentOS 9 Stream +image with KVM/libvirt and scripts to run nested switch VMs. + +### Building the Switch Host Image + +The switch host image must be built with the NXOS disk image embedded: + +1. Build the switch-host image with NXOS: + ```bash + cd images/ + make switch-host NXOS_IMAGE=/path/to/nexus9300v64.10.5.3.F.qcow2 + ``` + + This will: + - Download CentOS 9 Stream base image + - Download GNS3 "switch friendly" UEFI firmware (OVMF-edk2-stable202305.fd) + - Install libvirt, qemu-kvm, and switch management scripts + - Copy NXOS image to `/opt/nxos/` inside the image + - Copy firmware to `/usr/local/share/edk2/ovmf/` for better NXOS NIC compatibility + +3. Upload to OpenStack: + ```bash + openstack image create hotstack-switch-host \ + --disk-format qcow2 \ + --file switch-host-nxos.qcow2 \ + --public \ + --property hw_disk_bus=scsi \ + --property hw_vif_model=virtio \ + --property hw_video_model=qxl + ``` + +### NXOS Version Requirements + +**Important:** For networking-baremetal netconf-openconfig support, you need: +- **NXOS 10.2 or later** - Required for OpenConfig YANG model support +- NXOS 9.x versions do NOT have adequate OpenConfig support + +See [switch-images/README.md](../../switch-images/README.md) and +[runtime-scripts/README.md](../../switch-images/runtime-scripts/README.md) +for detailed instructions. + +## Switch NETCONF Configuration + +NETCONF is automatically enabled on the NXOS switch via the POAP (Power-On Auto +Provisioning) configuration file (`poap.cfg`). The following features are enabled: + +- `feature netconf` - Enables NETCONF protocol on port 830 +- `feature lacp` - Enables Link Aggregation Control Protocol + +After the switch boots and applies the POAP configuration, you can verify NETCONF +is running: + +```bash +switch# show netconf status +``` + +## Container Image Requirements + +The standard OpenStack operator images include the required networking-baremetal +components: + +- `networking-baremetal` - ML2 mechanism driver and ironic-neutron-agent +- `ncclient` - Python NETCONF client library +- `pyangbind` - Python bindings for YANG models +- OpenConfig Python bindings - For OpenConfig YANG models + +## Deployment + +Follow the standard HotStack deployment process with this scenario by setting +the scenario name to `sno-nxsw-netconf` in your deployment configuration. + +### Prerequisites + +1. **Switch Host Image**: The `hotstack-switch-host` image with NXOS 10.2+ + embedded must be available in your OpenStack cloud +2. **Nested Virtualization**: The OpenStack compute nodes must support nested + virtualization (Intel VT-x/AMD-V with nested EPT/RVI enabled) +3. **Sufficient Resources**: The switch host requires a `hotstack.xlarge` flavor + or larger to run the nested NXOS VM + +### How It Works + +1. Heat creates the switch-host VM with multiple network ports +2. Cloud-init configures the host and writes `/etc/hotstack-switch-vm/config` +3. The `start-switch-vm.sh` script: + - Creates Linux bridges for each network interface + - Starts the nested NXOS VM using libvirt/KVM + - Bridges host interfaces to the nested switch VM +4. The NXOS switch boots and uses POAP to fetch its configuration from the + controller +5. After POAP completes, the switch is ready for netconf-openconfig connections diff --git a/scenarios/sno-nxsw-netconf/automation-vars.yml b/scenarios/sno-nxsw-netconf/automation-vars.yml new file mode 100644 index 00000000..d51ecf5f --- /dev/null +++ b/scenarios/sno-nxsw-netconf/automation-vars.yml @@ -0,0 +1,129 @@ +--- +stages: + - name: TopoLVM Dependencies + stages: >- + {{ + lookup("ansible.builtin.template", + "common/stages/topolvm-deps-stages.yaml.j2") + }} + + - name: Dependencies + stages: >- + {{ + lookup("ansible.builtin.template", + "common/stages/deps-stages.yaml.j2") + }} + + - name: Cinder LVM + stages: >- + {{ + lookup("ansible.builtin.file", + "common/stages/cinder-lvm-label-stages.yaml") + }} + + - name: TopoLVM + stages: >- + {{ + lookup("ansible.builtin.template", + "common/stages/topolvm-stages.yaml.j2") + }} + + - name: OLM Openstack + stages: >- + {{ + lookup("ansible.builtin.template", + "common/stages/olm-openstack-stages.yaml.j2") + }} + + - name: NodeNetworkConfigurationPolicy (nncp) + manifest: manifests/networking/nncp.yaml + wait_conditions: + - >- + oc wait -n openstack nncp -l osp/nncm-config-type=standard + --for jsonpath='{.status.conditions[0].reason}'=SuccessfullyConfigured + --timeout=180s + + - name: NetworkAttchmentDefinition (NAD) + manifest: manifests/networking/nad.yaml + + - name: MetalLB - L2Advertisement and IPAddressPool + manifest: manifests/networking/metallb.yaml + + - name: Netconfig + manifest: manifests/networking/netconfig.yaml + + - name: Networking Baremetal config (netconf-openconfig) + manifest: manifests/networking-baremetal/config.yaml + wait_conditions: + - >- + oc wait -n openstack secret neutron-switch-config + --for jsonpath='{.metadata.name}'=neutron-switch-config + --timeout=30s + + - name: OpenstackControlPlane + manifest: manifests/control-plane.yaml + wait_conditions: + - >- + oc -n openstack wait openstackcontrolplanes.core.openstack.org controlplane + --for condition=OpenStackControlPlaneDNSReadyCondition --timeout=600s + + - name: Extra DNS LoadBalancer on Ironic network + manifest: manifests/dnsmasq-dns-ironic.yaml + wait_conditions: + - >- + oc wait -n openstack service dnsmasq-dns-ironic + --for jsonpath='.status.loadBalancer' --timeout=60s + + - name: Wait for OpenstackControlPlane + wait_conditions: + - >- + oc wait -n openstack openstackcontrolplane controlplane + --for condition=Ready --timeout=30m + + - name: Update openstack-operators OLM + stages: >- + {{ + lookup('ansible.builtin.template', + 'common/stages/openstack-olm-update.yaml.j2') + }} + run_conditions: + - >- + {{ + openstack_operators_update is defined and + openstack_operators_update | bool + }} + + - name: Wait for condition MinorUpdateAvailable True + wait_conditions: + - >- + oc -n openstack wait openstackversions.core.openstack.org controlplane + --for=condition=MinorUpdateAvailable=True --timeout=10m + run_conditions: + - "{{ openstack_update is defined and openstack_update | bool }}" + + - name: "Minor update :: Create OpenStackVersion patch" + documentation: | + This creates a patch file `{{ manifests_dir }}/patches/openstack_version_patch.yaml` + If `openstack_update_custom_images` is defined it will populate the customContainerImages + in the OpenstackVersion YAML patch. + shell: >- + {{ + lookup('ansible.builtin.template', + 'common/scripts/create_openstack_version_patch.sh.j2') + }} + run_conditions: + - "{{ openstack_update is defined and openstack_update | bool }}" + + - name: "Minor update :: Update the target version in the OpenStackVersion custom resource (CR)" + documentation: | + The `hotstack-openstack-version-patch` script will get the `availableVersion` + and us it to replace the string `__TARGET_VERSION__` in the patch file and + apply the patch using `oc patch` command. + command: >- + hotstack-openstack-version-patch --namespace openstack --name controlplane + --file {{ manifests_dir }}/patches/openstack_version_patch.yaml + wait_conditions: + - oc -n openstack wait openstackversions.core.openstack.org controlplane + --for=condition=Ready --timeout=10m + run_conditions: + - "{{ openstack_update is defined and openstack_update | bool }}" diff --git a/scenarios/sno-nxsw-netconf/bootstrap_vars.yml b/scenarios/sno-nxsw-netconf/bootstrap_vars.yml new file mode 100644 index 00000000..fab8dffc --- /dev/null +++ b/scenarios/sno-nxsw-netconf/bootstrap_vars.yml @@ -0,0 +1,59 @@ +--- +os_cloud: default +os_floating_network: public +os_router_external_network: public + +scenario: sno-nxsw-netconf +scenario_dir: scenarios +stack_template_path: "{{ scenario_dir }}/{{ scenario }}/heat_template.yaml" +automation_vars_file: "{{ scenario_dir }}/{{ scenario }}/automation-vars.yml" +test_operator_automation_vars_file: "{{ scenario_dir }}/{{ scenario }}/test-operator/automation-vars.yml" + +openstack_operators_image: quay.io/openstack-k8s-operators/openstack-operator-index:latest +openstack_operator_channel: alpha +openstack_operator_starting_csv: null + +openshift_version: stable-4.18 + +ntp_servers: [] +dns_servers: + - 8.8.8.8 + - 8.8.4.4 + +pull_secret_file: ~/pull-secret.txt + +ovn_k8s_gateway_config_host_routing: true +enable_iscsi: true +enable_multipath: true + +cinder_volume_pvs: + - /dev/vdc + - /dev/vdd + - /dev/vde + +stack_name: "hs-{{ scenario }}-{{ zuul.build[:8] | default('no-zuul') }}" +stack_parameters: + # On misconfigured clouds, uncomment these to avoid issues. + # Ref: https://access.redhat.com/solutions/7059376 + # net_value_specs: + # mtu: 1442 + dns_servers: "{{ dns_servers }}" + ntp_servers: "{{ ntp_servers }}" + controller_ssh_pub_key: "{{ controller_ssh_pub_key | default('') }}" + router_external_network: "{{ os_router_external_network | default('public') }}" + floating_ip_network: "{{ os_floating_network | default('public') }}" + controller_params: + image: hotstack-controller + flavor: hotstack.small + ocp_master_params: + image: ipxe-boot-usb + flavor: hotstack.xxlarge + switch_params: + image: hotstack-switch-host + flavor: hotstack.xlarge + # Nested NXOS image should be placed in /opt/nxos/ in the switch-host image + # or built into the image. Use NXOS 10.2 or later for OpenConfig support + ironic_params: + image: CentOS-Stream-GenericCloud-9 + cd_image: sushy-tools-blank-image + flavor: hotstack.medium diff --git a/scenarios/sno-nxsw-netconf/heat_template.yaml b/scenarios/sno-nxsw-netconf/heat_template.yaml new file mode 100644 index 00000000..b5608592 --- /dev/null +++ b/scenarios/sno-nxsw-netconf/heat_template.yaml @@ -0,0 +1,1071 @@ +--- +heat_template_version: rocky + +description: > + Heat template to set up SNO (Single Node OpenShift) infrastructure with Cisco NXOS switch (nested VM using networking-baremetal netconf-openconfig) + +parameters: + dns_servers: + type: comma_delimited_list + default: + - 8.8.8.8 + - 8.8.4.4 + ntp_servers: + type: comma_delimited_list + default: [] + controller_ssh_pub_key: + type: string + dataplane_ssh_pub_key: + type: string + router_external_network: + type: string + default: public + floating_ip_network: + type: string + default: public + net_value_specs: + type: json + default: {} + + controller_params: + type: json + default: + image: hotstack-controller + flavor: hotstack.small + nat64_appliance_params: + type: json + default: + image: nat64-appliance + flavor: hotstack.small + ocp_master_params: + type: json + default: + image: ipxe-boot-usb + flavor: hotstack.xxlarge + ocp_worker_params: + type: json + default: + image: ipxe-boot-usb + flavor: hotstack.xxlarge + compute_params: + type: json + default: + image: CentOS-Stream-GenericCloud-9 + flavor: hotstack.large + networker_params: + type: json + default: + image: CentOS-Stream-GenericCloud-9 + flavor: hotstack.small + bmh_params: + type: json + default: + image: CentOS-Stream-GenericCloud-9 + cd_image: sushy-tools-blank-image + flavor: hotstack.medium + ironic_params: + type: json + default: + image: CentOS-Stream-GenericCloud-9 + cd_image: sushy-tools-blank-image + flavor: hotstack.medium + switch_params: + type: json + default: + image: hotstack-switch-host + flavor: hotstack.xlarge + cdrom_disk_bus: + type: string + description: > + Disk bus type for CDROM device. 'sata' may be required for older versions + of OpenStack. Heat patch https://review.opendev.org/c/openstack/heat/+/966688 + is needed for 'sata' support. + default: scsi + constraints: + - allowed_values: + - sata + - scsi + +resources: + # + # Networks + # + machine-net: + type: OS::Neutron::Net + properties: + port_security_enabled: false + value_specs: {get_param: net_value_specs} + + ctlplane-net: + type: OS::Neutron::Net + properties: + port_security_enabled: false + value_specs: {get_param: net_value_specs} + + internal-api-net: + type: OS::Neutron::Net + properties: + port_security_enabled: false + value_specs: {get_param: net_value_specs} + + storage-net: + type: OS::Neutron::Net + properties: + port_security_enabled: false + value_specs: {get_param: net_value_specs} + + tenant-net: + type: OS::Neutron::Net + properties: + port_security_enabled: false + value_specs: {get_param: net_value_specs} + + ironic-net: + type: OS::Neutron::Net + properties: + port_security_enabled: false + value_specs: {get_param: net_value_specs} + + trunk-net: + type: OS::Neutron::Net + properties: + port_security_enabled: false + value_specs: {get_param: net_value_specs} + + tenant-vlan103: + type: OS::Neutron::Net + properties: + port_security_enabled: false + value_specs: {get_param: net_value_specs} + + tenant-vlan104: + type: OS::Neutron::Net + properties: + port_security_enabled: false + value_specs: {get_param: net_value_specs} + + ironic0-br-net: + type: OS::Neutron::Net + properties: + port_security_enabled: false + value_specs: {get_param: net_value_specs} + + ironic1-br-net: + type: OS::Neutron::Net + properties: + port_security_enabled: false + value_specs: {get_param: net_value_specs} + + + # + # Subnets + # + machine-subnet: + type: OS::Neutron::Subnet + properties: + network: {get_resource: machine-net} + ip_version: 4 + cidr: 192.168.32.0/24 + enable_dhcp: true + dns_nameservers: + - 192.168.32.254 + + ctlplane-subnet: + type: OS::Neutron::Subnet + properties: + network: {get_resource: ctlplane-net} + ip_version: 4 + cidr: 192.168.122.0/24 + enable_dhcp: false + allocation_pools: + - start: 192.168.122.100 + end: 192.168.122.150 + dns_nameservers: + - 192.168.122.80 + + internal-api-subnet: + type: OS::Neutron::Subnet + properties: + network: {get_resource: internal-api-net} + ip_version: 4 + cidr: 172.17.0.0/24 + enable_dhcp: false + allocation_pools: + - start: 172.17.0.100 + end: 172.17.0.150 + + storage-subnet: + type: OS::Neutron::Subnet + properties: + network: {get_resource: storage-net} + ip_version: 4 + cidr: 172.18.0.0/24 + enable_dhcp: false + allocation_pools: + - start: 172.18.0.100 + end: 172.18.0.150 + + tenant-subnet: + type: OS::Neutron::Subnet + properties: + network: {get_resource: tenant-net} + ip_version: 4 + cidr: 172.19.0.0/24 + enable_dhcp: false + allocation_pools: + - start: 172.19.0.100 + end: 172.19.0.150 + + ironic-subnet: + type: OS::Neutron::Subnet + properties: + network: {get_resource: ironic-net} + ip_version: 4 + cidr: 172.20.1.0/24 + enable_dhcp: false + allocation_pools: [{start: 172.20.1.100, end: 172.20.1.150}] + + trunk-subnet: + type: OS::Neutron::Subnet + properties: + network: {get_resource: trunk-net} + ip_version: 4 + cidr: 172.20.2.0/24 + enable_dhcp: false + allocation_pools: + - start: 172.20.2.100 + end: 172.20.2.150 + + tenant-vlan103-subnet: + type: OS::Neutron::Subnet + properties: + network: {get_resource: tenant-vlan103} + ip_version: 4 + cidr: 172.20.3.0/24 + enable_dhcp: false + allocation_pools: + - start: 172.20.3.100 + end: 172.20.3.150 + + tenant-vlan104-subnet: + type: OS::Neutron::Subnet + properties: + network: {get_resource: tenant-vlan104} + ip_version: 4 + cidr: 172.20.4.0/24 + enable_dhcp: false + allocation_pools: + - start: 172.20.4.100 + end: 172.20.4.150 + + ironic0-br-subnet: + type: OS::Neutron::Subnet + properties: + network: {get_resource: ironic0-br-net} + ip_version: 4 + cidr: 172.20.5.0/29 + enable_dhcp: false + gateway_ip: null + + ironic1-br-subnet: + type: OS::Neutron::Subnet + properties: + network: {get_resource: ironic1-br-net} + ip_version: 4 + cidr: 172.20.5.8/29 + enable_dhcp: false + gateway_ip: null + + # + # Routers + # + router: + type: OS::Neutron::Router + properties: + admin_state_up: true + external_gateway_info: + network: {get_param: router_external_network} + # enable_snat: true + + machine-net-router-interface: + type: OS::Neutron::RouterInterface + properties: + router: {get_resource: router} + subnet: {get_resource: machine-subnet} + + ctlplane-net-router-interface: + type: OS::Neutron::RouterInterface + properties: + router: {get_resource: router} + subnet: {get_resource: ctlplane-subnet} + + ironic-net-router-interface: + type: OS::Neutron::RouterInterface + properties: + router: {get_resource: router} + subnet: {get_resource: ironic-subnet} + + tenant-vlan103-router-interface: + type: OS::Neutron::RouterInterface + properties: + router: {get_resource: router} + subnet: {get_resource: tenant-vlan103-subnet} + + tenant-vlan104-router-interface: + type: OS::Neutron::RouterInterface + properties: + router: {get_resource: router} + subnet: {get_resource: tenant-vlan104-subnet} + + # + # Instances + # + controller_users: + type: OS::Heat::CloudConfig + properties: + cloud_config: + users: + - default + - name: zuul + gecos: "Zuul user" + sudo: ALL=(ALL) NOPASSWD:ALL + ssh_authorized_keys: + - {get_param: controller_ssh_pub_key} + + controller-write-files: + type: OS::Heat::CloudConfig + properties: + cloud_config: + write_files: + - path: /etc/dnsmasq.conf + content: | + # dnsmasq service config + # Include all files in /etc/dnsmasq.d except RPM backup files + conf-dir=/etc/dnsmasq.d,.rpmnew,.rpmsave,.rpmorig + no-resolv + owner: root:dnsmasq + - path: /etc/dnsmasq.d/forwarders.conf + content: + str_replace: + template: | + # DNS forwarders records + server=$dns1 + server=$dns2 + params: + $dns1: {get_param: [dns_servers, 0]} + $dns2: {get_param: [dns_servers, 1]} + owner: root:dnsmasq + - path: /etc/dnsmasq.d/host_records.conf + content: + str_replace: + template: | + # Host records + host-record=controller-0.openstack.lab,$controller0 + host-record=nxos.openstack.lab,$nxos0 + host-record=api.sno.openstack.lab,$master0 + host-record=api-int.sno.openstack.lab,$master0 + host-record=master-0.sno.openstack.lab,$master0 + params: + $controller0: {get_attr: [controller-machine-port, fixed_ips, 0, ip_address]} + $nxos0: {get_attr: [switch-machine-port, fixed_ips, 0, ip_address]} + $master0: {get_attr: [master0-machine-port, fixed_ips, 0, ip_address]} + owner: root:dnsmasq + - path: /etc/dnsmasq.d/wildcard_records.conf + content: + str_replace: + template: | + # Wildcard records + address=/apps.sno.openstack.lab/$addr + params: + $addr: {get_attr: [controller-machine-port, fixed_ips, 0, ip_address]} + owner: root:dnsmasq + - path: /etc/resolv.conf + content: | + nameserver: 127.0.0.1 + owner: root:root + - path: /etc/NetworkManager/conf.d/98-rc-manager.conf + content: | + [main] + rc-manager=unmanaged + owner: root:root + - path: /etc/haproxy/haproxy.cfg + content: | + global + log 127.0.0.1 local2 + pidfile /var/run/haproxy.pid + maxconn 4000 + daemon + defaults + mode http + log global + option dontlognull + option http-server-close + option redispatch + retries 3 + timeout http-request 10s + timeout queue 1m + timeout connect 10s + timeout client 1m + timeout server 1m + timeout http-keep-alive 10s + timeout check 10s + maxconn 3000 + listen api-server-6443 + bind *:6443 + mode tcp + server master-0 master-0.sno.openstack.lab:6443 check inter 1s + listen machine-config-server-22623 + bind *:22623 + mode tcp + server master-0 master-0.sno.openstack.lab:22623 check inter 1s + listen ingress-router-443 + bind *:443 + mode tcp + balance source + server master-0 master-0.sno.openstack.lab:443 check inter 1s + listen ingress-router-80 + bind *:80 + mode tcp + balance source + server master-0 master-0.sno.openstack.lab:80 check inter 1s + owner: root:root + - path: /etc/dnsmasq.d/tftpboot.conf + content: | + enable-tftp + tftp-root=/var/lib/tftpboot + tftp-mtu=1442 + owner: root:root + - path: /var/lib/tftpboot/poap.py + content: {get_file: poap.py} + owner: root:root + - path: /var/lib/tftpboot/poap.cfg + content: {get_file: poap.cfg} + owner: root:root + + controller-runcmd: + type: OS::Heat::CloudConfig + properties: + cloud_config: + runcmd: + - ['setenforce', 'permissive'] + - ['systemctl', 'enable', 'haproxy.service'] + - ['systemctl', 'start', 'haproxy.service'] + - ['sed', '-i', 's/Listen 80/Listen 8081/g', '/etc/httpd/conf/httpd.conf'] + - ['systemctl', 'enable', 'httpd.service'] + - ['systemctl', 'start', 'httpd.service'] + - ['systemctl', 'enable', 'dnsmasq.service'] + - ['systemctl', 'start', 'dnsmasq.service'] + + controller-init: + type: OS::Heat::MultipartMime + properties: + parts: + - config: {get_resource: controller_users} + - config: {get_resource: controller-write-files} + - config: {get_resource: controller-runcmd} + + controller-machine-port: + type: OS::Neutron::Port + properties: + network: {get_resource: machine-net} + mac_address: "fa:16:9e:81:f6:05" + fixed_ips: + - ip_address: 192.168.32.254 + + controller-floating-ip: + depends_on: machine-net-router-interface + type: OS::Neutron::FloatingIP + properties: + floating_network: {get_param: floating_ip_network} + port_id: {get_resource: controller-machine-port} + + controller: + type: OS::Nova::Server + properties: + image: {get_param: [controller_params, image]} + flavor: {get_param: [controller_params, flavor]} + networks: + - port: {get_resource: controller-machine-port} + user_data_format: RAW + user_data: {get_resource: controller-init} + + # OCP Masters + + # DHCP Opts value + extra-dhcp-opts-value: + type: OS::Heat::Value + properties: + type: json + value: + extra_dhcp_opts: + - opt_name: "60" + opt_value: "HTTPClient" + ip_version: 4 + - opt_name: "67" + opt_value: + str_replace: + template: http://$server_address:8081/boot-artifacts/agent.x86_64.ipxe + params: + $server_address: {get_attr: [controller-machine-port, fixed_ips, 0, ip_address]} + + master0-machine-port: + type: OS::Neutron::Port + properties: + network: {get_resource: machine-net} + port_security_enabled: false + mac_address: "fa:16:9e:81:f6:10" + fixed_ips: + - ip_address: 192.168.32.10 + value_specs: {get_attr: [extra-dhcp-opts-value, value]} + + master0-ctlplane-trunk-parent-port: + type: OS::Neutron::Port + properties: + network: {get_resource: ctlplane-net} + port_security_enabled: false + fixed_ips: + - ip_address: 192.168.122.10 + + master0-internal-api-port: + type: OS::Neutron::Port + properties: + network: {get_resource: internal-api-net} + port_security_enabled: false + fixed_ips: + - ip_address: 172.17.0.10 + + master0-storage-port: + type: OS::Neutron::Port + properties: + network: {get_resource: storage-net} + port_security_enabled: false + fixed_ips: + - ip_address: 172.18.0.10 + + master0-tenant-port: + type: OS::Neutron::Port + properties: + network: {get_resource: tenant-net} + port_security_enabled: false + fixed_ips: + - ip_address: 172.19.0.10 + + master0-trunk0: + type: OS::Neutron::Trunk + properties: + port: {get_resource: master0-ctlplane-trunk-parent-port} + sub_ports: + - port: {get_resource: master0-internal-api-port} + segmentation_id: 20 + segmentation_type: vlan + - port: {get_resource: master0-storage-port} + segmentation_id: 21 + segmentation_type: vlan + - port: {get_resource: master0-tenant-port} + segmentation_id: 22 + segmentation_type: vlan + + master0-ironic-trunk-parent-port: + type: OS::Neutron::Port + properties: + network: {get_resource: trunk-net} + port_security_enabled: false + fixed_ips: + - ip_address: 172.20.2.10 + + master0-ironic-subport: + type: OS::Neutron::Port + properties: + network: {get_resource: ironic-net} + port_security_enabled: false + fixed_ips: + - ip_address: 172.20.1.10 + + master0-tenant-vlan103-subport: + type: OS::Neutron::Port + properties: + network: {get_resource: tenant-vlan103} + port_security_enabled: false + fixed_ips: + - ip_address: 172.20.3.10 + + master0-tenant-vlan104-subport: + type: OS::Neutron::Port + properties: + network: {get_resource: tenant-vlan104} + port_security_enabled: false + fixed_ips: + - ip_address: 172.20.4.10 + + master0-ironic-trunk: + type: OS::Neutron::Trunk + properties: + port: {get_resource: master0-ironic-trunk-parent-port} + sub_ports: + - port: {get_resource: master0-ironic-subport} + segmentation_id: 101 + segmentation_type: vlan + - port: {get_resource: master0-tenant-vlan103-subport} + segmentation_id: 103 + segmentation_type: vlan + - port: {get_resource: master0-tenant-vlan104-subport} + segmentation_id: 104 + segmentation_type: vlan + + master0-lvms-vol0: + type: OS::Cinder::Volume + properties: + size: 20 + + master0-cinder-vol0: + type: OS::Cinder::Volume + properties: + size: 20 + + master0-cinder-vol1: + type: OS::Cinder::Volume + properties: + size: 20 + + master0-cinder-vol2: + type: OS::Cinder::Volume + properties: + size: 20 + + master0: + type: OS::Nova::Server + properties: + image: {get_param: [ocp_master_params, image]} + flavor: {get_param: [ocp_master_params, flavor]} + block_device_mapping_v2: + - boot_index: -1 + device_type: disk + volume_id: {get_resource: master0-lvms-vol0} + - boot_index: -1 + device_type: disk + volume_id: {get_resource: master0-cinder-vol0} + - boot_index: -1 + device_type: disk + volume_id: {get_resource: master0-cinder-vol1} + - boot_index: -1 + device_type: disk + volume_id: {get_resource: master0-cinder-vol2} + networks: + - port: {get_resource: master0-machine-port} + - port: {get_attr: [master0-trunk0, port_id]} + - port: {get_attr: [master0-ironic-trunk, port_id]} + + # + # Switch + # + + switch-extra-dhcp-opts-value: + type: OS::Heat::Value + properties: + type: json + value: + extra_dhcp_opts: + - opt_name: "66" + opt_value: + str_replace: + template: "$server_address" + params: + $server_address: {get_attr: [controller-machine-port, fixed_ips, 0, ip_address]} + ip_version: 4 + - opt_name: "67" + opt_value: "poap.py" + ip_version: 4 + + switch-users: + type: OS::Heat::CloudConfig + properties: + cloud_config: + users: + - default + - name: zuul + gecos: "Zuul user" + sudo: ALL=(ALL) NOPASSWD:ALL + ssh_authorized_keys: + - {get_param: controller_ssh_pub_key} + - {get_param: dataplane_ssh_pub_key} + + switch-write-files: + type: OS::Heat::CloudConfig + properties: + cloud_config: + write_files: + - path: /etc/hotstack-switch-vm/config + content: + str_replace: + template: | + # Switch VM configuration + SWITCH_MODEL=nxos + + # Network interfaces (from OpenStack) + MGMT_INTERFACE=eth0 # VM management (unbridged, for SSH) + SWITCH_MGMT_INTERFACE=eth1 # Switch management port (bridge mode) + TRUNK_INTERFACE=eth2 # Switch trunk port (passthrough mode) + BM_INTERFACE_START=eth3 # First baremetal interface (passthrough mode) + BM_INTERFACE_COUNT=2 # Number of baremetal interfaces + + # MAC addresses (from OpenStack Neutron ports) + SWITCH_MGMT_MAC=$switch_mgmt_mac + TRUNK_MAC=$trunk_mac + BM_MACS=($bm_mac_0 $bm_mac_1) + + # Switch configuration + SWITCH_MGMT_IP=$switch_mgmt_ip + SWITCH_MGMT_PREFIX=$prefix_len + CONSOLE_PORT=55001 # Telnet console port + PARKING_VLAN=102 # Default/parking VLAN for idle baremetal ports + SWITCH_CMD_DELAY=2 # Delay (seconds) between switch commands + params: + $switch_mgmt_ip: {get_attr: [switch-switch-mgmt-port, fixed_ips, 0, ip_address]} + $switch_mgmt_mac: {get_attr: [switch-switch-mgmt-port, mac_address]} + $trunk_mac: {get_attr: [switch-trunk-parent-port, mac_address]} + $bm_mac_0: {get_attr: [switch-ironic0-br-port, mac_address]} + $bm_mac_1: {get_attr: [switch-ironic1-br-port, mac_address]} + $prefix_len: + str_split: + - '/' + - {get_attr: [machine-subnet, cidr]} + - 1 + + switch-runcmd: + type: OS::Heat::CloudConfig + properties: + cloud_config: + runcmd: + - ['/usr/local/bin/start-switch-vm.sh'] + + switch-init: + type: OS::Heat::MultipartMime + properties: + parts: + - config: {get_resource: switch-users} + - config: {get_resource: switch-write-files} + - config: {get_resource: switch-runcmd} + + switch-machine-port: + type: OS::Neutron::Port + properties: + network: {get_resource: machine-net} + port_security_enabled: false + mac_address: "22:57:f8:dd:fe:aa" + fixed_ips: + - ip_address: 192.168.32.6 + + switch-switch-mgmt-port: + type: OS::Neutron::Port + properties: + network: {get_resource: machine-net} + port_security_enabled: false + mac_address: "22:57:f8:dd:fe:08" + fixed_ips: + - ip_address: 192.168.32.7 + value_specs: {get_attr: [switch-extra-dhcp-opts-value, value]} + + switch-trunk-parent-port: + type: OS::Neutron::Port + properties: + network: {get_resource: trunk-net} + port_security_enabled: false + mac_address: "22:57:f8:dd:fe:09" + + switch-trunk-ironic-port: + type: OS::Neutron::Port + properties: + network: {get_resource: ironic-net} + port_security_enabled: false + mac_address: "22:57:f8:dd:fe:10" + + switch-trunk-tenant-vlan103-port: + type: OS::Neutron::Port + properties: + network: {get_resource: tenant-vlan103} + port_security_enabled: false + mac_address: "22:57:f8:dd:fe:0a" + + switch-trunk-tenant-vlan104-port: + type: OS::Neutron::Port + properties: + network: {get_resource: tenant-vlan104} + port_security_enabled: false + mac_address: "22:57:f8:dd:fe:0b" + + switch-trunk: + type: OS::Neutron::Trunk + properties: + port: {get_resource: switch-trunk-parent-port} + sub_ports: + # Ironic VLAN + - port: {get_resource: switch-trunk-ironic-port} + segmentation_id: 101 + segmentation_type: vlan + # Tenant VLANs + - port: {get_resource: switch-trunk-tenant-vlan103-port} + segmentation_id: 103 + segmentation_type: vlan + - port: {get_resource: switch-trunk-tenant-vlan104-port} + segmentation_id: 104 + segmentation_type: vlan + + switch-ironic0-br-port: + type: OS::Neutron::Port + properties: + network: {get_resource: ironic0-br-net} + port_security_enabled: false + mac_address: "22:57:f8:dd:fe:0c" + + switch-ironic1-br-port: + type: OS::Neutron::Port + properties: + network: {get_resource: ironic1-br-net} + port_security_enabled: false + mac_address: "22:57:f8:dd:fe:0d" + + switch: + type: OS::Nova::Server + properties: + image: {get_param: [switch_params, image]} + flavor: {get_param: [switch_params, flavor]} + config_drive: true + networks: + - port: {get_resource: switch-machine-port} + - port: {get_resource: switch-switch-mgmt-port} + - port: {get_attr: [switch-trunk, port_id]} + - port: {get_resource: switch-ironic0-br-port} + - port: {get_resource: switch-ironic1-br-port} + user_data_format: RAW + user_data: {get_resource: switch-init} + + # + # Ironics + # + ironic0-port: + type: OS::Neutron::Port + properties: + network: {get_resource: ironic0-br-net} + port_security_enabled: false + + ironic0: + type: OS::Nova::Server + properties: + flavor: {get_param: [ironic_params, flavor]} + block_device_mapping_v2: + - device_type: disk + boot_index: 1 + image_id: {get_param: [ironic_params, image]} + volume_size: 40 + delete_on_termination: true + - device_type: cdrom + disk_bus: {get_param: cdrom_disk_bus} + boot_index: 0 + image_id: {get_param: [ironic_params, cd_image]} + volume_size: 5 + delete_on_termination: true + networks: + - port: {get_resource: ironic0-port} + + ironic1-port: + type: OS::Neutron::Port + properties: + network: {get_resource: ironic1-br-net} + port_security_enabled: false + + ironic1: + type: OS::Nova::Server + properties: + flavor: {get_param: [ironic_params, flavor]} + block_device_mapping_v2: + - device_type: disk + boot_index: 1 + image_id: {get_param: [ironic_params, image]} + volume_size: 40 + delete_on_termination: true + - device_type: cdrom + disk_bus: {get_param: cdrom_disk_bus} + boot_index: 0 + image_id: {get_param: [ironic_params, cd_image]} + volume_size: 5 + delete_on_termination: true + networks: + - port: {get_resource: ironic1-port} + + +outputs: + controller_floating_ip: + description: Controller Floating IP + value: {get_attr: [controller-floating-ip, floating_ip_address]} + controller_ansible_host: + description: > + Controller ansible host, this struct can be passed to the ansible.builtin.add_host module + value: + name: controller-0 + ansible_ssh_user: zuul + ansible_host: {get_attr: [controller-floating-ip, floating_ip_address]} + ansible_ssh_common_args: '-o StrictHostKeyChecking=no' + groups: controllers + + sushy_emulator_uuids: + description: UUIDs of instances to manage with sushy-tools - RedFish virtual BMC + value: + ironic0: {get_resource: ironic0} + ironic1: {get_resource: ironic1} + + ironic_nodes: + description: Ironic nodes YAML, used with openstack baremetal create to enroll nodes in Openstack Ironic + value: + nodes: + - name: ironic0 + driver: redfish + bios_interface: no-bios + boot_interface: redfish-virtual-media + driver_info: + redfish_address: http://sushy-emulator.apps.sno.openstack.lab + redfish_system_id: + str_replace: + template: "/redfish/v1/Systems/$SYS_ID" + params: + $SYS_ID: {get_resource: ironic0} + redfish_username: admin + redfish_password: password + ports: + - address: {get_attr: [ironic0-port, mac_address]} + physical_network: bmnet + local_link_connection: + switch_info: nxos.openstack.lab + switch_id: "22:dd:fe:aa:1b:08" + port_id: "ethernet1/2" + - name: ironic1 + driver: redfish + bios_interface: no-bios + boot_interface: redfish-virtual-media + driver_info: + redfish_address: http://sushy-emulator.apps.sno.openstack.lab + redfish_system_id: + str_replace: + template: "/redfish/v1/Systems/$SYS_ID" + params: + $SYS_ID: {get_resource: ironic1} + redfish_username: admin + redfish_password: password + ports: + - address: {get_attr: [ironic1-port, mac_address]} + physical_network: bmnet + local_link_connection: + switch_info: nxos.openstack.lab + switch_id: "22:dd:fe:aa:1b:08" + port_id: "ethernet1/3" + + ocp_install_config: + description: OCP install-config.yaml + value: + apiVersion: v1 + baseDomain: openstack.lab + controlPlane: + architecture: amd64 + hyperthreading: Disabled + name: master + replicas: 1 + compute: + - architecture: amd64 + hyperthreading: Disabled + name: worker + replicas: 0 + metadata: + name: sno + networking: + clusterNetwork: + - cidr: 10.128.0.0/16 + hostPrefix: 23 + machineNetwork: + - cidr: {get_attr: [machine-subnet, cidr]} + serviceNetwork: + - 172.30.0.0/16 + networkType: OVNKubernetes + platform: + none: {} + pullSecret: _replaced_ + sshKey: {get_param: dataplane_ssh_pub_key} + + ocp_agent_config: + description: OCP agent-config.yaml + value: + apiVersion: v1beta1 + kind: AgentConfig + metadata: + name: sno + rendezvousIP: {get_attr: [master0-machine-port, fixed_ips, 0, ip_address]} + additionalNTPSources: {get_param: ntp_servers} + bootArtifactsBaseURL: + str_replace: + template: http://$server_address:8081/boot-artifacts + params: + $server_address: {get_attr: [controller-machine-port, fixed_ips, 0, ip_address]} + hosts: + - hostname: master-0 + role: master + interfaces: + - name: eth0 + macAddress: {get_attr: [master0-machine-port, mac_address]} + - name: eth1 + macAddress: {get_attr: [master0-ctlplane-trunk-parent-port, mac_address]} + - name: eth2 + macAddress: {get_attr: [master0-ironic-trunk-parent-port, mac_address]} + networkConfig: + interfaces: + - name: eth0 + type: ethernet + state: up + mac-address: {get_attr: [master0-machine-port, mac_address]} + ipv4: + enabled: true + dhcp: true + ipv6: + enabled: false + - name: eth1 + type: ethernet + state: down + mac-address: {get_attr: [master0-ctlplane-trunk-parent-port, mac_address]} + - name: eth2 + type: ethernet + state: down + mac-address: {get_attr: [master0-ironic-trunk-parent-port, mac_address]} + + ansible_inventory: + description: Ansible inventory + value: + all: + children: + controllers: + vars: + ocps: + vars: + switches: + vars: + localhosts: + hosts: + localhost: + ansible_connection: local + controllers: + hosts: + controller0: + ansible_host: {get_attr: [controller-machine-port, fixed_ips, 0, ip_address]} + ansible_user: zuul + ansible_ssh_common_args: '-o StrictHostKeyChecking=no' + ansible_ssh_private_key_file: '~/.ssh/id_rsa' + ocps: + hosts: + master0: + ansible_host: {get_attr: [master0-machine-port, fixed_ips, 0, ip_address]} + ansible_user: core + ansible_ssh_common_args: '-o StrictHostKeyChecking=no' + ansible_ssh_private_key_file: '~/.ssh/id_rsa' + switches: + hosts: + switch0: + ansible_host: {get_attr: [switch-machine-port, fixed_ips, 0, ip_address]} + ansible_user: admin + ansible_ssh_common_args: '-o StrictHostKeyChecking=no' + ansible_ssh_private_key_file: '~/.ssh/id_rsa' diff --git a/scenarios/sno-nxsw-netconf/manage-poap-md5sum.sh b/scenarios/sno-nxsw-netconf/manage-poap-md5sum.sh new file mode 100755 index 00000000..bec6ac79 --- /dev/null +++ b/scenarios/sno-nxsw-netconf/manage-poap-md5sum.sh @@ -0,0 +1,24 @@ +#!/bin/bash +# Script to manage POAP md5sum around black formatting +# This script: removes md5sum -> runs black -> restores md5sum + +set -e + +cd "$(dirname "$0")" +f=poap.py + +# Step 1: Remove the md5sum line +sed -i '/^# *md5sum=/d' "$f" + +# Step 2: Restore and update the md5sum line +# Insert the md5sum line after the shebang (line 2) +sed -i '1a\#md5sum="placeholder"' "$f" + +# Generate new md5sum excluding the md5sum line itself +sed '/^# *md5sum=/d' "$f" > "$f.md5" + +# Update the md5sum line with the correct hash +sed -i "s/^# *md5sum=.*/# md5sum=\"$(md5sum "$f.md5" | sed 's/ .*//')\"/" "$f" + +# Clean up temporary files +rm -f "$f.md5" diff --git a/scenarios/sno-nxsw-netconf/manifests/control-plane.yaml b/scenarios/sno-nxsw-netconf/manifests/control-plane.yaml new file mode 100644 index 00000000..ec206ecc --- /dev/null +++ b/scenarios/sno-nxsw-netconf/manifests/control-plane.yaml @@ -0,0 +1,499 @@ +--- +apiVersion: v1 +data: + server-ca-passphrase: MTIzNDU2Nzg= +kind: Secret +metadata: + name: octavia-ca-passphrase + namespace: openstack +type: Opaque +--- +apiVersion: v1 +data: + AdminPassword: MTIzNDU2Nzg= + AodhDatabasePassword: MTIzNDU2Nzg= + AodhPassword: MTIzNDU2Nzg= + BarbicanDatabasePassword: MTIzNDU2Nzg= + BarbicanPassword: MTIzNDU2Nzg= + BarbicanSimpleCryptoKEK: r0wDZ1zrD5upafX9RDfYqvDkW2LENBWH7Gz9+Tr3NdM= + CeilometerPassword: MTIzNDU2Nzg= + CinderDatabasePassword: MTIzNDU2Nzg= + CinderPassword: MTIzNDU2Nzg= + DatabasePassword: MTIzNDU2Nzg= + DbRootPassword: MTIzNDU2Nzg= + DesignateDatabasePassword: MTIzNDU2Nzg= + DesignatePassword: MTIzNDU2Nzg= + GlanceDatabasePassword: MTIzNDU2Nzg= + GlancePassword: MTIzNDU2Nzg= + HeatAuthEncryptionKey: NzY3YzNlZDA1NmNiYWEzYjlkZmVkYjhjNmY4MjViZjA= + HeatDatabasePassword: MTIzNDU2Nzg= + HeatPassword: MTIzNDU2Nzg= + IronicDatabasePassword: MTIzNDU2Nzg= + IronicInspectorDatabasePassword: MTIzNDU2Nzg= + IronicInspectorPassword: MTIzNDU2Nzg= + IronicPassword: MTIzNDU2Nzg= + KeystoneDatabasePassword: MTIzNDU2Nzg= + ManilaDatabasePassword: MTIzNDU2Nzg= + ManilaPassword: MTIzNDU2Nzg= + MetadataSecret: MTIzNDU2Nzg0Mg== + NeutronDatabasePassword: MTIzNDU2Nzg= + NeutronPassword: MTIzNDU2Nzg= + NovaAPIDatabasePassword: MTIzNDU2Nzg= + NovaCell0DatabasePassword: MTIzNDU2Nzg= + NovaCell1DatabasePassword: MTIzNDU2Nzg= + NovaPassword: MTIzNDU2Nzg= + OctaviaDatabasePassword: MTIzNDU2Nzg= + OctaviaHeartbeatKey: MTIzNDU2Nzg= + OctaviaPassword: MTIzNDU2Nzg= + PlacementDatabasePassword: MTIzNDU2Nzg= + PlacementPassword: MTIzNDU2Nzg= + SwiftPassword: MTIzNDU2Nzg= +kind: Secret +metadata: + name: osp-secret + namespace: openstack +type: Opaque +--- +apiVersion: core.openstack.org/v1beta1 +kind: OpenStackControlPlane +metadata: + name: controlplane + namespace: openstack +spec: + barbican: + enabled: false + ceilometer: + enabled: false + cinder: + apiOverride: + route: + haproxy.router.openshift.io/timeout: 60s + template: + cinderAPI: + override: + service: + internal: + metadata: + annotations: + metallb.universe.tf/address-pool: internalapi + metallb.universe.tf/allow-shared-ip: internalapi + metallb.universe.tf/loadBalancerIPs: 172.17.0.80 + spec: + type: LoadBalancer + replicas: 1 + cinderBackup: + customServiceConfig: | + [DEFAULT] + backup_driver = cinder.backup.drivers.swift.SwiftBackupDriver + networkAttachments: + - storage + replicas: 1 + cinderScheduler: + replicas: 1 + cinderVolumes: + lvm-iscsi: + customServiceConfig: | + [lvm] + image_volume_cache_enabled = false + volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver + volume_group = cinder-volumes + target_protocol = iscsi + target_helper = lioadm + volume_backend_name = lvm_iscsi + target_ip_address=172.18.0.10 + target_secondary_ip_addresses = 172.19.0.10 + nodeSelector: + openstack.org/cinder-lvm: "" + replicas: 1 + customServiceConfig: | + # Debug logs by default, jobs can override as needed. + [DEFAULT] + debug = true + databaseInstance: openstack + preserveJobs: false + secret: osp-secret + uniquePodNames: true + designate: + enabled: false + dns: + template: + options: + - key: server + values: + - 192.168.32.254 + override: + service: + metadata: + annotations: + metallb.universe.tf/address-pool: ctlplane + metallb.universe.tf/allow-shared-ip: ctlplane + metallb.universe.tf/loadBalancerIPs: 192.168.122.80 + spec: + type: LoadBalancer + replicas: 1 + galera: + enabled: true + templates: + openstack: + replicas: 1 + secret: osp-secret + storageRequest: 5G + openstack-cell1: + replicas: 1 + secret: osp-secret + storageRequest: 5G + glance: + apiOverrides: + default: + route: + haproxy.router.openshift.io/timeout: 60s + template: + customServiceConfig: | + [DEFAULT] + debug = True + enabled_backends = default_backend:swift + + [glance_store] + default_backend = default_backend + + [default_backend] + swift_store_create_container_on_put = True + swift_store_auth_version = 3 + swift_store_auth_address = {{ .KeystoneInternalURL }} + swift_store_endpoint_type = internalURL + swift_store_user = service:glance + swift_store_key = {{ .ServicePassword }} + databaseInstance: openstack + glanceAPIs: + default: + networkAttachments: + - storage + override: + service: + internal: + metadata: + annotations: + metallb.universe.tf/address-pool: internalapi + metallb.universe.tf/allow-shared-ip: internalapi + metallb.universe.tf/loadBalancerIPs: 172.17.0.80 + spec: + type: LoadBalancer + replicas: 1 + preserveJobs: false + storage: + storageClass: lvms-local-storage + storageRequest: 10G + uniquePodNames: true + heat: + enabled: false + horizon: + enabled: false + ironic: + enabled: true + template: + databaseInstance: openstack + ironicAPI: + customServiceConfig: | + [DEFAULT] + default_network_interface=neutron + override: + service: + internal: + metadata: + annotations: + metallb.universe.tf/address-pool: ironic + metallb.universe.tf/allow-shared-ip: ironic + metallb.universe.tf/loadBalancerIPs: 172.20.1.80 + spec: + type: LoadBalancer + replicas: 1 + ironicConductors: + # yamllint disable + - customServiceConfig: | + [DEFAULT] + default_network_interface=neutron + + [conductor] + power_state_change_timeout = 120 + + [redfish] + kernel_append_params = console=ttyS0 sshkey="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFiWZeyxnqZ/EZQzOT+kYYsk94UeM0+X5yXOPy77P9IL5xnPWSUVp/S9t0gz6J49t30Giz/ANS9l56h2E4M2VCJsu/amXhqCh19LFovv5vyLW9VSagn51aROIt3vO5u6P7AgY4mmbDRMz9w+OFvFyb3sJ1QNye38By0DDIqxVJf8Ue3hPMrmNUAgBkw0hEiNUd7p7oz6TNO90Ihvkt5GzfNXdvSo799/dD5EcM/EAnjwC4x+S2H3Yfuzq0fPeo+N6boB/Dsd1cYlMb1mKkSn5OLq8s0vZlLuYsZVOxVkN5BR3KLABB5Gw5/x7jK0n9+86CrD1IfnTYNG/OPy6pQ/ax" + + [neutron] + cleaning_network = provisioning + provisioning_network = provisioning + rescuing_network = provisioning + inspection_network = provisioning + # yamllint enable + networkAttachments: + - ironic + provisionNetwork: ironic + replicas: 1 + storageRequest: 10G + ironicInspector: + customServiceConfig: | + [capabilities] + boot_mode = true + + [processing] + update_pxe_enabled = false + inspectionNetwork: ironic + networkAttachments: + - ironic + override: + service: + internal: + metadata: + annotations: + metallb.universe.tf/address-pool: ctlplane + metallb.universe.tf/allow-shared-ip: ctlplane + metallb.universe.tf/loadBalancerIPs: 192.168.122.80 + spec: + type: LoadBalancer + preserveJobs: false + replicas: 1 + ironicNeutronAgent: + replicas: 1 + preserveJobs: false + rpcTransport: oslo + secret: osp-secret + keystone: + apiOverride: + route: {} + template: + databaseInstance: openstack + override: + service: + internal: + metadata: + annotations: + metallb.universe.tf/address-pool: internalapi + metallb.universe.tf/allow-shared-ip: internalapi + metallb.universe.tf/loadBalancerIPs: 172.17.0.80 + spec: + type: LoadBalancer + preserveJobs: false + replicas: 1 + secret: osp-secret + manila: + enabled: false + memcached: + templates: + memcached: + replicas: 1 + neutron: + apiOverride: + route: {} + template: + customServiceConfig: | + [DEFAULT] + vlan_transparent = true + agent_down_time = 600 + router_distributed = true + router_scheduler_driver = neutron.scheduler.l3_agent_scheduler.ChanceScheduler + allow_automatic_l3agent_failover = true + debug = true + + [agent] + report_interval = 300 + + [database] + max_retries = -1 + db_max_retries = -1 + + [keystone_authtoken] + region_name = regionOne + memcache_use_advanced_pool = True + + [oslo_messaging_notifications] + driver = noop + + [oslo_middleware] + enable_proxy_headers_parsing = true + + [oslo_policy] + policy_file = /etc/neutron/policy.yaml + + [ovs] + igmp_snooping_enable = true + + [ovn] + ovsdb_probe_interval = 60000 + ovn_emit_need_to_frag = true + + [ml2] + global_physnet_mtu = 1442 + type_drivers = geneve,vxlan,vlan,flat,local + tenant_network_types = geneve,vlan,flat + + [ml2_type_vlan] + network_vlan_ranges = datacentre,bmnet:100:120 + databaseInstance: openstack + extraMounts: + - name: switchConf + extraVol: + - volumes: + - name: neutron-switch-config + secret: + secretName: neutron-switch-config + mounts: + - name: neutron-switch-config + mountPath: /etc/neutron/neutron.conf.d/03-ml2-networking-baremetal.conf + subPath: 03-ml2-networking-baremetal.conf + readOnly: true + ml2MechanismDrivers: + - ovn + - baremetal + networkAttachments: + - internalapi + override: + service: + internal: + metadata: + annotations: + metallb.universe.tf/address-pool: internalapi + metallb.universe.tf/allow-shared-ip: internalapi + metallb.universe.tf/loadBalancerIPs: 172.17.0.80 + spec: + type: LoadBalancer + preserveJobs: false + replicas: 1 + secret: osp-secret + nova: + apiOverride: + route: {} + template: + apiServiceTemplate: + override: + service: + internal: + metadata: + annotations: + metallb.universe.tf/address-pool: internalapi + metallb.universe.tf/allow-shared-ip: internalapi + metallb.universe.tf/loadBalancerIPs: 172.17.0.80 + spec: + type: LoadBalancer + replicas: 1 + cellTemplates: + cell0: + cellDatabaseAccount: nova-cell0 + cellDatabaseInstance: openstack + cellMessageBusInstance: rabbitmq + hasAPIAccess: true + cell1: + cellDatabaseAccount: nova-cell1 + cellDatabaseInstance: openstack-cell1 + cellMessageBusInstance: rabbitmq-cell1 + hasAPIAccess: true + novaComputeTemplates: + compute-ironic: + computeDriver: ironic.IronicDriver + metadataServiceTemplate: + override: + service: + metadata: + annotations: + metallb.universe.tf/address-pool: internalapi + metallb.universe.tf/allow-shared-ip: internalapi + metallb.universe.tf/loadBalancerIPs: 172.17.0.80 + spec: + type: LoadBalancer + replicas: 1 + preserveJobs: false + schedulerServiceTemplate: + replicas: 1 + secret: osp-secret + octavia: + enabled: false + ovn: + template: + ovnController: + networkAttachment: tenant + nicMappings: + datacentre: ospbr + bmnet: ironicbr + ovnDBCluster: + ovndbcluster-nb: + dbType: NB + networkAttachment: internalapi + replicas: 1 + storageRequest: 10G + ovndbcluster-sb: + dbType: SB + networkAttachment: internalapi + replicas: 1 + storageRequest: 10G + ovnNorthd: + logLevel: info + nThreads: 1 + replicas: 1 + resources: {} + tls: {} + placement: + apiOverride: + route: {} + template: + databaseInstance: openstack + override: + service: + internal: + metadata: + annotations: + metallb.universe.tf/address-pool: internalapi + metallb.universe.tf/allow-shared-ip: internalapi + metallb.universe.tf/loadBalancerIPs: 172.17.0.80 + spec: + type: LoadBalancer + preserveJobs: false + replicas: 1 + secret: osp-secret + rabbitmq: + templates: + rabbitmq: + override: + service: + metadata: + annotations: + metallb.universe.tf/address-pool: internalapi + metallb.universe.tf/loadBalancerIPs: 172.17.0.85 + spec: + type: LoadBalancer + replicas: 1 + rabbitmq-cell1: + override: + service: + metadata: + annotations: + metallb.universe.tf/address-pool: internalapi + metallb.universe.tf/loadBalancerIPs: 172.17.0.86 + spec: + type: LoadBalancer + replicas: 1 + secret: osp-secret + storageClass: lvms-local-storage + notificationsBus: + cluster: rabbitmq + swift: + enabled: true + proxyOverride: + route: {} + template: + swiftProxy: + override: + service: + internal: + metadata: + annotations: + metallb.universe.tf/address-pool: internalapi + metallb.universe.tf/allow-shared-ip: internalapi + metallb.universe.tf/loadBalancerIPs: 172.17.0.80 + spec: + type: LoadBalancer + replicas: 1 + swiftRing: + ringReplicas: 1 + swiftStorage: + replicas: 1 + telemetry: + enabled: false diff --git a/scenarios/sno-nxsw-netconf/manifests/dnsmasq-dns-ironic.yaml b/scenarios/sno-nxsw-netconf/manifests/dnsmasq-dns-ironic.yaml new file mode 100644 index 00000000..fc4ae709 --- /dev/null +++ b/scenarios/sno-nxsw-netconf/manifests/dnsmasq-dns-ironic.yaml @@ -0,0 +1,27 @@ +--- +apiVersion: v1 +kind: Service +metadata: + annotations: + core.openstack.org/ingress_create: "false" + metallb.io/ip-allocated-from-pool: ironic + metallb.universe.tf/address-pool: ironic + metallb.universe.tf/allow-shared-ip: ironic + metallb.universe.tf/loadBalancerIPs: 172.20.1.80 + name: dnsmasq-dns-ironic + namespace: openstack + labels: + service: dnsmasq +spec: + ports: + - name: dnsmasq + port: 53 + protocol: UDP + targetPort: 5353 + - name: dnsmasq-tcp + port: 53 + protocol: TCP + targetPort: 5353 + selector: + service: dnsmasq + type: LoadBalancer diff --git a/scenarios/sno-nxsw-netconf/manifests/networking/metallb.yaml b/scenarios/sno-nxsw-netconf/manifests/networking/metallb.yaml new file mode 100644 index 00000000..8b9bdb15 --- /dev/null +++ b/scenarios/sno-nxsw-netconf/manifests/networking/metallb.yaml @@ -0,0 +1,110 @@ +--- +apiVersion: metallb.io/v1beta1 +kind: IPAddressPool +metadata: + labels: + osp/lb-addresses-type: standard + name: ironic + namespace: metallb-system +spec: + addresses: + - 172.20.1.80-172.20.1.90 +--- +apiVersion: metallb.io/v1beta1 +kind: IPAddressPool +metadata: + labels: + osp/lb-addresses-type: standard + name: ctlplane + namespace: metallb-system +spec: + addresses: + - 192.168.122.80-192.168.122.90 +--- +apiVersion: metallb.io/v1beta1 +kind: IPAddressPool +metadata: + labels: + osp/lb-addresses-type: standard + name: internalapi + namespace: metallb-system +spec: + addresses: + - 172.17.0.80-172.17.0.90 +--- +apiVersion: metallb.io/v1beta1 +kind: IPAddressPool +metadata: + labels: + osp/lb-addresses-type: standard + name: storage + namespace: metallb-system +spec: + addresses: + - 172.18.0.80-172.18.0.90 +--- +apiVersion: metallb.io/v1beta1 +kind: IPAddressPool +metadata: + labels: + osp/lb-addresses-type: standard + name: tenant + namespace: metallb-system +spec: + addresses: + - 172.19.0.80-172.19.0.90 +--- +apiVersion: metallb.io/v1beta1 +kind: L2Advertisement +metadata: + name: ironic + namespace: metallb-system +spec: + interfaces: + - ironicbr.101 + ipAddressPools: + - ironic +--- +apiVersion: metallb.io/v1beta1 +kind: L2Advertisement +metadata: + name: ctlplane + namespace: metallb-system +spec: + interfaces: + - ospbr + ipAddressPools: + - ctlplane +--- +apiVersion: metallb.io/v1beta1 +kind: L2Advertisement +metadata: + name: internalapi + namespace: metallb-system +spec: + interfaces: + - internalapi + ipAddressPools: + - internalapi +--- +apiVersion: metallb.io/v1beta1 +kind: L2Advertisement +metadata: + name: storage + namespace: metallb-system +spec: + interfaces: + - storage + ipAddressPools: + - storage +--- +apiVersion: metallb.io/v1beta1 +kind: L2Advertisement +metadata: + name: tenant + namespace: metallb-system +spec: + interfaces: + - tenant + ipAddressPools: + - tenant diff --git a/scenarios/sno-nxsw-netconf/manifests/networking/nad.yaml b/scenarios/sno-nxsw-netconf/manifests/networking/nad.yaml new file mode 100644 index 00000000..18d6aad1 --- /dev/null +++ b/scenarios/sno-nxsw-netconf/manifests/networking/nad.yaml @@ -0,0 +1,155 @@ +--- +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + labels: + osp/net: ctlplane + osp/net-attach-def-type: standard + name: ctlplane + namespace: openstack +spec: + config: | + { + "cniVersion": "0.3.1", + "name": "ctlplane", + "type": "macvlan", + "master": "ospbr", + "ipam": { + "type": "whereabouts", + "range": "192.168.122.0/24", + "range_start": "192.168.122.30", + "range_end": "192.168.122.70" + } + } +--- +# NAD for ovn-controller to access the bmnet physical network (like datacentre) +# Must be created BEFORE ovn-operator to prevent host-device NAD creation +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + labels: + osp/net: bmnet + osp/net-attach-def-type: standard + name: bmnet + namespace: openstack +spec: + config: | + { + "cniVersion": "0.3.1", + "name": "bmnet", + "type": "bridge", + "bridge": "ironicbr", + "ipam": {} + } +--- +# NAD for OpenStack pods (ironic-conductor, etc.) to attach to provisioning network +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + labels: + osp/net: ironic + osp/net-attach-def-type: standard + name: ironic + namespace: openstack +spec: + config: | + { + "cniVersion": "0.3.1", + "name": "ironicnet", + "type": "macvlan", + "master": "ironicbr.101", + "mtu": 1442, + "ipam": { + "type": "whereabouts", + "range": "172.20.1.0/24", + "range_start": "172.20.1.30", + "range_end": "172.20.1.70" + } + } +--- +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + labels: + osp/net: datacentre + osp/net-attach-def-type: standard + name: datacentre + namespace: openstack +spec: + config: | + { + "cniVersion": "0.3.1", + "name": "datacentre", + "type": "bridge", + "bridge": "ospbr", + "ipam": {} + } +--- +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + labels: + osp/net: internalapi + osp/net-attach-def-type: standard + name: internalapi + namespace: openstack +spec: + config: | + { + "cniVersion": "0.3.1", + "name": "internalapi", + "type": "macvlan", + "master": "internalapi", + "ipam": { + "type": "whereabouts", + "range": "172.17.0.0/24", + "range_start": "172.17.0.30", + "range_end": "172.17.0.70" + } + } +--- +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + labels: + osp/net: storage + osp/net-attach-def-type: standard + name: storage + namespace: openstack +spec: + config: | + { + "cniVersion": "0.3.1", + "name": "storage", + "type": "macvlan", + "master": "storage", + "ipam": { + "type": "whereabouts", + "range": "172.18.0.0/24", + "range_start": "172.18.0.30", + "range_end": "172.18.0.70" + } + } +--- +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + labels: + osp/net: tenant + osp/net-attach-def-type: standard + name: tenant + namespace: openstack +spec: + config: | + { + "cniVersion": "0.3.1", + "name": "tenant", + "type": "macvlan", + "master": "tenant", + "ipam": { + "type": "whereabouts", + "range": "172.19.0.0/24", + "range_start": "172.19.0.30", + "range_end": "172.19.0.70" + } + } diff --git a/scenarios/sno-nxsw-netconf/manifests/networking/netconfig.yaml b/scenarios/sno-nxsw-netconf/manifests/networking/netconfig.yaml new file mode 100644 index 00000000..637d66d3 --- /dev/null +++ b/scenarios/sno-nxsw-netconf/manifests/networking/netconfig.yaml @@ -0,0 +1,50 @@ +--- +apiVersion: network.openstack.org/v1beta1 +kind: NetConfig +metadata: + name: netconfig + namespace: openstack +spec: + networks: + - dnsDomain: ctlplane.openstack.lab + mtu: 1442 + name: ctlplane + subnets: + - allocationRanges: + - end: 192.168.122.120 + start: 192.168.122.100 + - end: 192.168.122.200 + start: 192.168.122.150 + cidr: 192.168.122.0/24 + gateway: 192.168.122.1 + name: subnet1 + - dnsDomain: internalapi.openstack.lab + mtu: 1442 + name: internalapi + subnets: + - allocationRanges: + - end: 172.17.0.250 + start: 172.17.0.100 + cidr: 172.17.0.0/24 + name: subnet1 + vlan: 20 + - dnsDomain: storage.openstack.lab + mtu: 1442 + name: storage + subnets: + - allocationRanges: + - end: 172.18.0.250 + start: 172.18.0.100 + cidr: 172.18.0.0/24 + name: subnet1 + vlan: 21 + - dnsDomain: tenant.openstack.lab + mtu: 1442 + name: tenant + subnets: + - allocationRanges: + - end: 172.19.0.250 + start: 172.19.0.100 + cidr: 172.19.0.0/24 + name: subnet1 + vlan: 22 diff --git a/scenarios/sno-nxsw-netconf/manifests/networking/nncp.yaml b/scenarios/sno-nxsw-netconf/manifests/networking/nncp.yaml new file mode 100644 index 00000000..a6141855 --- /dev/null +++ b/scenarios/sno-nxsw-netconf/manifests/networking/nncp.yaml @@ -0,0 +1,139 @@ +--- +apiVersion: nmstate.io/v1 +kind: NodeNetworkConfigurationPolicy +metadata: + labels: + osp/nncm-config-type: standard + name: master-0 + namespace: openstack +spec: + desiredState: + dns-resolver: + config: + search: [] + server: + - 192.168.32.254 + interfaces: + - name: internalapi + type: vlan + description: internalapi vlan interface + ipv4: + address: + - ip: 172.17.0.10 + prefix-length: "24" + dhcp: false + enabled: true + ipv6: + enabled: false + mtu: 1442 + state: up + vlan: + base-iface: eth1 + id: "20" + - name: storage + type: vlan + description: storage vlan interface + ipv4: + address: + - ip: 172.18.0.10 + prefix-length: "24" + dhcp: false + enabled: true + ipv6: + enabled: false + mtu: 1442 + state: up + vlan: + base-iface: eth1 + id: "21" + - name: tenant + type: vlan + description: tenant vlan interface + ipv4: + address: + - ip: 172.19.0.10 + prefix-length: "24" + dhcp: false + enabled: true + ipv6: + enabled: false + mtu: 1442 + state: up + vlan: + base-iface: eth1 + id: "22" + - description: ctlplane interface + mtu: 1442 + name: eth1 + state: up + type: ethernet + - name: ospbr + type: linux-bridge + description: linux-bridge over ctlplane interface + bridge: + options: + stp: + enabled: false + port: + - name: eth1 + vlan: {} + ipv4: + address: + - ip: 192.168.122.10 + prefix-length: "24" + dhcp: false + enabled: true + ipv6: + enabled: false + mtu: 1442 + state: up + - description: ironic interface + mtu: 1442 + name: eth2 + state: up + type: ethernet + - name: ironicbr + type: linux-bridge + description: ironic-bridge + bridge: + options: + stp: + enabled: false + port: + - name: eth2 + ipv4: + enabled: false + ipv6: + enabled: false + mtu: 1442 + - name: ironicbr.101 + type: vlan + description: ironic provisioning vlan interface + state: up + vlan: + base-iface: ironicbr + id: 101 + ipv4: + address: + - ip: 172.20.1.10 + prefix-length: "24" + dhcp: false + enabled: true + ipv6: + enabled: false + mtu: 1442 + route-rules: + config: [] + routes: + config: + - destination: 172.20.3.0/24 + next-hop-address: 172.20.1.1 + next-hop-interface: ironicbr.101 + table-id: 254 + - destination: 172.20.4.0/24 + next-hop-address: 172.20.1.1 + next-hop-interface: ironicbr.101 + table-id: 254 + nodeSelector: + kubernetes.io/hostname: master-0 + node-role.kubernetes.io/worker: "" diff --git a/scenarios/sno-nxsw-netconf/manifests/openstack-version.yaml b/scenarios/sno-nxsw-netconf/manifests/openstack-version.yaml new file mode 100644 index 00000000..f4e8c47e --- /dev/null +++ b/scenarios/sno-nxsw-netconf/manifests/openstack-version.yaml @@ -0,0 +1,8 @@ +--- +apiVersion: core.openstack.org/v1beta1 +kind: OpenStackVersion +metadata: + name: controlplane + namespace: openstack +spec: + customContainerImages: {} diff --git a/scenarios/sno-nxsw-netconf/poap.cfg b/scenarios/sno-nxsw-netconf/poap.cfg new file mode 100644 index 00000000..fe652b82 --- /dev/null +++ b/scenarios/sno-nxsw-netconf/poap.cfg @@ -0,0 +1,70 @@ +hostname nxos.openstack.lab +vdc nxos.openstack.lab id 1 + limit-resource vlan minimum 16 maximum 4094 + limit-resource vrf minimum 2 maximum 4096 + limit-resource port-channel minimum 0 maximum 511 + limit-resource m4route-mem minimum 58 maximum 58 + limit-resource m6route-mem minimum 8 maximum 8 + +feature netconf +feature lacp +! feature lldp + +no password strength-check +username admin password 5 $5$LFGHKE$ND3U7npkwgMUzxLjjgDQxCx8JFJ5.ZlFyQTTt1vZgA5 role network-admin +ssh key rsa 2048 force + +system default switchport + +vrf context management + ip name-server 192.168.32.254 + ip route 0.0.0.0/0 192.168.32.1 + +vlan 102 + name Native VLAN + +interface mgmt0 + ip address dhcp + vrf member management + +interface Ethernet1/1 + description "Trunk port" + switchport + switchport mode trunk + switchport trunk allowed vlan 101,102,103,104 + no shutdown + +interface Ethernet1/2 + description "Access port ironic0" + switchport + switchport mode access + switchport access vlan 102 + no shutdown + +interface Ethernet1/3 + description "Access port ironic1" + switchport + switchport mode access + switchport access vlan 102 + no shutdown + +interface Ethernet1/4 + switchport + switchport mode access + switchport access vlan 102 + no shutdown + +interface Ethernet1/5 + switchport + switchport mode access + switchport access vlan 102 + no shutdown + +interface Ethernet1/6 + switchport + switchport mode access + switchport access vlan 102 + no shutdown + +line console +line vty diff --git a/scenarios/sno-nxsw-netconf/poap.py b/scenarios/sno-nxsw-netconf/poap.py new file mode 100644 index 00000000..4ec37451 --- /dev/null +++ b/scenarios/sno-nxsw-netconf/poap.py @@ -0,0 +1,437 @@ +#!/bin/env python3 +# md5sum="9e17ba7c52a86b52319cf76678fdf9aa" + +# If any changes are made to this script, please run the below command +# in bash shell to update the above md5sum. This is used for integrity check. +# f=poap.py ; sed '/^# *md5sum/d' "$f" > "$f.md5" ; sed -i \ +# "s/^# *md5sum=.*/#md5sum=\"$(md5sum $f.md5 | sed 's/ .*//')\"/" $f + +# Protocol Authentication Support: +# - SCP: Always requires username and password +# - HTTP/HTTPS: Authentication optional (anonymous or username/password) +# - TFTP: No authentication support (anonymous only) +# +# Authentication parameters (username/password) are only required for SCP. +# For HTTP/HTTPS, if you provide username, you must also provide password. +# +# Example usage: +# - Anonymous HTTP: protocol="http", hostname="server.com" (no auth needed) +# - Authenticated HTTP: protocol="http", hostname="server.com", port=8080, username="user", password="pass" +# - Anonymous TFTP: protocol="tftp", hostname="server.com" (no auth supported) +# - Authenticated SCP: protocol="scp", hostname="server.com", username="user", password="pass" (auth required) + +import os +import re +import signal +import sys +import syslog +import traceback +import time + +try: + from cisco import cli +except ImportError: + from cli import * + +# Default configuration options +DEFAULT_OPTS = { + "hostname": "192.168.32.254", + "protocol": "tftp", + "cfg_path": "/poap.cfg", + "dest_path": "bootflash:poap.cfg", + "ignore_cert": True, + # username, password and port are optional - only needed for SCP or authenticated HTTP/HTTPS + # port is optional for custom ports on any protocol (e.g., SCP:2222, HTTP:8080, HTTPS:8088, TFTP:6969) +} + +# Valid configuration options +VALID_OPTS = { + "username", + "password", + "hostname", + "protocol", + "port", + "cfg_path", + "dest_path", + "ignore_cert", + "vrf", +} + +# Required configuration parameters (always required) +REQUIRED_OPTS = {"hostname"} + +# Logging prefix for syslog messages +SYSLOG_PREFIX = "POAPHandler" + + +def get_log_file_path(): + """Generate the log file path with timestamp and PID""" + return "/bootflash/%s_poap_%s_script.log" % ( + time.strftime("%Y%m%d%H%M%S", time.gmtime()), + os.environ["POAP_PID"], + ) + + +class POAPHandler: + """ + POAP (Power-On Auto Provisioning) handler for Cisco NXOS switches. + Handles configuration download and application for switch bootstrap. + """ + + def __init__(self): + """Initialize the POAP handler with default settings.""" + + # Set up signal handler + signal.signal(signal.SIGTERM, self.sigterm_handler) + + # Initialize logging + self.syslog_prefix = SYSLOG_PREFIX + self.log_file_handler = None # Will be set when using context manager + + self.opts = DEFAULT_OPTS.copy() + + # Validate required parameters + self._validate_required_opts() + + # Check that options are valid + self.validate_opts() + + @property + def dest_path(self): + """Normalized destination path without trailing slashes""" + return self.opts["dest_path"].rstrip("/") + + @property + def cfg_path(self): + """Normalized config path without trailing slashes""" + return self.opts["cfg_path"].rstrip("/") + + def _validate_required_opts(self): + """Validates that required options are provided""" + missing_params = REQUIRED_OPTS.difference(self.opts.keys()) + + if missing_params: + self._log("Required parameters are missing:") + self.abort("Missing %s" % ", ".join(missing_params)) + + # Protocol-specific validation for authentication parameters + protocol = self.opts.get("protocol", "http") + + # SCP always requires authentication + if protocol == "scp": + username = self.opts.get("username") + password = self.opts.get("password") + if not username or not password: + self.abort("SCP protocol requires both username and password") + + # HTTP/HTTPS with authentication requires both username and password + if protocol in ["http", "https"]: + username = self.opts.get("username") + password = self.opts.get("password") + if (username and not password) or (password and not username): + self.abort( + "HTTP/HTTPS authentication requires both username and password" + ) + + # Validate port if provided + port = self.opts.get("port") + if port is not None: + try: + port_int = int(port) + if not (1 <= port_int <= 65535): + self.abort("Port must be between 1 and 65535") + except (ValueError, TypeError): + self.abort("Port must be a valid integer") + + def validate_opts(self): + """ + Validates that the options provided by the user are valid. + Aborts the script if they are not. + """ + # Find any invalid options (ones not in VALID_OPTS) + invalid_opts = set(self.opts.keys()) - VALID_OPTS + if invalid_opts: + self._log( + "Invalid options detected: %s (check spelling, capitalization, and underscores)" + % ", ".join(invalid_opts) + ) + self.abort() + + def abort(self, msg=None): + """Aborts the POAP script + + :param msg: The message to log before aborting + """ + if msg: + self._log(msg) + + # Destination config + self.cleanup_file_from_option("dest_cfg") + + # Log file will be closed by context manager + exit(1) + + def _redact_passwords(self, message): + """Redacts passwords from log messages for security + + :param message: The log message to redact passwords from + :return: The message with passwords replaced with '' + """ + parts = re.split("\s+", message.strip()) + for index, part in enumerate(parts): + # blank out the password after the password keyword (terminal password *****, etc.) + if part == "password" and len(parts) >= index + 2: + parts[index + 1] = "" + + return " ".join(parts) + + def _log(self, info): + """ + Log the trace into console and poap_script log file in bootflash + + :param info: The information that needs to be logged. + """ + # Redact sensitive information before logging + info = self._redact_passwords(info) + + # Add syslog prefix + info = "%s - %s" % (self.syslog_prefix, info) + + syslog.syslog(9, info) + if self.log_file_handler is not None: + print(info, file=self.log_file_handler, flush=True) + + def remove_file(self, filename): + """Removes a file if it exists and it's not a directory. + + :param filename: The file to remove + """ + if os.path.isfile(filename): + try: + os.remove(filename) + except (IOError, OSError) as e: + self._log("Failed to remove %s: %s" % (filename, str(e))) + + def cleanup_file_from_option(self, option, bootflash_root=False): + """Removes a file indicated by the option in the POAP opts and removes it if it exists. + + Handle the cases where the variable is unused or not set yet. + + :param option: The option to remove + :param bootflash_root: Whether to remove the file from the bootflash root + """ + try: + filename = self.opts[option] + if filename is None: + return # Nothing to clean up + + if bootflash_root: + path = "/bootflash" + else: + path = self.dest_path + + self.remove_file(os.path.join(path, filename)) + self.remove_file(os.path.join(path, "%s.tmp" % filename)) + except KeyError: + # Option doesn't exist, nothing to clean up + pass + + def sigterm_handler(self, signum, stack): + """ + A signal handler for the SIGTERM signal. Cleans up and exits + + :param signum: The signal number + :param stack: The stack trace + """ + self.abort("SIGTERM signal received") + + def process_cfg_file(self): + """ + Processes the downloaded switch configuration file. + Copies the config to the scheduled config file for bootstrap replay. + """ + self._log("Processing Config file") + + # Copy config directly to scheduled-config + self._log("Command: copy %s scheduled-config" % self.opts["dest_path"]) + cli("copy %s scheduled-config" % self.opts["dest_path"]) + + self._log("Config processed and prepared for scheduled application") + + def _build_copy_cmd(self, source, dest): + """Build the copy command with all necessary options + + :param source: Source file path on remote server + :param dest: Destination path on local switch + :return: Complete copy command string + """ + # Extract parameters from opts + protocol = self.opts["protocol"] + host = self.opts["hostname"] + user = self.opts.get("username") + password = self.opts.get("password") + vrf = self.opts["vrf"] + ignore_ssl = self.opts["ignore_cert"] + port = self.opts.get("port") + # Build copy command with Cisco NX-OS terminal automation features + parts = [] + + # terminal dont-ask: Auto-answer "yes" to all confirmation prompts + parts.append("terminal dont-ask") + + # Determine if authentication is needed based on protocol and credentials + auth_needed = protocol in ["scp"] or ( + protocol in ["http", "https"] and user and password + ) + + if auth_needed: + if password: + # terminal password: Pre-store password for automatic authentication + # The password will be used automatically for any auth prompts during copy + parts.append("terminal password %s" % password) + + # Build the copy URL - only include user@ if authentication is needed + url = "%s://" % protocol + if auth_needed and user: + url += "%s@" % user + + # Add hostname with optional port + url += host + if port: + url += ":%s" % port + + # Ensure source path starts with / for proper URL construction + if not source.startswith("/"): + source = "/" + source + url += source + + # Build the copy command with URL and destination + cmd = "copy %s %s" % (url, dest) + + # Add ignore-certificate if needed + if protocol == "https" and ignore_ssl: + cmd += " ignore-certificate" + + # Add VRF + cmd += " vrf %s" % vrf + + # Add the complete copy command to the parts + parts.append(cmd) + + # Join all command parts with semicolon separator + return " ; ".join(parts) + + def copy(self, source, dest): + """Copies the files + + Copy the provided file from source to destination via network transfer. + + :param source: The source file to copy + :param dest: The destination file to copy + """ + self._log("Copying %s to %s" % (source, dest)) + + # Build the copy command using the dedicated method + cmd = self._build_copy_cmd(source, dest) + self._log("Copy command: %s" % cmd) + + try: + cli(cmd) + except Exception as e: + exc_type, exc_value, exc_tb = sys.exc_info() + traceback_str = "".join( + traceback.format_exception(exc_type, exc_value, exc_tb) + ) + self._log("Copy failed, Traceback: %s" % traceback_str) + self.abort("Copy failed: %s" % str(e)) + + self._log("Copy completed successfully") + + def get_currently_booted_image_filename(self): + match = None + try: + output = cli("show version") + except Exception as e: + self.abort("Show version failed: %s" % str(e)) + + match = re.search("NXOS image file is:\s+(.+)", output) + + if match: + directory, image = os.path.split(match.group(1)) + return image.strip() + + def install_nxos(self): + """Install the NXOS image on the switch + + Assume the current image is the one we want to install. + """ + boot_image = self.get_currently_booted_image_filename() + if not boot_image: + self.abort("No image found to install") + + # Build the install command + parts = [] + parts.append("terminal dont-ask ") + parts.append("install all nxos %s no-reload non-interruptive" % boot_image) + parts.append("terminal dont-ask") + parts.append("write erase") + cmd = " ; ".join(parts) + self._log("Install command: %s" % cmd) + + try: + cli(cmd) + time.sleep(5) + except Exception as e: + self._log("Failed to ISSU to image %s" % boot_image) + self.abort(str(e)) + + def run_with_logging(self): + """Execute the POAP provisioning process with proper log file management""" + log_file = get_log_file_path() + + with open(log_file, "w+") as log_file_handler: + self.log_file_handler = log_file_handler + self._log("Logfile name: %s" % log_file) + + try: + self.run() + finally: + # Ensure log_file_handler is reset when context exits + self.log_file_handler = None + + def run(self): + """Execute the POAP provisioning process""" + + # Set dynamic VRF value from environment + self.opts.setdefault("vrf", os.environ.get("POAP_VRF", "management")) + + # # create the directory structure needed, if any + # self.create_dest_dirs() + + # Copy config + self.copy(self.opts["cfg_path"], self.opts["dest_path"]) + self.process_cfg_file() + + # Install the NXOS image + self.install_nxos() + + +def main(): + """Main entry point for the POAP script""" + poap_handler = POAPHandler() + poap_handler.run_with_logging() + + +if __name__ == "__main__": + try: + main() + except Exception: + exc_type, exc_value, exc_tb = sys.exc_info() + # Create a temporary handler just for error logging + handler = POAPHandler() + + # Log the full traceback as a single formatted string + traceback_str = "".join(traceback.format_exception(exc_type, exc_value, exc_tb)) + handler._log("Exception occurred:\n%s" % traceback_str) + + handler.abort() diff --git a/scenarios/sno-nxsw-netconf/test-operator/README.md b/scenarios/sno-nxsw-netconf/test-operator/README.md new file mode 100644 index 00000000..dc44ea00 --- /dev/null +++ b/scenarios/sno-nxsw-netconf/test-operator/README.md @@ -0,0 +1,42 @@ + +# AI generated README + +## Tempest Tests Configuration for Test Operator + +The YAML file, `tempest-tests.yml`, is a configuration for running Tempest +tests, which is a validation framework for OpenStack. Here's a breakdown of the +configuration: + +1. **apiVersion, kind, and metadata**: These fields define the API version, + kind (type) of resource, and metadata (name and namespace) for the Tempest + test job. + +2. **spec**: This section contains the configuration for the Tempest test job. + + - **networkAttachments**: This field specifies the network attachment for + the test job. In this case, it's set to `ctlplane`. + - **storageClass**: This field sets the storage class for the test job to + `lvms-local-storage`. + - **privileged**: This field is set to `true`, which means the test + containers will have elevated privileges. + - **workflow**: This section defines the steps to be executed in the test + job. There are two steps in this configuration: + - **ironic-scenario-testing**: This step runs scenario tests for Ironic, + the OpenStack bare-metal provisioning service. The `tempestconfRun` + section configures Tempest settings for this step, such as disabling + isolated networks, setting the number of available nodes, and specifying + the compute flavor and hypervisor type. The `tempestRun` section + specifies the concurrency level and the list of tests to include and + exclude. + - **ironic-api-testing**: This step runs API tests for Ironic. Similar to + the previous step, the `tempestconfRun` section configures Tempest + settings, and the `tempestRun` section specifies the concurrency level + and the list of tests to include and exclude. + +In summary, this YAML file configures a Tempest test job to run two types of +tests for Ironic: scenario tests and API tests. The tests are executed with +specific configurations and concurrency levels. diff --git a/scenarios/sno-nxsw-netconf/test-operator/automation-vars.yml b/scenarios/sno-nxsw-netconf/test-operator/automation-vars.yml new file mode 100644 index 00000000..b8cbc29c --- /dev/null +++ b/scenarios/sno-nxsw-netconf/test-operator/automation-vars.yml @@ -0,0 +1,284 @@ +--- +stages: + - name: Apply ironic network-attachement-definition + manifest: manifests/nad.yaml + wait_conditions: + - >- + oc wait -n sushy-emulator network-attachment-definitions.k8s.cni.cncf.io ironic + --for jsonpath='{.metadata.annotations}' --timeout=30s + + - name: Patch RedFish Sushy Emulator Deployment - add network attachment + shell: | + set -xe -o pipefail + + TMP_DIR="$(mktemp -d)" + trap 'rm -rf -- "$TMP_DIR"' EXIT + + oc project sushy-emulator + + cat << EOF > ${TMP_DIR}/sushy-emulator-network-annotations-patch.yaml + spec: + template: + metadata: + annotations: + k8s.v1.cni.cncf.io/networks: '[{"name":"ironic","namespace":"sushy-emulator","interface":"ironic"}]' + EOF + + oc patch deployments.apps sushy-emulator --patch-file ${TMP_DIR}/sushy-emulator-network-annotations-patch.yaml + wait_conditions: + - "oc -n sushy-emulator wait deployments.apps sushy-emulator --for condition=Available --timeout=300s" + + - name: Set a multiattach volume type and create it if needed + shell: | + set -xe -o pipefail + oc project openstack + + oc rsh openstackclient openstack volume type show multiattach &>/dev/null || \ + oc rsh openstackclient openstack volume type create multiattach + + oc rsh openstackclient openstack volume type set --property multiattach=" True" multiattach + + - name: Create public network if needed + shell: | + set -xe -o pipefail + oc project openstack + + oc rsh openstackclient openstack network show public &>/dev/null || \ + oc rsh openstackclient openstack network create public \ + --external \ + --no-share \ + --default \ + --provider-network-type flat \ + --provider-physical-network datacentre + + - name: Create subnet on public network if needed + shell: | + set -xe -o pipefail + oc project openstack + + oc rsh openstackclient openstack subnet show public_subnet &>/dev/null || \ + oc rsh openstackclient openstack subnet create public_subnet \ + --network public \ + --subnet-range 192.168.122.0/24 \ + --allocation-pool start=192.168.122.171,end=192.168.122.250 \ + --gateway 192.168.122.1 \ + --dhcp + + - name: Create private network if needed + shell: | + set -xe -o pipefail + oc project openstack + + oc rsh openstackclient openstack network show private &>/dev/null || \ + oc rsh openstackclient openstack network create private --share + + - name: Create subnet on private network if needed + shell: | + set -xe -o pipefail + oc project openstack + + oc rsh openstackclient openstack subnet show private_subnet &>/dev/null || \ + oc rsh openstackclient openstack subnet create private_subnet \ + --network private \ + --subnet-range 10.2.0.0/24 \ + --allocation-pool start=10.2.0.10,end=10.2.0.250 \ + --gateway 10.2.0.1 \ + --dhcp + + - name: Create network for ironic provisioning if needed + shell: | + set -xe -o pipefail + oc project openstack + + oc rsh openstackclient openstack network show provisioning &>/dev/null || \ + oc rsh openstackclient \ + openstack network create provisioning \ + --share \ + --disable-port-security \ + --provider-physical-network bmnet \ + --provider-network-type vlan \ + --provider-segment 101 + + - name: Create subnet for ironic provisioning if needed + shell: | + set -xe -o pipefail + oc project openstack + + oc rsh openstackclient openstack subnet show provisioning-subnet &>/dev/null || \ + oc rsh openstackclient \ + openstack subnet create provisioning-subnet \ + --network provisioning \ + --subnet-range 172.20.1.0/24 \ + --gateway 172.20.1.1 \ + --dns-nameserver 172.20.1.80 \ + --allocation-pool start=172.20.1.100,end=172.20.1.200 + + - name: Create baremetal flavor if needed + shell: | + set -xe -o pipefail + oc project openstack + + oc rsh openstackclient openstack flavor show baremetal &>/dev/null || \ + oc rsh openstackclient \ + openstack flavor create baremetal \ + --id 123456789-1234-1234-1234-000000000001 \ + --ram 1024 \ + --vcpus 1 \ + --disk 15 \ + --property resources:VCPU=0 \ + --property resources:MEMORY_MB=0 \ + --property resources:DISK_GB=0 \ + --property resources:CUSTOM_BAREMETAL=1 \ + --property capabilities:boot_mode=uefi + + - name: Copy ironic_nodes.yaml to the openstackclient pod + shell: | + set -xe -o pipefail + oc project openstack + oc cp ~/data/ironic_nodes.yaml openstackclient:ironic_nodes.yaml + + - name: Enroll nodes in ironic + shell: | + set -xe -o pipefail + oc project openstack + oc rsh openstackclient openstack baremetal create ironic_nodes.yaml + + - name: Wait for ironic nodes to get to state - enroll + shell: | + oc project openstack + + counter=0 + max_retries=100 + node_state=enroll + until ! oc rsh openstackclient openstack baremetal node list -f value -c "Provisioning State" | grep -P "^(?!${node_state}).*$"; do + ((counter++)) + if (( counter > max_retries )); then + echo "ERROR: Timeout. Nodes did not reach state: enroll" + exit 1 + fi + echo "Waiting for nodes to reach state enroll" + sleep 10 + done + + - name: Manage ironic nodes + shell: | + set -xe -o pipefail + oc project openstack + + oc rsh openstackclient openstack baremetal node manage ironic0 + oc rsh openstackclient openstack baremetal node manage ironic1 + + - name: Wait for ironic nodes to get to state - manageable + shell: | + oc project openstack + + counter=0 + max_retries=100 + node_state=manageable + until ! oc rsh openstackclient openstack baremetal node list -f value -c "Provisioning State" | grep -P "^(?!${node_state}).*$"; do + ((counter++)) + if (( counter > max_retries )); then + echo "ERROR: Timeout. Nodes did not reach state: manageable" + exit 1 + fi + echo "Waiting for nodes to reach state manageable" + sleep 10 + done + + - name: Power off the ironic nodes + shell: | + set -xe -o pipefail + oc project openstack + + oc rsh openstackclient openstack baremetal node power off ironic0 + oc rsh openstackclient openstack baremetal node power off ironic1 + + - name: Set capabilities boot_mode:uefi for ironic nodes + shell: | + set -xe -o pipefail + oc project openstack + + oc rsh openstackclient openstack baremetal node set --property capabilities='boot_mode:uefi' ironic0 + oc rsh openstackclient openstack baremetal node set --property capabilities='boot_mode:uefi' ironic1 + + - name: Ensure ironic nodes are powered off + shell: | + oc project openstack + + counter=0 + max_retries=100 + power_state="off" + until ! oc rsh openstackclient openstack baremetal node list -f value -c "Power State" | grep -P "^power.(?!${power_state}).*$"; do + ((counter++)) + if (( counter > max_retries )); then + echo "ERROR: Timeout. Nodes did not reach power state: power off" + exit 1 + fi + echo "Waiting for nodes to reach power state off" + sleep 10 + done + + - name: Provide ironic nodes + shell: | + set -xe -o pipefail + oc project openstack + + oc rsh openstackclient openstack baremetal node provide ironic0 + oc rsh openstackclient openstack baremetal node provide ironic1 + + - name: Wait for ironic nodes to get to state - available + shell: | + oc project openstack + + counter=0 + max_retries=100 + node_state=available + while true; do + node_states=$(oc rsh openstackclient openstack baremetal node list -f value -c "Provisioning State") + + # Check if all nodes are in available state + if ! echo "$node_states" | grep -P "^(?!${node_state}).*$" > /dev/null; then + echo "All nodes have reached state: ${node_state}" + break + fi + + # Check for failed states and exit immediately if found + if echo "$node_states" | grep -q "failed"; then + echo "ERROR: One or more nodes are in a failed state:" + oc rsh openstackclient openstack baremetal node list + exit 1 + fi + + ((counter++)) + if (( counter > max_retries )); then + echo "ERROR: Timeout. Nodes did not reach state: available" + exit 1 + fi + + echo "Waiting for nodes to reach state: available" + sleep 10 + done + + - name: Wait for expected compute services (OSPRH-10942) + wait_conditions: + - >- + timeout --foreground 5m hotstack-nova-discover-hosts + --namespace openstack --num-computes 1 + + - name: Run tempest + documentation: >- + Executes Ironic baremetal scenario tests using the Tempest framework. + Tests validate provisioning and managing baremetal servers through OpenStack. + manifest: tempest-tests.yml + wait_conditions: + - >- + oc wait -n openstack tempests.test.openstack.org tempest-tests + --for condition=ServiceConfigReady --timeout=120s + wait_pod_completion: + - namespace: openstack + labels: + operator: test-operator + service: tempest + workflowStep: "0" + timeout: 900 + poll_interval: 15 diff --git a/scenarios/sno-nxsw-netconf/test-operator/manifests/nad.yaml b/scenarios/sno-nxsw-netconf/test-operator/manifests/nad.yaml new file mode 100644 index 00000000..e279d732 --- /dev/null +++ b/scenarios/sno-nxsw-netconf/test-operator/manifests/nad.yaml @@ -0,0 +1,21 @@ +--- +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + name: ironic + namespace: sushy-emulator +spec: + config: | + { + "cniVersion": "0.3.1", + "name": "ironicnet", + "type": "macvlan", + "master": "ironicbr.101", + "mtu": 1442, + "ipam": { + "type": "whereabouts", + "range": "172.20.1.0/24", + "range_start": "172.20.1.71", + "range_end": "172.20.1.75" + } + } diff --git a/scenarios/sno-nxsw-netconf/test-operator/tempest-tests.yml b/scenarios/sno-nxsw-netconf/test-operator/tempest-tests.yml new file mode 100644 index 00000000..6aad02ac --- /dev/null +++ b/scenarios/sno-nxsw-netconf/test-operator/tempest-tests.yml @@ -0,0 +1,37 @@ +--- +apiVersion: test.openstack.org/v1beta1 +kind: Tempest +metadata: + name: tempest-tests + namespace: openstack +spec: + networkAttachments: + - ctlplane + privileged: true + workflow: + - stepName: ironic-scenario-testing + storageClass: lvms-local-storage + tempestconfRun: + create: true + overrides: | + auth.create_isolated_networks false + baremetal.available_nodes 2 + baremetal.max_microversion 1.82 + baremetal.use_provision_network true + compute-feature-enabled.disk_config false + compute-feature-enabled.interface_attach false + compute.fixed_network_name provisioning + compute.flavor_ref 123456789-1234-1234-1234-000000000001 + compute.hypervisor_type ironic + compute.build_timeout 900 + network.shared_physical_network true + service_available.ironic_inspector false + service_available.ironic true + service_available.murano false + validation.connect_method fixed + validation.network_for_ssh provisioning + tempestRun: + concurrency: 4 + includeList: | + ^ironic_tempest_plugin.tests.scenario.test_baremetal_multitenancy.BaremetalMultitenancy.test_baremetal_multitenancy$ + ^ironic_tempest_plugin.tests.scenario.test_baremetal_single_tenant.BaremetalSingleTenant.test_baremetal_single_tenant$ diff --git a/switch-images/.gitignore b/switch-images/.gitignore new file mode 100644 index 00000000..f737e0ee --- /dev/null +++ b/switch-images/.gitignore @@ -0,0 +1,11 @@ +# Built images +switch-host.qcow2 +switch-host.qcow2.gz +switch-host-base.qcow2 + +# Temporary files +*.tmp + +# Firmware downloads (downloaded during build) +firmware/OVMF-edk2-stable202305.fd +firmware/*.zip diff --git a/switch-images/Makefile b/switch-images/Makefile new file mode 100644 index 00000000..88c49e99 --- /dev/null +++ b/switch-images/Makefile @@ -0,0 +1,141 @@ +# Copyright Red Hat, Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +SWITCH_HOST_IMAGE_URL ?= https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-x86_64-9-latest.x86_64.qcow2 +SWITCH_HOST_BASE_IMAGE ?= switch-host-base.qcow2 +SWITCH_HOST_IMAGE_NAME ?= switch-host.qcow2 +SWITCH_HOST_IMAGE_FORMAT ?= raw +SWITCH_HOST_INSTALL_PACKAGES ?= libvirt,qemu-kvm,qemu-img,expect,unzip,jq,iproute,nmap-ncat,telnet,git,vim-enhanced,tmux,bind-utils,bash-completion,nmstate,tcpdump,python3-jinja2 + +# Switch vendor image locations (optional, leave empty if not available) +# These will be copied into the image at /opt// during build +FORCE10_10_IMAGE ?= +FORCE10_9_IMAGE ?= +NXOS_IMAGE ?= +SONIC_IMAGE ?= + +# NXOS-specific UEFI firmware (required for NXOS, copied to /usr/local/share/edk2/ovmf/) +NXOS_UEFI_FIRMWARE_FILE ?= OVMF-edk2-stable202305.fd +NXOS_UEFI_FIRMWARE_URL ?= https://sourceforge.net/projects/gns-3/files/Qemu%20Appliances/$(NXOS_UEFI_FIRMWARE_FILE).zip/download + +all: switch-host + +clean: switch-host_clean + +switch-host: switch-host_download switch-host_firmware switch-host_copy switch-host_customize switch-host_convert + +switch-host_download: + @if [ ! -f $(SWITCH_HOST_BASE_IMAGE) ]; then \ + echo "Downloading base image to $(SWITCH_HOST_BASE_IMAGE)..."; \ + curl -L -o $(SWITCH_HOST_BASE_IMAGE) $(SWITCH_HOST_IMAGE_URL); \ + else \ + echo "Base image $(SWITCH_HOST_BASE_IMAGE) already exists, skipping download."; \ + fi + +switch-host_firmware: +ifneq ($(NXOS_IMAGE),) + @echo "Downloading NXOS UEFI firmware..." + @if [ ! -f firmware/$(NXOS_UEFI_FIRMWARE_FILE) ]; then \ + mkdir -p firmware; \ + curl -L -o firmware/$(NXOS_UEFI_FIRMWARE_FILE).zip \ + $(NXOS_UEFI_FIRMWARE_URL); \ + cd firmware && unzip $(NXOS_UEFI_FIRMWARE_FILE).zip; \ + rm firmware/$(NXOS_UEFI_FIRMWARE_FILE).zip; \ + echo "Firmware downloaded: firmware/$(NXOS_UEFI_FIRMWARE_FILE)"; \ + else \ + echo "Firmware already exists: firmware/$(NXOS_UEFI_FIRMWARE_FILE)"; \ + fi +endif + +switch-host_copy: + @echo "Creating working copy: $(SWITCH_HOST_BASE_IMAGE) -> $(SWITCH_HOST_IMAGE_NAME)" + @cp $(SWITCH_HOST_BASE_IMAGE) $(SWITCH_HOST_IMAGE_NAME) + +switch-host_customize: + @echo "Customizing switch-host image..." +ifneq ($(FORCE10_10_IMAGE),) + @test -f $(FORCE10_10_IMAGE) || (echo "ERROR: Force10 OS10 image not found: $(FORCE10_10_IMAGE)" && exit 1) +endif +ifneq ($(FORCE10_9_IMAGE),) + @test -f $(FORCE10_9_IMAGE) || (echo "ERROR: Force10 OS9 image not found: $(FORCE10_9_IMAGE)" && exit 1) +endif +ifneq ($(NXOS_IMAGE),) + @test -f $(NXOS_IMAGE) || (echo "ERROR: NXOS image not found: $(NXOS_IMAGE)" && exit 1) +endif +ifneq ($(SONIC_IMAGE),) + @test -f $(SONIC_IMAGE) || (echo "ERROR: SONiC image not found: $(SONIC_IMAGE)" && exit 1) +endif + virt-customize -a $(SWITCH_HOST_IMAGE_NAME) \ + --install $(SWITCH_HOST_INSTALL_PACKAGES) \ + --timezone UTC \ + --copy-in runtime-scripts/start-switch-vm.sh:/usr/local/bin \ + --chmod 0755:/usr/local/bin/start-switch-vm.sh \ + --run-command 'mkdir -p /usr/local/lib/hotstack-switch-vm' \ + --copy-in runtime-scripts/common.sh:/usr/local/lib/hotstack-switch-vm \ + --chmod 0644:/usr/local/lib/hotstack-switch-vm/common.sh \ + --run-command 'mkdir -p /etc/hotstack-switch-vm' \ + --run-command 'mkdir -p /var/lib/hotstack-switch-vm' \ + $(if $(FORCE10_10_IMAGE), \ + --copy-in runtime-scripts/force10_10:/usr/local/lib/hotstack-switch-vm \ + --chmod 0755:/usr/local/lib/hotstack-switch-vm/force10_10/setup.sh \ + --chmod 0755:/usr/local/lib/hotstack-switch-vm/force10_10/wait.sh \ + --chmod 0755:/usr/local/lib/hotstack-switch-vm/force10_10/configure.sh \ + --chmod 0644:/usr/local/lib/hotstack-switch-vm/force10_10/utils.sh \ + --chmod 0644:/usr/local/lib/hotstack-switch-vm/force10_10/domain.xml.j2 \ + --chmod 0644:/usr/local/lib/hotstack-switch-vm/force10_10/nmstate.yaml.j2 \ + --run-command 'mkdir -p /opt/force10_10' \ + --copy-in $(FORCE10_10_IMAGE):/opt/force10_10 \ + --run-command 'basename $(FORCE10_10_IMAGE) > /opt/force10_10/image-info.txt',) \ + $(if $(FORCE10_9_IMAGE), \ + --run-command 'mkdir -p /opt/force10_9' \ + --copy-in $(FORCE10_9_IMAGE):/opt/force10_9,) \ + $(if $(NXOS_IMAGE), \ + --copy-in runtime-scripts/nxos:/usr/local/lib/hotstack-switch-vm \ + --chmod 0755:/usr/local/lib/hotstack-switch-vm/nxos/setup.sh \ + --chmod 0755:/usr/local/lib/hotstack-switch-vm/nxos/wait.sh \ + --chmod 0755:/usr/local/lib/hotstack-switch-vm/nxos/configure.sh \ + --chmod 0644:/usr/local/lib/hotstack-switch-vm/nxos/nxos-switch.service.j2 \ + --chmod 0644:/usr/local/lib/hotstack-switch-vm/nxos/nmstate.yaml.j2 \ + --run-command 'mkdir -p /opt/nxos' \ + --copy-in $(NXOS_IMAGE):/opt/nxos \ + --run-command 'mkdir -p /usr/local/share/edk2/ovmf' \ + --copy-in firmware/$(NXOS_UEFI_FIRMWARE_FILE):/usr/local/share/edk2/ovmf \ + --chmod 0644:/usr/local/share/edk2/ovmf/$(NXOS_UEFI_FIRMWARE_FILE) \ + --run-command 'echo "NXOS_IMAGE_FILE=\"$(shell basename $(NXOS_IMAGE))\"" > /opt/nxos/image-config' \ + --run-command 'echo "UEFI_FIRMWARE_FILE=\"$(NXOS_UEFI_FIRMWARE_FILE)\"" >> /opt/nxos/image-config',) \ + $(if $(SONIC_IMAGE), \ + --run-command 'mkdir -p /opt/sonic' \ + --copy-in $(SONIC_IMAGE):/opt/sonic,) \ + --selinux-relabel + @echo "Switch-host image customization complete" + +switch-host_convert: +ifeq ($(SWITCH_HOST_IMAGE_FORMAT),raw) + @echo "Converting switch-host image to raw format (in-place)..." + qemu-img convert -p -f qcow2 -O raw $(SWITCH_HOST_IMAGE_NAME) $(SWITCH_HOST_IMAGE_NAME).tmp + mv $(SWITCH_HOST_IMAGE_NAME).tmp $(SWITCH_HOST_IMAGE_NAME) + @echo "Switch-host image converted to raw format: $(SWITCH_HOST_IMAGE_NAME)" +endif + +switch-host_clean: + rm -f $(SWITCH_HOST_IMAGE_NAME) + rm -f $(SWITCH_HOST_IMAGE_NAME).tmp + rm -f firmware/$(NXOS_UEFI_FIRMWARE_FILE) + rm -f firmware/$(NXOS_UEFI_FIRMWARE_FILE).zip + +switch-host_clean_all: switch-host_clean + rm -f $(SWITCH_HOST_BASE_IMAGE) + rm -f firmware/$(NXOS_UEFI_FIRMWARE_FILE) + rm -f firmware/$(NXOS_UEFI_FIRMWARE_FILE).zip diff --git a/switch-images/README.md b/switch-images/README.md new file mode 100644 index 00000000..48147e5a --- /dev/null +++ b/switch-images/README.md @@ -0,0 +1,350 @@ +# Switch Images + +Build infrastructure for creating virtual switch host images used in hotstack scenarios. These images enable running network switch operating systems as nested VMs inside OpenStack instances. + +## Overview + +The switch-images directory provides a completely separate build system from the main cloud images in `../images/`. Switch images use nested virtualization and require special runtime scripts, UEFI firmware, and switch vendor software. + +## Directory Structure + +``` +switch-images/ +├── Makefile # Switch image build logic +├── README.md # This file +├── runtime-scripts/ # Scripts embedded in the image +│ ├── start-switch-vm.sh # Main entry point (called by cloud-init) +│ ├── common.sh # Shared logging and console helpers +│ ├── nxos/ # NXOS-specific scripts +│ │ ├── setup.sh +│ │ ├── wait.sh +│ │ ├── configure.sh +│ │ ├── nxos-switch.service.j2 +│ │ └── nmstate.yaml.j2 # Macvtap network template +│ ├── force10_10/ # Force10 OS10-specific scripts +│ │ ├── setup.sh +│ │ ├── wait.sh +│ │ ├── configure.sh +│ │ ├── utils.sh # Network bridge helpers +│ │ ├── domain.xml.j2 +│ │ └── nmstate.yaml.j2 # Bridge network template +│ └── README.md # Runtime scripts documentation +└── firmware/ # Build artifacts (not committed) + └── OVMF-edk2-stable202305.fd # NXOS UEFI firmware (downloaded when building with NXOS_IMAGE) +``` + +## Building Switch Images + +### Prerequisites + +- CentOS 9 Stream (or similar) build host +- `libguestfs-tools-c` package installed (`virt-customize`) +- `qemu-img` for image conversion +- `curl` and `unzip` for firmware download +- Switch vendor software images (optional, see below) + +### Quick Start + +Build a basic switch-host image without vendor software: + +```bash +cd switch-images +make switch-host +``` + +This creates `switch-host.qcow2` (converted to raw format by default). + +### Building with Switch Vendor Images + +To include switch operating system images in the build: + +#### Force10 OS10 + +```bash +# Download Force10 OS10 virtualization image +# (requires Dell support account) +make switch-host FORCE10_10_IMAGE=/path/to/OS10_Virtualization*.zip +``` + +#### Cisco NXOS + +```bash +# Download Cisco NXOS qcow2 image +# (requires Cisco DevNet account or support portal access) +make switch-host NXOS_IMAGE=/path/to/nexus9300v.*.qcow2 +``` + +#### Multiple Switch Models + +```bash +# Build with multiple switch types +make switch-host \ + FORCE10_10_IMAGE=/path/to/OS10_Virtualization*.zip \ + NXOS_IMAGE=/path/to/nexus9300v.*.qcow2 +``` + +### Build Variables + +Control the build process with make variables: + +```bash +# Base image +SWITCH_HOST_IMAGE_URL=https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-x86_64-9-latest.x86_64.qcow2 +SWITCH_HOST_BASE_IMAGE=switch-host-base.qcow2 # Cached download +SWITCH_HOST_IMAGE_NAME=switch-host.qcow2 # Output image + +# Format (raw or qcow2) +SWITCH_HOST_IMAGE_FORMAT=raw + +# Additional packages to install +SWITCH_HOST_INSTALL_PACKAGES=libvirt,qemu-kvm,qemu-img,expect,unzip,jq,iproute,nmap-ncat,telnet,git,vim-enhanced,tmux,bind-utils,bash-completion,nmstate,tcpdump,python3-jinja2 + +# Switch vendor images (optional) +FORCE10_10_IMAGE=/path/to/image.zip +FORCE10_9_IMAGE=/path/to/image +NXOS_IMAGE=/path/to/image.qcow2 +SONIC_IMAGE=/path/to/image + +# NXOS-specific UEFI firmware (automatically downloaded when NXOS_IMAGE is set) +NXOS_UEFI_FIRMWARE_FILE=OVMF-edk2-stable202305.fd # Firmware filename +NXOS_UEFI_FIRMWARE_URL=https://sourceforge.net/projects/gns-3/files/Qemu%20Appliances/$(NXOS_UEFI_FIRMWARE_FILE).zip/download +``` + +### Build Targets + +- `make switch-host` - Build complete switch-host image +- `make switch-host_download` - Download base image only +- `make switch-host_firmware` - Download NXOS UEFI firmware (if NXOS_IMAGE is set) +- `make switch-host_clean` - Remove build artifacts +- `make switch-host_clean_all` - Remove everything including cached base image + +## What Gets Built + +The build process creates a CentOS 9 Stream image with: + +### Installed Packages + +- **Virtualization**: libvirt, qemu-kvm, qemu-img +- **Networking**: nmstate, iproute, tcpdump +- **Utilities**: expect, unzip, jq, nmap-ncat, telnet, git, vim, tmux, bash-completion +- **Python**: python3-jinja2 (for templating) + +### Runtime Scripts (embedded at build time) + +Scripts are copied into the image at specific locations: + +``` +/usr/local/bin/ +└── start-switch-vm.sh # Main entry point + +/usr/local/lib/hotstack-switch-vm/ +├── common.sh # Shared logging and console helpers +├── force10_10/ # Model-specific scripts +│ ├── setup.sh +│ ├── wait.sh +│ ├── configure.sh +│ ├── utils.sh # Network bridge helpers +│ ├── domain.xml.j2 +│ └── nmstate.yaml.j2 # Bridge configuration template +└── nxos/ + ├── setup.sh + ├── wait.sh + ├── configure.sh + ├── nxos-switch.service.j2 + └── nmstate.yaml.j2 # Macvtap configuration template +``` + +### UEFI Firmware (NXOS Only) + +When building with `NXOS_IMAGE`, UEFI firmware is automatically downloaded and included: + +``` +/usr/local/share/edk2/ovmf/ +└── OVMF-edk2-stable202305.fd # GNS3 "switch friendly" firmware (default) +``` + +To use a different firmware version: +```bash +make switch-host NXOS_IMAGE=/path/to/nexus.qcow2 NXOS_UEFI_FIRMWARE_FILE=OVMF-newer.fd +``` + +### Switch Vendor Software (if provided) + +When vendor images are provided via build variables, they are embedded in the image: + +``` +/opt/force10_10/ # If FORCE10_10_IMAGE provided +├── OS10_Virtualization*.zip +└── image-info.txt + +/opt/nxos/ # If NXOS_IMAGE provided +├── nexus9300v.*.qcow2 +└── image-info.txt + +/opt/sonic/ # If SONIC_IMAGE provided +└── sonic-vs.img +``` + +## Runtime Behavior + +Once deployed, the switch-host image: + +1. Boots as an OpenStack instance with multiple network ports +2. Receives configuration via cloud-init (see scenario heat templates) +3. Executes `/usr/local/bin/start-switch-vm.sh` via cloud-init's `runcmd` +4. Sets up network bridges (Force10) or direct passthrough (NXOS) +5. Extracts/prepares switch disk images from `/opt//` +6. Launches nested switch VM using libvirt +7. Waits for switch to boot (400-500 seconds for Force10, 10-20 minutes for NXOS) +8. Configures the switch (Force10: console, NXOS: POAP) +9. Writes status to `/var/lib/hotstack-switch-vm/status` + +See `runtime-scripts/README.md` for detailed documentation on how the runtime scripts work. + +## Obtaining Switch Vendor Images + +### Cisco NXOS (Nexus 9000v) + +1. Create a Cisco account at https://devnetsandbox.cisco.com/ or https://software.cisco.com/ +2. Navigate to "Downloads" → "Switches" → "Nexus 9000 Series Switches" +3. Search for "Nexus 9000/3000 Virtual Switch for KVM" +4. Download the `.qcow2` image (e.g., `nexus9300v64.10.3.1.F.qcow2`) +5. Use the downloaded file with `NXOS_IMAGE=/path/to/nexus9300v*.qcow2` + +### Force10 OS10 + +1. Create a Dell support account at https://www.dell.com/support/ +2. Navigate to "Networking" → "Force10" → "OS10" +3. Search for "OS10 Virtualization Image" +4. Download the `.zip` archive (e.g., `OS10_Virtualization_10.5.6.0.zip`) +5. Use the downloaded file with `FORCE10_10_IMAGE=/path/to/OS10_Virtualization*.zip` + +### Dell SONiC + +SONiC images are available from the community at https://sonic-net.github.io/SONiC/ + +## Upload to OpenStack + +After building, upload the image to your OpenStack environment: + +```bash +# Convert to raw format if needed (already done if SWITCH_HOST_IMAGE_FORMAT=raw) +# qemu-img convert -f qcow2 -O raw switch-host.qcow2 switch-host.raw + +# Upload to Glance +openstack image create \ + --disk-format raw \ + --container-format bare \ + --file switch-host.qcow2 \ + --property hw_disk_bus=scsi \ + --property hw_scsi_model=virtio-scsi \ + --property hw_vif_multiqueue_enabled=true \ + --property hw_qemu_guest_agent=yes \ + hotstack-switch-host +``` + +## Usage in Scenarios + +Reference the image in scenario heat templates: + +```yaml +parameters: + switch_host_image: + type: string + default: hotstack-switch-host + +resources: + switch_host_instance: + type: OS::Nova::Server + properties: + image: { get_param: switch_host_image } + flavor: { get_param: switch_host_flavor } + networks: + - port: { get_resource: switch_mgmt_port } + - port: { get_resource: switch_trunk_port } + # ... more ports + user_data_format: RAW + user_data: + str_replace: + template: | + #cloud-config + write_files: + - path: /etc/hotstack-switch-vm/config + content: | + SWITCH_MODEL=nxos + MGMT_INTERFACE=eth0 + SWITCH_MGMT_INTERFACE=eth1 + TRUNK_INTERFACE=eth2 + BM_INTERFACE_START=eth3 + BM_INTERFACE_COUNT=8 + SWITCH_MGMT_IP=172.24.5.20/24 + runcmd: + - /usr/local/bin/start-switch-vm.sh +``` + +See `../scenarios/sno-nxsw/` or `../scenarios/sno-2nics-force10-10/` for complete examples. + +## Troubleshooting + +### Build Failures + +**Problem**: `virt-customize` fails with permission errors + +**Solution**: Ensure you have proper permissions and libvirt is running: +```bash +sudo systemctl start libvirtd +sudo usermod -a -G libvirt $USER +``` + +**Problem**: Firmware download fails + +**Solution**: Download manually from https://sourceforge.net/projects/gns-3/files/Qemu%20Appliances/ + +### Runtime Issues + +**Problem**: Switch VM doesn't start + +**Solution**: Check logs in the OpenStack instance: +```bash +# SSH to the switch-host instance +tail -f /var/log/cloud-init-output.log +cat /var/lib/hotstack-switch-vm/status +virsh list --all +``` + +**Problem**: Switch not responding after boot + +**Solution**: Connect to the switch serial console: +```bash +telnet localhost 55001 +``` + +For more runtime troubleshooting, see `runtime-scripts/README.md`. + +## Differences from `../images/` + +| Feature | `images/` | `switch-images/` | +|---------|-----------|------------------| +| **Purpose** | Basic cloud images | Complex nested virtualization images | +| **Build complexity** | Simple (download + customize) | Complex (firmware, vendor images, runtime scripts) | +| **Runtime** | Direct cloud-init | Nested VM orchestration | +| **Examples** | controller, blank, nat64 | switch-host (with Force10, NXOS, SONiC) | +| **Dependencies** | Minimal packages | Full KVM/libvirt stack | +| **Size** | ~1-2GB | ~5-10GB (with vendor images) | + +## Contributing + +When adding support for new switch models: + +1. Create a new directory under `runtime-scripts//` +2. Add `setup.sh`, `wait.sh`, `configure.sh`, and `domain.xml.j2` +3. Update the Makefile to handle the new model's vendor image +4. Document the model in `runtime-scripts/README.md` +5. Create an example scenario in `../scenarios/` + +## See Also + +- `runtime-scripts/README.md` - Detailed runtime scripts documentation +- `../scenarios/sno-nxsw-netconf/README.md` - NXOS scenario example +- `../scenarios/sno-2nics-force10-10/README.md` - Force10 scenario example +- `../images/README.md` - Basic cloud images documentation diff --git a/images/switch-host-scripts/README.md b/switch-images/runtime-scripts/README.md similarity index 71% rename from images/switch-host-scripts/README.md rename to switch-images/runtime-scripts/README.md index 53ec8380..749a9f04 100644 --- a/images/switch-host-scripts/README.md +++ b/switch-images/runtime-scripts/README.md @@ -51,6 +51,9 @@ write_files: TRUNK_INTERFACE=eth2 # Switch trunk (bridged) BM_INTERFACE_START=eth3 # Baremetal ports start (bridged) BM_INTERFACE_COUNT=8 + SWITCH_MGMT_MAC=52:54:00:xx:xx:xx # MAC address from Neutron port + TRUNK_MAC=52:54:00:yy:yy:yy # MAC address from Neutron port + BM_MACS=(52:54:00:aa:aa:aa 52:54:00:bb:bb:bb ...) # MAC addresses array SWITCH_MGMT_IP=172.24.5.20/24 runcmd: @@ -67,12 +70,13 @@ runcmd: 6. Calls configuration script to configure the switch ### 4. Model-Specific Setup -Model scripts (e.g., `force10_10/setup.sh`): -1. Extract/prepare switch disk images from `/opt/force10_10/` -2. Convert VMDK images to qcow2 format for KVM compatibility -3. Create Linux bridges using nmstate for each switch port -4. Render libvirt domain XML from Jinja2 template -5. Define and start VM using libvirt (virsh) +Model scripts (e.g., `force10_10/setup.sh`, `nxos/setup.sh`): +1. Extract/prepare switch disk images from `/opt//` +2. Convert VMDK images to qcow2 format for KVM compatibility (if needed) +3. For Force10: Create Linux bridges using nmstate for each switch port +4. For NXOS: Extract MAC addresses and interface names for direct passthrough mode +5. Render libvirt domain XML from Jinja2 template +6. Define and start VM using libvirt (virsh) ### 5. Wait for Switch Boot Wait scripts (e.g., `force10_10/wait.sh`): @@ -110,18 +114,20 @@ The startup script writes status information to `/var/lib/hotstack-switch-vm/sta └── start-switch-vm.sh # Main entry point (called by cloud-init) /usr/local/lib/hotstack-switch-vm/ -├── common.sh # Shared functions library -├── bridges.nmstate.yaml.j2 # Jinja2 template for bridge config +├── common.sh # Shared logging and console helpers ├── force10_10/ # Model-specific directory │ ├── setup.sh # Setup and start VM │ ├── wait.sh # Wait for switch to boot │ ├── configure.sh # Initial configuration -│ └── domain.xml.j2 # Libvirt domain XML template -└── nxos/ # Cisco NXOS directory +│ ├── utils.sh # Network bridge helpers +│ ├── domain.xml.j2 # Libvirt domain XML template +│ └── nmstate.yaml.j2 # Network bridge template +└── nxos/ # NXOS directory ├── setup.sh # Setup and start VM ├── wait.sh # Wait for switch to boot ├── configure.sh # No-op (POAP handles config) - └── domain.xml.j2 # Libvirt domain XML template + ├── nxos-switch.service.j2 # Systemd service template + └── nmstate.yaml.j2 # Macvtap network template /etc/hotstack-switch-vm/ └── config # Configuration file (from cloud-init) @@ -141,7 +147,10 @@ The startup script writes status information to `/var/lib/hotstack-switch-vm/sta /opt/nxos/ # Pre-installed NXOS images ├── *.qcow2 # Cisco NXOS qcow2 image -└── image-info.txt # Original filename metadata +└── image-config # Build-time metadata (sourceable shell variables) + +/usr/local/share/edk2/ovmf/ # UEFI firmware (NXOS only) +└── OVMF-edk2-stable202305.fd # GNS3 "switch friendly" UEFI firmware (default) ``` ## Configuration @@ -221,7 +230,7 @@ exit_code=1 ## Common Functions -The `common.sh` library provides: +The `common.sh` library provides shared functions used by all switch models: **`log `** - Logs timestamped messages to stderr @@ -229,6 +238,21 @@ The `common.sh` library provides: **`die `** - Logs error and exits with status 1 +**`send_switch_config `** +- Sends a command to switch console via telnet +- Filters non-printable characters from output +- Returns command output + +**`wait_for_switch_prompt [use_enable]`** +- Waits for switch to boot and respond with expected prompt +- Sends carriage returns to trigger prompt +- Polls telnet console with configurable retry logic +- Returns 0 on success, 1 on timeout + +## Force10-Specific Functions + +The `force10_10/utils.sh` library provides Force10 OS10-specific network bridge functions: + **`build_bridge_config `** - Validates network interfaces exist - Builds JSON array of bridge configurations @@ -239,16 +263,64 @@ The `common.sh` library provides: - Applies configuration atomically using nmstatectl - Creates all bridges in a single operation -**`send_switch_config `** -- Sends a command to switch console via telnet -- Filters non-printable characters from output -- Returns command output +## Network Interface Modes -**`wait_for_switch_prompt [use_enable]`** -- Waits for switch to boot and respond with expected prompt -- Sends carriage returns to trigger prompt -- Polls telnet console with configurable retry logic -- Returns 0 on success, 1 on timeout +Different switch models use different approaches for connecting the nested VM to OpenStack networks: + +### Force10 OS10 - Linux Bridge Mode + +Uses traditional Linux bridges created with nmstate: + +- **How it works**: Creates `sw-br0`, `sw-br1`, etc. bridges that connect host interfaces to VM interfaces +- **Configuration**: Bridges configured via nmstate YAML templates +- **MAC addresses**: VM generates its own MAC addresses (libvirt defaults) +- **Use case**: Works well when switch doesn't need to directly respond to OpenStack DHCP + +### Cisco NXOS - Direct Passthrough Mode + +Uses direct passthrough mode for exclusive interface access: + +- **How it works**: VM gets exclusive access to host interfaces using `type='direct'` mode='passthrough' +- **MAC addresses**: VM inherits/uses the host interface MAC addresses (from OpenStack ports) +- **Benefits**: + - **No MAC conflicts**: Eliminates bridge MAC address collision warnings + - Transparent DHCP: OpenStack's DHCP server sees requests from the correct MAC + - Best performance (direct hardware access) + - Simpler setup (no bridge creation needed) +- **Trade-off**: Host loses access to those interfaces while VM is running +- **Use case**: Required for POAP which needs DHCP responses from OpenStack + +**Key difference**: With passthrough mode, the VM has exclusive access to the network interfaces. When the nested NXOS switch sends DHCP with the host interface's MAC (e.g., `22:57:f8:dd:fe:08`), OpenStack recognizes it as the `switch-switch-mgmt-port` and responds with POAP configuration options. No bridges means no MAC address conflicts. + +## UEFI Firmware (NXOS Only) + +When building with `NXOS_IMAGE`, a "switch friendly" UEFI firmware from the GNS3 project +is automatically included: + +- **Firmware**: `OVMF-edk2-stable202305.fd` (default) +- **Source**: https://sourceforge.net/projects/gns-3/files/Qemu%20Appliances/ +- **Location**: `/usr/local/share/edk2/ovmf/OVMF-edk2-stable202305.fd` +- **Purpose**: Provides better compatibility with Cisco NXOS virtual switches + +This firmware is automatically downloaded during the `make switch-host` build process +when `NXOS_IMAGE` is set, and is used by the NXOS domain.xml.j2 template. The firmware +resolves NIC initialization issues that can occur with the standard OVMF firmware when +running NXOS virtual switches. + +Build-time metadata (image filename and firmware filename) is stored in `/opt/nxos/image-config` +as sourceable shell variables during the image build process, ensuring the runtime scripts +use the same firmware version that was embedded in the image. + +**Example `/opt/nxos/image-config`:** +```bash +NXOS_IMAGE_FILE="nexus9300v.10.3.7.M.qcow2" +UEFI_FIRMWARE_FILE="OVMF-edk2-stable202305.fd" +``` + +To use a different UEFI firmware version during build: +```bash +make NXOS_IMAGE=/path/to/nexus.qcow2 NXOS_UEFI_FIRMWARE_FILE=OVMF-newer.fd +``` ## Supported Switch Models diff --git a/switch-images/runtime-scripts/common.sh b/switch-images/runtime-scripts/common.sh new file mode 100755 index 00000000..c6d23e9c --- /dev/null +++ b/switch-images/runtime-scripts/common.sh @@ -0,0 +1,85 @@ +#!/bin/bash +# Copyright Red Hat, Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +# +# Common functions for virtual switch management +# Provides logging and console helpers shared across all switch models +# Source this file in your scripts: source "$LIB_DIR/common.sh" + +# Logging functions +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >&2 +} + +die() { + log "ERROR: $*" + exit 1 +} + +# Send a command to switch console via telnet +# Usage: send_switch_config +send_switch_config() { + local host="$1" + local port="$2" + local cmd="$3" + local delay="${SWITCH_CMD_DELAY:-1}" + + log "Send command ($host:$port): $cmd" + + # Send command to switch and strip non-ASCII characters + echo "$cmd" | nc -w1 "$host" "$port" 2>/dev/null | strings + + # Brief sleep to allow command execution + sleep "$delay" +} + +# Wait for switch to boot and respond with expected prompt +# Usage: wait_for_switch_prompt [use_enable] +wait_for_switch_prompt() { + local host="$1" + local port="$2" + local sleep_first="$3" + local max_attempts="$4" + local expected_string="$5" + local use_enable="${6:-False}" + + log "Waiting for $sleep_first seconds before polling the switch on $host:$port" + sleep "$sleep_first" + + for attempt in $(seq 1 "$max_attempts"); do + log "Attempt $attempt/$max_attempts: Checking for prompt..." + + # Connect, send input, then wait for response (keep connection open) + local output + if [ "$use_enable" != "False" ]; then + # Send carriage returns then 'en' command, keep connection open to read response + output=$( (printf "\r\n\r\nen\r\n"; sleep 3) | nc "$host" "$port" 2>/dev/null | tr -cd '\11\12\15\40-\176') + else + # Send carriage returns to trigger prompt, keep connection open to read response + output=$( (printf "\r\n\r\n"; sleep 3) | nc "$host" "$port" 2>/dev/null | tr -cd '\11\12\15\40-\176') + fi + + if echo "$output" | grep -q "$expected_string"; then + log "Got switch prompt - Switch ready for configuration." + return 0 + fi + + log "Switch not online yet, waiting..." + sleep 10 + done + + log "ERROR: Switch did not respond with expected prompt after $max_attempts attempts" + return 1 +} diff --git a/images/switch-host-scripts/force10_10/configure.sh b/switch-images/runtime-scripts/force10_10/configure.sh similarity index 81% rename from images/switch-host-scripts/force10_10/configure.sh rename to switch-images/runtime-scripts/force10_10/configure.sh index af8f66c3..f9fbeec7 100755 --- a/images/switch-host-scripts/force10_10/configure.sh +++ b/switch-images/runtime-scripts/force10_10/configure.sh @@ -1,4 +1,19 @@ #!/bin/bash +# Copyright Red Hat, Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +# # Configure Force10 OS10 switch after boot # Based on ironic devstack Force10 OS10 configuration @@ -9,6 +24,8 @@ LIB_DIR="${LIB_DIR:-/usr/local/lib/hotstack-switch-vm}" # Source common functions # shellcheck disable=SC1091 source "$LIB_DIR/common.sh" +# shellcheck disable=SC1091 +source "$LIB_DIR/force10_10/utils.sh" # Load configuration if [ -f /etc/hotstack-switch-vm/config ]; then diff --git a/images/switch-host-scripts/force10_10/domain.xml.j2 b/switch-images/runtime-scripts/force10_10/domain.xml.j2 similarity index 76% rename from images/switch-host-scripts/force10_10/domain.xml.j2 rename to switch-images/runtime-scripts/force10_10/domain.xml.j2 index 8d7cab85..cc6e3416 100644 --- a/images/switch-host-scripts/force10_10/domain.xml.j2 +++ b/switch-images/runtime-scripts/force10_10/domain.xml.j2 @@ -1,3 +1,19 @@ + {{ vm_name }} 4 diff --git a/switch-images/runtime-scripts/force10_10/nmstate.yaml.j2 b/switch-images/runtime-scripts/force10_10/nmstate.yaml.j2 new file mode 100644 index 00000000..97ca4049 --- /dev/null +++ b/switch-images/runtime-scripts/force10_10/nmstate.yaml.j2 @@ -0,0 +1,36 @@ +--- +# Copyright Red Hat, Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +interfaces: +{% for bridge in bridges %} + - name: {{ bridge.name }} + type: linux-bridge + state: up + ipv4: + enabled: false + ipv6: + enabled: false + bridge: + port: + - name: {{ bridge.port }} + - name: {{ bridge.port }} + type: ethernet + state: up + ipv4: + enabled: false + ipv6: + enabled: false +{% endfor %} diff --git a/images/switch-host-scripts/force10_10/setup.sh b/switch-images/runtime-scripts/force10_10/setup.sh similarity index 91% rename from images/switch-host-scripts/force10_10/setup.sh rename to switch-images/runtime-scripts/force10_10/setup.sh index 99ec01c0..2463b631 100755 --- a/images/switch-host-scripts/force10_10/setup.sh +++ b/switch-images/runtime-scripts/force10_10/setup.sh @@ -1,4 +1,19 @@ #!/bin/bash +# Copyright Red Hat, Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +# # Setup and start Force10 OS10 virtual switch using libvirt # Based on ironic devstack create_network_simulator_vm_force10_10 @@ -10,6 +25,8 @@ LIB_DIR="${LIB_DIR:-/usr/local/lib/hotstack-switch-vm}" # Source common functions # shellcheck disable=SC1091 source "$LIB_DIR/common.sh" +# shellcheck disable=SC1091 +source "$SCRIPT_DIR/utils.sh" WORK_DIR="${WORK_DIR:-/var/lib/hotstack-switch-vm}" CONSOLE_PORT="${CONSOLE_PORT:-55001}" diff --git a/images/switch-host-scripts/common.sh b/switch-images/runtime-scripts/force10_10/utils.sh old mode 100755 new mode 100644 similarity index 56% rename from images/switch-host-scripts/common.sh rename to switch-images/runtime-scripts/force10_10/utils.sh index ce3de831..fb3a7380 --- a/images/switch-host-scripts/common.sh +++ b/switch-images/runtime-scripts/force10_10/utils.sh @@ -1,16 +1,22 @@ #!/bin/bash -# Common functions for virtual switch management -# Source this file in your scripts: source "$(dirname "${BASH_SOURCE[0]}")/common.sh" - -# Logging functions -log() { - echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >&2 -} - -die() { - log "ERROR: $*" - exit 1 -} +# Copyright Red Hat, Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +# +# Force10 OS10 utility functions +# Provides network bridge management +# Source this file after common.sh # Build bridge configuration JSON from interface list # Usage: build_bridge_config @@ -61,12 +67,13 @@ build_bridge_config() { } # Create multiple Linux bridges using nmstate (single atomic operation) -# Usage: create_bridges +# Usage: create_bridges [template_path] [output_file] # Where bridges_json is a JSON array like: [{"name":"sw-br0","port":"eth0"},{"name":"sw-br1","port":"eth1"}] +# template_path defaults to model-specific nmstate.yaml.j2 in caller's directory create_bridges() { local bridges_json="$1" - local template_dir="${LIB_DIR:-/usr/local/lib/hotstack-switch-vm}" - local output_file="${2:-/tmp/bridges-nmstate.yaml}" + local template_path="${2:-$SCRIPT_DIR/nmstate.yaml.j2}" + local output_file="${3:-/tmp/bridges-nmstate.yaml}" log "Creating bridges using nmstate" @@ -76,7 +83,7 @@ import json from jinja2 import Template # Load template -with open("$template_dir/bridges.nmstate.yaml.j2", "r") as f: +with open("$template_path", "r") as f: template = Template(f.read()) # Parse bridges configuration @@ -94,7 +101,7 @@ with open("$output_file", "w") as f: print(f"Rendered nmstate config for {len(bridges)} bridges") EOF then - die "Failed to render nmstate template from $template_dir/bridges.nmstate.yaml.j2" + die "Failed to render nmstate template from $template_path" fi # Apply configuration with nmstate @@ -107,59 +114,3 @@ EOF rm -f "$output_file" return 0 } - -# Send a command to switch console via telnet -# Usage: send_switch_config -send_switch_config() { - local host="$1" - local port="$2" - local cmd="$3" - local delay="${SWITCH_CMD_DELAY:-1}" - - log "Send command ($host:$port): $cmd" - - # Send command to switch and strip non-ASCII characters - echo "$cmd" | nc -w1 "$host" "$port" 2>/dev/null | strings - - # Brief sleep to allow command execution - sleep "$delay" -} - -# Wait for switch to boot and respond with expected prompt -# Usage: wait_for_switch_prompt [use_enable] -wait_for_switch_prompt() { - local host="$1" - local port="$2" - local sleep_first="$3" - local max_attempts="$4" - local expected_string="$5" - local use_enable="${6:-False}" - - log "Waiting for $sleep_first seconds before polling the switch on $host:$port" - sleep "$sleep_first" - - for attempt in $(seq 1 "$max_attempts"); do - log "Attempt $attempt/$max_attempts: Checking for prompt..." - - # Connect, send input, then wait for response (keep connection open) - local output - if [ "$use_enable" != "False" ]; then - # Send carriage returns then 'en' command, keep connection open to read response - output=$( (printf "\r\n\r\nen\r\n"; sleep 3) | nc "$host" "$port" 2>/dev/null | tr -cd '\11\12\15\40-\176') - else - # Send carriage returns to trigger prompt, keep connection open to read response - output=$( (printf "\r\n\r\n"; sleep 3) | nc "$host" "$port" 2>/dev/null | tr -cd '\11\12\15\40-\176') - fi - - if echo "$output" | grep -q "$expected_string"; then - log "Got switch prompt - Switch ready for configuration." - return 0 - fi - - log "Switch not online yet, waiting..." - sleep 10 - done - - log "ERROR: Switch did not respond with expected prompt after $max_attempts attempts" - return 1 -} diff --git a/images/switch-host-scripts/force10_10/wait.sh b/switch-images/runtime-scripts/force10_10/wait.sh similarity index 59% rename from images/switch-host-scripts/force10_10/wait.sh rename to switch-images/runtime-scripts/force10_10/wait.sh index f5b06808..e8562009 100644 --- a/images/switch-host-scripts/force10_10/wait.sh +++ b/switch-images/runtime-scripts/force10_10/wait.sh @@ -1,4 +1,19 @@ #!/bin/bash +# Copyright Red Hat, Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +# # Wait for Force10 OS10 switch to boot and be ready # This script waits for the switch prompt to appear on the serial console @@ -9,6 +24,8 @@ LIB_DIR="${LIB_DIR:-/usr/local/lib/hotstack-switch-vm}" # Source common functions # shellcheck disable=SC1091 source "$LIB_DIR/common.sh" +# shellcheck disable=SC1091 +source "$LIB_DIR/force10_10/utils.sh" # Load configuration if [ -f /etc/hotstack-switch-vm/config ]; then diff --git a/images/switch-host-scripts/nxos/configure.sh b/switch-images/runtime-scripts/nxos/configure.sh similarity index 55% rename from images/switch-host-scripts/nxos/configure.sh rename to switch-images/runtime-scripts/nxos/configure.sh index 465a069b..0f342660 100644 --- a/images/switch-host-scripts/nxos/configure.sh +++ b/switch-images/runtime-scripts/nxos/configure.sh @@ -1,11 +1,25 @@ #!/bin/bash -# Configure Cisco NXOS switch +# Copyright Red Hat, Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +# +# Configure NXOS switch # For POAP-enabled switches, this is a no-op as POAP handles configuration # This script exists to satisfy the start-switch-vm.sh workflow set -euo pipefail -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" LIB_DIR="${LIB_DIR:-/usr/local/lib/hotstack-switch-vm}" # Source common functions diff --git a/switch-images/runtime-scripts/nxos/nmstate.yaml.j2 b/switch-images/runtime-scripts/nxos/nmstate.yaml.j2 new file mode 100644 index 00000000..0501d18b --- /dev/null +++ b/switch-images/runtime-scripts/nxos/nmstate.yaml.j2 @@ -0,0 +1,25 @@ +--- +# Copyright Red Hat, Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +interfaces: +{% for i in range(num_data_interfaces) %} + - name: macvtap{{ i }} + type: mac-vtap + state: up + mac-vtap: + base-iface: {{ data_interface_devs[i] }} + mode: passthru +{% endfor %} diff --git a/switch-images/runtime-scripts/nxos/nxos-switch.service.j2 b/switch-images/runtime-scripts/nxos/nxos-switch.service.j2 new file mode 100644 index 00000000..98e06222 --- /dev/null +++ b/switch-images/runtime-scripts/nxos/nxos-switch.service.j2 @@ -0,0 +1,51 @@ +# Copyright Red Hat, Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +[Unit] +Description=NXOS Virtual Switch +After=network-online.target + +[Service] +Type=simple +Restart=on-failure +RestartSec=10 + +# Launch QEMU with macvtap passthrough +ExecStart=/bin/bash -c '\ +QEMU_CMD="/usr/libexec/qemu-kvm \ + -enable-kvm \ + -cpu host \ + -machine q35 \ + -smbios type=0,uefi=on \ + -drive if=pflash,format=raw,readonly=on,file={{ uefi_firmware }} \ + -drive file={{ nxos_disk }},id=disk0,format=qcow2,if=none \ + -device ahci,id=ahci \ + -device ide-hd,drive=disk0,bus=ahci.0 \ + -m 8096M \ + -smp cpus=2 \ + -display none \ + -serial telnet:localhost:{{ console_port }},server,nowait"; \ +{% for i in range(num_data_interfaces) %}\ +TAP{{ i }}=$(cat /sys/class/net/macvtap{{ i }}/ifindex); \ +QEMU_CMD="$QEMU_CMD -netdev tap,fd={{ i + 3 }},id=hostnet{{ i }} {{ i + 3 }}<>/dev/tap$TAP{{ i }}"; \ +QEMU_CMD="$QEMU_CMD -device e1000,netdev=hostnet{{ i }},id=net{{ i }},mac={{ data_interface_macs[i] }}"; \ +{% endfor %}\ +exec $QEMU_CMD' + +# PID file for tracking +PIDFile={{ work_dir }}/nxos-switch.pid + +[Install] +WantedBy=multi-user.target diff --git a/switch-images/runtime-scripts/nxos/setup.sh b/switch-images/runtime-scripts/nxos/setup.sh new file mode 100644 index 00000000..3b477337 --- /dev/null +++ b/switch-images/runtime-scripts/nxos/setup.sh @@ -0,0 +1,205 @@ +#!/bin/bash +# Copyright Red Hat, Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +# +# Setup and start NXOS virtual switch using direct QEMU +# NXOS uses POAP (Power-On Auto Provisioning) for automatic configuration + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +LIB_DIR="${LIB_DIR:-/usr/local/lib/hotstack-switch-vm}" + +# Source common functions +# shellcheck disable=SC1091 +source "$LIB_DIR/common.sh" + +WORK_DIR="${WORK_DIR:-/var/lib/hotstack-switch-vm}" +CONSOLE_PORT="${CONSOLE_PORT:-55001}" +VM_NAME="${VM_NAME:-cisco-nxos}" + +# Load build-time metadata +# shellcheck source=/dev/null +source /opt/nxos/image-config + +UEFI_FIRMWARE="/usr/local/share/edk2/ovmf/$UEFI_FIRMWARE_FILE" +BASE_IMAGE="/opt/nxos/$NXOS_IMAGE_FILE" + +log "Using NXOS image: $BASE_IMAGE" + +# Load configuration (already validated by start-switch-vm.sh) +# shellcheck source=/dev/null +source /etc/hotstack-switch-vm/config + +# Configuration variables (all with defaults for standalone use) +# Values from /etc/hotstack-switch-vm/config override defaults +MGMT_INTERFACE="${MGMT_INTERFACE:-eth0}" # VM management (unbridged) +SWITCH_MGMT_INTERFACE="${SWITCH_MGMT_INTERFACE:-eth1}" # Switch management port (bridge) +SWITCH_MGMT_MAC="${SWITCH_MGMT_MAC:-52:54:00:00:01:01}" # Management interface MAC +TRUNK_INTERFACE="${TRUNK_INTERFACE:-eth2}" # Switch trunk port (passthrough) +TRUNK_MAC="${TRUNK_MAC:-52:54:00:00:01:02}" # Trunk interface MAC +BM_INTERFACE_START="${BM_INTERFACE_START:-eth3}" # Baremetal ports start (passthrough) +BM_INTERFACE_COUNT="${BM_INTERFACE_COUNT:-8}" # Number of baremetal ports + +# BM_MACS array - use from config or generate defaults +if [ "${#BM_MACS[@]}" -eq 0 ]; then + for ((i=0; i mgmt0" +log " Interface 1: ${DATA_INTERFACE_DEVS[1]} (MAC: ${DATA_INTERFACE_MACS[1]}) -> Ethernet1/1" +for ((i=2; i Ethernet1/$i" +done + +# Ensure work directory exists +mkdir -p "$WORK_DIR" + +# Save interface arrays to JSON for use by Python scripts +printf '%s\n' "${DATA_INTERFACE_MACS[@]}" | jq -R . | jq -s . > "$WORK_DIR/data_interface_macs.json" +printf '%s\n' "${DATA_INTERFACE_DEVS[@]}" | jq -R . | jq -s . > "$WORK_DIR/data_interface_devs.json" + +log "Setting up macvtap interfaces for NXOS switch" + +# Render nmstate template for macvtap interfaces +if ! python3 << EOF +from jinja2 import Template +import json + +# Load data interface arrays from JSON files +with open("$WORK_DIR/data_interface_macs.json", "r") as f: + data_interface_macs = json.load(f) + +with open("$WORK_DIR/data_interface_devs.json", "r") as f: + data_interface_devs = json.load(f) + +# Template variables +context = { + "num_data_interfaces": $NUM_DATA_INTERFACES, + "data_interface_devs": data_interface_devs +} + +# Render nmstate configuration +with open("$SCRIPT_DIR/nmstate.yaml.j2", "r") as f: + template = Template(f.read()) +with open("$WORK_DIR/nmstate.yaml", "w") as f: + f.write(template.render(**context)) + +print("Nmstate configuration generated successfully") +EOF +then + die "Failed to render macvtap nmstate template" +fi + +# Apply macvtap configuration using nmstate +nmstatectl apply "$WORK_DIR/nmstate.yaml" || die "Failed to apply macvtap nmstate configuration" +log "Macvtap interfaces created via nmstate" + +# Libvirt image directory for proper SELinux context and permissions +LIBVIRT_IMAGE_DIR="/var/lib/libvirt/images" +mkdir -p "$LIBVIRT_IMAGE_DIR" + +# Create a working copy of the NXOS image for the VM +NXOS_DISK="$LIBVIRT_IMAGE_DIR/nxos-disk.qcow2" + +log "Creating working copy of NXOS image: $NXOS_DISK" +[ -f "$NXOS_DISK" ] && rm "$NXOS_DISK" +qemu-img create -f qcow2 -F qcow2 -b "$BASE_IMAGE" "$NXOS_DISK" || die "Failed to create backing image" + +# Render systemd service files from Jinja2 templates +log "Rendering systemd service files..." + +if ! python3 << EOF +from jinja2 import Template +import json + +# Load data interface arrays from JSON files +with open("$WORK_DIR/data_interface_macs.json", "r") as f: + data_interface_macs = json.load(f) + +with open("$WORK_DIR/data_interface_devs.json", "r") as f: + data_interface_devs = json.load(f) + +# Template variables +context = { + "vm_name": "$VM_NAME", + "nxos_disk": "$NXOS_DISK", + "console_port": "$CONSOLE_PORT", + "work_dir": "$WORK_DIR", + "num_data_interfaces": $NUM_DATA_INTERFACES, + "data_interface_macs": data_interface_macs, + "data_interface_devs": data_interface_devs, + "uefi_firmware": "$UEFI_FIRMWARE" +} + +# Render switch systemd service +with open("$SCRIPT_DIR/nxos-switch.service.j2", "r") as f: + template = Template(f.read()) +with open("/etc/systemd/system/nxos-switch.service", "w") as f: + f.write(template.render(**context)) + +print("Systemd service generated successfully") +EOF +then + die "Failed to render systemd service template" +fi + +log "NXOS switch systemd service written to /etc/systemd/system/nxos-switch.service" + +# Reload systemd and start service +log "Reloading systemd daemon..." +systemctl daemon-reload || die "Failed to reload systemd" + +log "Enabling and starting NXOS switch service..." +systemctl enable nxos-switch.service || die "Failed to enable nxos-switch service" +systemctl start nxos-switch.service || die "Failed to start nxos-switch service" + +log "NXOS VM started successfully via systemd" +log "Console available at: telnet localhost $CONSOLE_PORT" +log "Or use: journalctl -u nxos-switch.service -f" +log "Note: NXOS switch will use POAP for automatic configuration" +log " POAP files (poap.py, poap.cfg) should be available via TFTP/HTTP" +log "" +log "Service management:" +log " systemctl status nxos-switch.service # Check VM status" +log " systemctl restart nxos-switch.service # Restart VM" +log " systemctl stop nxos-switch.service # Stop VM" +log "" +log "Network configuration:" +log " Direct QEMU with macvtap passthrough (created via nmstate):" +log " Interface 0: ${DATA_INTERFACE_DEVS[0]} -> macvtap0 -> mgmt0 (MAC: ${DATA_INTERFACE_MACS[0]})" +for ((i=1; i macvtap$i -> Ethernet1/$i (MAC: ${DATA_INTERFACE_MACS[i]})" +done + +exit 0 diff --git a/images/switch-host-scripts/nxos/wait.sh b/switch-images/runtime-scripts/nxos/wait.sh similarity index 57% rename from images/switch-host-scripts/nxos/wait.sh rename to switch-images/runtime-scripts/nxos/wait.sh index 75abf670..ab8fa2ca 100644 --- a/images/switch-host-scripts/nxos/wait.sh +++ b/switch-images/runtime-scripts/nxos/wait.sh @@ -1,10 +1,24 @@ #!/bin/bash -# Wait for Cisco NXOS switch to boot and be ready +# Copyright Red Hat, Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +# +# Wait for NXOS switch to boot and be ready # NXOS uses POAP, so we just wait for the loader prompt or POAP to start set -euo pipefail -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" LIB_DIR="${LIB_DIR:-/usr/local/lib/hotstack-switch-vm}" # Source common functions @@ -13,14 +27,12 @@ source "$LIB_DIR/common.sh" CONSOLE_PORT="${CONSOLE_PORT:-55001}" WAIT_TIMEOUT="${WAIT_TIMEOUT:-1200}" # 20 minutes for NXOS boot + POAP +POAP_SETTLE_TIME="${POAP_SETTLE_TIME:-60}" # Time to wait after boot detected log "Waiting for NXOS switch to boot and POAP to complete..." -log "This may take up to 20 minutes..." log "Console available at: telnet localhost $CONSOLE_PORT" # Wait for POAP completion or login prompt -# POXOS will either show "Abort Power On Auto Provisioning" or complete POAP and show login -# We're looking for the login prompt which indicates boot is complete MAX_ATTEMPTS=$((WAIT_TIMEOUT / 10)) SLEEP_TIME=10 @@ -34,23 +46,17 @@ for attempt in $(seq 1 $MAX_ATTEMPTS); do log "Boot check attempt $attempt/$MAX_ATTEMPTS..." # Try to read from console - OUTPUT=$(timeout 5 nc -w 1 localhost "$CONSOLE_PORT" 2>/dev/null | tr -dc '[:print:]\n' || true) + OUTPUT=$(nc -w 2 localhost "$CONSOLE_PORT" 2>/dev/null | tr -dc '[:print:]\n' || true) if echo "$OUTPUT" | grep -qE "(login:|switch\(boot\)#|Abort Power On Auto Provisioning)"; then - log "NXOS boot detected!" - log "Switch is booting with POAP..." + log "NXOS boot detected - POAP is active" + log "Waiting ${POAP_SETTLE_TIME}s for POAP to settle..." + sleep "$POAP_SETTLE_TIME" - # Give POAP some time to complete - log "Waiting for POAP to complete configuration..." - sleep 60 - - log "NXOS switch boot complete - POAP should be active" + log "NXOS switch ready" exit 0 fi - # Send newline to potentially trigger output - echo "" | nc -w 1 localhost "$CONSOLE_PORT" >/dev/null 2>&1 || true - sleep "$SLEEP_TIME" done diff --git a/images/switch-host-scripts/start-switch-vm.sh b/switch-images/runtime-scripts/start-switch-vm.sh similarity index 82% rename from images/switch-host-scripts/start-switch-vm.sh rename to switch-images/runtime-scripts/start-switch-vm.sh index fd24cc4e..5fc3f139 100755 --- a/images/switch-host-scripts/start-switch-vm.sh +++ b/switch-images/runtime-scripts/start-switch-vm.sh @@ -1,4 +1,19 @@ #!/bin/bash +# Copyright Red Hat, Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +# # Main entry point for starting virtual switch VM # Delegates to model-specific setup script # This script should be called once at instance boot via cloud-init