Cisco ACI with VMware VDS Integration

Adding ESXi Host Considerations

When adding VMware ESXi hosts to a Virtual Machine Manager (VMM) domain with VMware vSphere Distributed Switch (VDS), ensure the ESXi host version is compatible with the Distributed Virtual Switch (DVS) version already deployed in vCenter. Consult VMware documentation for detailed compatibility requirements. Incompatibility between ESXi host and DVS versions will prevent vCenter from adding the ESXi host to the DVS, resulting in an error. Modifying the DVS version setting via Cisco APIC is not possible; lowering the DVS version requires removing and reapplying the VMM domain configuration with a lower setting.

ESXi 6.5 Hosts with VIC Cards and UCS Servers

Important: For ESXi 6.5 hosts using UCS B-Series or C-Series servers with VIC cards, vmnics may go down during a port state event (e.g., link flap, TOR reload). To prevent this, install the eNIC driver from the VMware website instead of using the default driver: VMware Website Driver Link.

VMware vCenter High Availability

VMware vCenter High Availability (VCHA), introduced in VMware vSphere 6.5, addresses the single point of failure for VMware vCenter. In case of an active node failure, the passive node takes over with the same IP address, credentials, and information. No new VMM configuration is required for VCHA. Cisco APIC will automatically reconnect once the passive node is reachable.

Guidelines for Upgrading VMware DVS from 5.x to 6.x and VMM Integration

This section outlines guidelines for upgrading VMware Distributed Virtual Switch (DVS) from version 5.x to 6.x and integrating it with VMM.

Pushing the VMM Domain After Deleting It

If a VMware Distributed Virtual Switch (DVS) created in Cisco APIC is accidentally deleted from VMware vCenter, the APIC policy will not be reapplied. To push the VMM domain again, disconnect and then reconnect the Cisco APIC VMware vCenter connectivity. This action ensures APIC reapplies the VMM domain and recreates the DVS in vCenter.

Read-Only VMM Domains

Introduced in Cisco APIC Release 3.1(1), read-only VMM domains allow viewing inventory information for a VDS in VMware vCenter that APIC does not manage. While you can associate EPGs and configure policies, these policies are not pushed to the VDS, and no faults are raised for read-only domains. The workflow and prerequisites are similar to creating other VMM domains.

Prerequisites for Creating a VMM Domain Profile

To configure a VMM domain profile, ensure the following:

• All fabric nodes are discovered and configured.
• Inband (inb) or out-of-band (oob) management has been configured on the APIC.

vCenter Domain Operational Workflow

The diagram illustrates a sequential workflow for vCenter domain operations:

1. The APIC Administrator creates an Application Policy.
2. The APIC connects to VMware vCenter.
3. The VMware vCenter Server creates Port Groups.
4. ESXi Hosts are attached to the Virtual Distributed Switch (VDS).
5. APIC automatically maps Endpoint Groups (EPGs) to Port Groups.
6. The APIC pushes the policy to the ACI Fabric, completing the Application Network Profile.

The APIC administrator configures vCenter domain policies in APIC, providing connectivity information such as:

• vCenter IP address, vCenter credentials, VMM domain policies, and VMM domain SPAN configuration.
• Policies including VLAN pools and domain types (e.g., VMware VDS, Cisco Nexus 1000V switch).
• Connectivity to physical leaf interfaces using Attach Entity Profiles (AEPs).

The APIC automatically connects to vCenter and creates a VDS (or uses an existing one) matching the VMM domain name.

Note: If using an existing VDS, it must reside within a folder of the same name.

Procedure

1. On the menu bar, navigate to Fabric > Access Policies.
2. In the navigation pane, click Quick Start, then in the central pane, click Configure an interface, PC, and VPC.
3. In the 'Configure an interface, PC, and VPC' dialog box:
   • Expand Configured Switch Interfaces.
   • Click the '+' icon.
   • Ensure the 'Quick' radio button is selected.
   • From the 'Switches' drop-down list, select the appropriate leaf ID. The 'Switch Profile Name' field will auto-populate.
   • Click the '+' icon to configure switch interfaces.
   • In the 'Interface Type' area, select the appropriate radio button.
   • In the 'Interfaces' field, enter the desired interface range. The 'Interface Selector Name' field will auto-populate.
   • In the 'Interface Policy Group' area, select the 'Create One' radio button.
   • From the 'Link Level Policy' drop-down list, choose the desired link level policy.
   • From the 'CDP Policy' drop-down list, choose the desired CDP policy.
   • Similarly, choose desired interface policies from other available policy areas.
   • In the 'Attached Device Type' area, select ESX Hosts.
   • In the 'Domain' area, ensure 'Create One' is selected. Enter the domain name in the 'Domain Name' field.
   • In the 'VLAN' area, ensure 'Create One' is selected. Enter the VLAN range in the 'VLAN Range' field.
   • It is recommended to use a range of at least 200 VLAN numbers. Avoid including your manually assigned infra VLAN in this range to prevent potential faults, especially with OpFlex integration.
   • In the 'vCenter Login Name' field, enter the login name.
   • Optionally, from the 'Security Domains' drop-down list, choose the appropriate security domain.
   • Enter the password in the 'Password' field and re-enter it in the 'Confirm Password' field.
   • Expand 'vCenter'.
4. In the 'Create vCenter Controller' dialog box, enter the required information and click OK.
5. In the 'Configure Interface, PC, And VPC' dialog box, complete the following actions:
   • If Port Channel Mode and vSwitch Policy areas are not specified, policies configured earlier will apply.
   • From the 'Port Channel Mode' drop-down list, choose a mode.
   • In the 'vSwitch Policy' area, select the radio button to enable CDP or LLDP.
   • From the 'NetFlow Exporter Policy' drop-down list, choose or create a policy. A NetFlow exporter policy configures external collector reachability.
   • Choose values from the 'Active Flow TimeOut', 'Idle Flow Timeout', and 'Sampling Rate' drop-down lists.
   • Click SAVE twice, then click SUBMIT.
6. Verify the new domain and profiles by navigating to Virtual Networking > Inventory, expanding VMM Domains > VMware > Domain_name > vCenter_name. In the work pane, check the 'Properties' for the VMM domain name and vCenter properties to confirm the controller is online and inventory is available.

Promoting a Read-Only VMM Domain Caveats

• Promoting a read-only domain requires a specific network folder structure for the VDS on the vCenter server. If the VDS is not in a folder, create one with the same name as the VDS and move the VDS into it before promoting to read-write. Failure to do so may result in APIC creating a new VDS in a new folder.
• When creating port-groups for read-only VMM domains intended for promotion, use the format <tenant-name>|<application-name>|<EPG-name>.

Promoting a Read-Only VMM Domain Using the Cisco APIC GUI

When a VMM domain is promoted to fully managed and an EPG is associated, port-groups named in the standard format are automatically added to the EPG. If a different naming format was used, VMs must be manually reassigned from old port-groups to new APIC-created ones.

• Create an EPG and associate it with the VMM domain. A fault will occur if the port-group lacks an EPG policy.
• Remove VMs from the old port-group and attach them to the EPG.

Note: This process may cause traffic loss.

• After detaching VMs, delete the old port-group from vCenter.
• When migrating from read-only to read-write, use a unique VLAN range separate from the physical domain range to avoid VLAN exhaustion.
• To use the same EPG across multiple VMMs and vCenters, configure a Link Aggregation Group (LAG) policy with the same name as the domain. An EPG can only connect to one LAG policy. For different LAG policies, associate each with a different EPG. See 'Enhanced LACP Policy Support' (page 13).

Procedure:

1. Log in to Cisco APIC.
2. Associate an Access Entity Profile (AEP) with the read-only VMM domain: Navigate to Fabric > Access Policies > Policies > Global > Attachable Access Entity Profiles. Select an AEP and associate it with the read-only VMM domain.
3. Promote the VMM domain: Navigate to Virtual Networking > Inventory, expand the VMM Domains > VMware folder, and select the read-only VMM Domain. Change the 'Access Mode' to Read Write Mode. Select a VLAN Pool and click Submit.
4. Create a new Link Aggregation Group (LAG) policy if using vCenter 5.5 or later, as described in 'Create LAGs for DVS Uplink Port Groups Using the Cisco APIC GUI' (page 14).
5. Associate the LAG policy with appropriate EPGs if using vCenter 5.5 or later, as described in 'Associate Application EPGs to VMware vCenter Domains with Enhanced LACP Policies Using the Cisco APIC GUI' (page 15).

What to do next: EPGs attached to the VMM domain and configured policies will now be pushed to the VDS in VMware vCenter.

Creating a Trunk Port Group Using the GUI

This section details creating a trunk port group via the GUI.

Before you begin: Ensure the trunk port group is tenant independent.

Procedure:

1. Log in to the APIC GUI.
2. On the menu bar, choose Virtual Networking.
3. In the navigation pane, choose VMM Domains > VMware > domain > Trunk Port Groups and right-click Create Trunk Port Group.
4. In the 'Create Trunk Port Group' dialog box:
   • Name: Enter the EPG name.
   • Promiscuous Mode: Select 'Disabled' (default) or 'Enabled'. 'Enabled' allows VMs to receive unicast traffic not destined for their MAC addresses.
   • Trunk Portgroup Immediacy: Select 'Immediate' or 'On Demand' (default). This determines when policies are resolved on leaf switches.
   • MAC changes: Select 'Disabled' or 'Enabled' (default). 'Enabled' allows new MAC addresses for the VM network adapter.
   • Forged transmits: Select 'Disabled' or 'Enabled' (default). 'Enabled' allows forged transmits, where a network adapter sends traffic identifying itself as another. This security policy ensures the virtual network adapter's effective address matches the source address in the Ethernet frame.
   • Enhanced Lag Policy: Choose the uplink with the desired Link Aggregation Control Protocol (LACP) policy. This policy uses DVS uplink port groups configured in LAGs with a load-balancing algorithm. At least one uplink must have an LACP policy applied to improve load balancing. Refer to 'Enhanced LACP Policy Support' (page 13) for more information.
   • VLAN Ranges: Click '+' and enter the VLAN range (e.g., vlan-100 vlan-200).
   • If no VLAN Range is specified, the VLAN list is taken from the domain's VLAN namespace.
   • Click Update.
5. Click Submit.

Guidelines for Cisco UCS B-Series Servers

When integrating blade server systems into Cisco ACI for VMM integration (e.g., Cisco Unified Computing System (UCS) blade servers), consider these guidelines:

Note: This example demonstrates configuring a port channel access policy for Cisco UCS blade servers. Similar steps apply for virtual port channels or individual link access policies based on uplink connection. If no port channel is explicitly configured on APIC for UCS blade server uplinks, the default behavior is MAC pinning.

• VM endpoint learning relies on Cisco Discovery Protocol (CDP) or Link Layer Discovery Protocol (LLDP). If supported, CDP must be enabled from the leaf switch port through blade switches to the blade adapters.
• Ensure the management address type, length, and value (TLV) is enabled on the blade switch (CDP or LLDP) and advertised to servers and fabric switches. Management TLV address configuration must be consistent across CDP and LLDP on the blade switch.
• Cisco APIC does not manage fabric interconnects or blade servers; UCS-specific policies like CDP or port channel must be configured via UCS Manager.
• VLANs defined in the VLAN pool used by the attachable access entity profile on APIC must also be manually created on UCS and allowed on appropriate uplinks connecting to the fabric, including the infrastructure VLAN if applicable. Refer to the Cisco UCS Manager GUI Configuration Guide.
• Both CDP and LLDP are supported with Cisco UCS B-series servers from UCSM 2.2.4b. LLDP is not supported with earlier firmware.
• CDP is disabled by default in Cisco UCS Manager. Enable CDP by creating a Network Control Policy.
• Do not enable fabric failover on adapters in UCS server service profiles. Cisco recommends allowing the hypervisor to handle failover at the virtual switch layer for proper load balancing.

Setting up an Access Policy for a Blade Server Using the GUI

Note: Symptom: Changes to management IP of unmanaged nodes (e.g., blade switch, fabric interconnect) update in VMware vCenter, but vCenter does not send events to Cisco APIC, causing APIC to be out of sync. Workaround: Trigger an inventory pull for the VMware vCenter controller managing ESX servers behind the unmanaged node.

Before you begin: To operate with Cisco APIC, Cisco UCS Fabric Interconnect must be version 2.2(1c) or later. All components (BIOS, CIMC, adapter) must also be version 2.2(1c) or later. Refer to the Cisco UCS Manager CLI Configuration Guide for details.

Procedure:

1. On the menu bar, choose Fabric > Access Policies.
2. In the navigation pane, click Quick Start.
3. In the central pane, click Configure an interface, PC, and VPC.
4. In the 'Configure Interface, PC, and VPC' dialog box, click the '+' icon to select switches.
5. In the 'Switches' field, choose the desired switch IDs from the drop-down list.
6. Click the '+' icon to configure switch interfaces.
7. In the 'Interface Type' field, click the VPC radio button.
8. In the 'Interfaces' field, enter the appropriate interface or interface range connected to the blade server.
9. In the 'Interface Selector Name' field, enter a name.
10. From the 'CDP Policy' drop-down list, choose 'default'. (CDP must be disabled between the leaf switch and the blade server.)
11. From the 'LLDP Policy' drop-down list, choose 'default'. (LLDP must be enabled for receive and transmit states between the leaf switch and the blade server.)
12. From the 'LACP Policy' drop-down list, choose Create LACP Policy. (LACP policy must be set to active between the leaf switch and the blade server.)
13. In the 'Create LACP Policy' dialog box:
    • Name: Enter a name for the policy.
    • Mode: Select the 'Active' radio button.
    • Keep default values and click Submit.
14. From the 'Attached Device Type' field drop-down list, choose ESX Hosts.
15. In the 'Domain Name' field, enter an appropriate name.
16. In the 'VLAN Range' field, enter the range.
17. In the 'vCenter Login Name' field, enter the login name.
18. In the 'Password' and 'Confirm Password' fields, enter the password.
19. Expand the 'vCenter' field, and in the 'Create vCenter Controller' dialog box, enter the content and click OK.
20. In the 'vSwitch Policy' field, configure as follows: (Between blade server and ESX hypervisor: CDP enabled, LLDP disabled, LACP disabled; MAC Pinning must be set.)
    • Check the 'MAC Pinning' box.
    • Check the 'CDP' box.
    • Leave the 'LLDP' box unchecked as LLDP must remain disabled.
21. Click Save, then Save again. Click Submit. The access policy is set.

Custom User Account with Minimum VMware vCenter Privileges

Setting specific VMware vCenter privileges allows Cisco APIC to send API commands for DVS creation, publish port groups, and relay alerts. For APIC to configure VMware vCenter, your credentials must have the following minimum privileges:

• Alarms: APIC creates two alarms (DVS and port-group). Alarms are raised when EPG or Domain policy is deleted on APIC. Alarms for DVS or port-group cannot be deleted if VMs are attached.
• Distributed Switch
• dvPort Group
• Folder
• Network: APIC manages network settings like port-group addition/deletion, host/DVS MTU, LLDP/CDP, and LACP.
• Host:
  • Host.Configuration.Advanced settings
  • Host.Local operations.Reconfigure virtual machine
  • Host.Configuration.Network configuration
• Virtual machine: If using Service Graph, the 'Virtual machine' privilege is needed for virtual appliances used in Service Graph.
  • Virtual machine.Configuration.Modify device settings
  • Virtual machine.Configuration.Settings

For deploying service VMs using the service VM orchestration feature, enable these additional privileges:

• Datastore:
  • Allocate space
  • Browse datastore
  • Low level file operations
  • Remove file
• Host:
  • Local operations.Delete virtual machine
  • Local operations.Reconfigure virtual machine
• Resource:
  • Assign virtual machine to resource pool
• Virtual machine:
  • Inventory.Create new
  • Inventory.Create from existing
  • Inventory.Remove
  • Configuration.Add new disk
  • Provisioning.Deploy template

Quarantine Port Groups

The quarantine port group feature provides a method to clear port group assignments. When a VMware vSphere Distributed Switch (VDS) is created in VMware vCenter, a quarantine port group is created by default, blocking all ports. As part of integration with Layer 4 to Layer 7 virtual service appliances (e.g., load balancers, firewalls), APIC creates service port groups in vCenter for service stitching and orchestrates virtual appliance placement. When a service graph is deleted, service VMs are automatically moved to the quarantine port group. This auto-move applies only to APIC-orchestrated service VMs.

You can further manage ports in the quarantine port group, such as migrating them to another port group (e.g., a VM network).

The quarantine port group mechanism does not apply to regular tenant endpoint groups (EPGs) or their associated port groups and VMs. If a tenant EPG is deleted, tenant VMs in its associated port group remain intact and are not moved to the quarantine port group.

On-Demand VMM Inventory Refresh

Triggered Inventory offers a manual option to refresh APIC inventory from the VMM controller. It is not required for normal operations and should be used judiciously when errors occur. APIC automatically pulls inventory during process restarts, leadership changes, or periodic audits to align VMM inventory with the controller's. If VMware vCenter APIs error out, APIC might not download the full inventory despite retries, indicated by a fault. Triggered inventory allows initiating an inventory pull from APIC to vCenter.

APIC does not synchronize VMM configuration with VMware vCenter VDS configuration. Direct changes to VDS settings in vCenter are not overwritten by APIC, except for PVLAN configuration.

Physically Migrating the ESXi Host

Complete the following tasks to physically migrate ESXi hosts:

1. Put the host into maintenance mode or evacuate VM workloads by another method.
2. Remove the ESXi host from the VMware VDS, Cisco Application Centric Infrastructure (ACI) Virtual Edge, or Cisco Application Virtual Switch.
3. Physically recable the ESXi host to the new leaf switch or pair of leaf switches.
4. Add the ESXi host back to the VMware VDS, Cisco Application Centric Infrastructure (ACI) Virtual Edge, or Cisco Application Virtual Switch.