Cisco Nexus 9000: Packet Tracer Tool Explained

Introduction

Packet Tracer is a built-in utility on Nexus 9000 switches used to trace the path of packets through the switch. It can be invoked from the command line and configured based on IP addresses and/or Layer 4 characteristics. It is not suitable for matching ARP traffic.

This tool helps determine if a flow is passing through the switch and provides counters to track flow statistics, which can be useful for diagnosing intermittent or complete packet loss.

Prerequisites

Cisco recommends having a basic understanding of the following topics:

• Cisco Nexus 9000 hardware architecture

Components Used

The information in this document is based on the following software and hardware versions:

• Cisco Nexus 9500
• SW Version 7.0(3)I2(2a)

Use Case Scenarios

• Applicable only to IPv4 flows (IPv6 and non-IP are not supported).
• The tool does not display packet internal details as seen in Wireshark.
• Useful for intermittent packet loss where packet loss confirmation is needed via Ping or other utilities.
• Useful for complete packet loss.

Supported Hardware

Only Top-of-Rack (TOR) switches with Line Cards/Fabric Modules or Broadcom Trident II ASICs are supported. The supported list includes:

• N9K-C9372TX
• N9K-C9372PX
• N9K-C9332PQ
• N9K-C9396TX
• N9K-C9396PX
• N9K-C93128TX
• N9K-C9336PQ
• N9K-X9564PX
• N9K-X9564TX
• N9K-X9636PQ

Unsupported Hardware

• N9K-C93180YC-EX
• N9K-X9732C-EX
• N9K-C9232C
• N9K-C9272Q
• N9K-C92160YC

Note: If a specific line card/TOR is not listed, contact TAC.

How to Use Packet Tracer

Configuration

The packet-tracer command is an EXEC level command.

To configure packet tracing:

N9K-9508# test packet-tracer src_ip <src_ip> dst_ip <dst_ip>
N9K-9508# test packet-tracer start
N9K-9508# test packet-tracer stop
N9K-9508# test packet-tracer show

The above commands program triggers on all Broadcom Trident II ASICs present on line cards or fabric modules. When a flow with matching characteristics passes through these modules, their counters will be displayed, aiding in identifying the path within the switch (Ingress module —> one of the Fabric modules —> Egress module).

Counters can be used to pinpoint drops.

Background Information

Fabric modules interconnect in I/O module slots. All fabric modules are active and forward traffic. Each fabric module has two instances of Broadcom Trident II ASIC (T2).

The diagram illustrates the Nexus 9000 switch architecture, showing modules and fabric connections. It details the path of traffic through different components like NS (ALE) and NFE (T2) ASICs across modules and fabric. The diagram shows numbered paths indicating ingress and egress points on modules and fabric.

Diagram Description:

The diagram depicts a Nexus 9000 architecture with two main modules (Module 1 and Module 2) connected via Fabric 2. Each module contains NS (ALE) and NFE(T2) ASIC components. Fabric 2 has two NFE(T2) ASIC instances. Numbered arrows indicate traffic flow paths: 1. Ingress on module 1, 2. Ingress on fabric module, 3. Ingress on module 2, 4. Ingress on module 2, 5. Ingress on fabric module, 6. Ingress on module 1 (from fabric). The diagram also shows port types like 100G/40G/10G SERDES or QSFP28.

The primary steps to configure to match traffic of interest are:

switch# test packet-tracer {<src-ip>|<dst-ip>|<src-l4-port>|<dst-l4-port>} [<protocol>] [detail-fp|detail-hg]

The configuration required is:

switch# test packet-tracer src_ip <src_ip>
switch# test packet-tracer dst_ip <dst_ip>
switch# test packet-tracer protocol <protocol>

There is no need to apply to a specific interface. The above config installs filter ACLs on all LC/FM for all T2 ASIC instances. It displays the packet count for the module where traffic ingress. This matches traffic of interest on modules, both line cards and fabric.

Problem

Port Access-lists (PACLs) are used to verify if a specific physical interface received traffic of interest. However, on the Nexus platform, some line cards do not have TCAM etched for PACLs. TCAM fragmentation may require reloading the module. In such cases, Packet Tracer can be used to verify traffic of interest. You can also trace packets going to fabric ports and then to egress modules. Therefore, Packet Tracer provides more information on how traffic is forwarded within the switch.

Packet Tracer utilizes TCAM entries logged for SPAN.

Solution

NS - North Star ASIC, T2 - Trident II ASIC, NFE - Network Forwarding Engine, ALE - ACI Leaf Engine.

For more details on Nexus 9000 switch architecture, refer to:

Cisco Nexus 9000 Series Switches White Paper

Example Scenario:

ICMP SRC IP: 10.1.1.1/24

ICMP DST IP: 10.2.2.1/24

Configuration Example:

N9K-9508# test packet-tracer src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1
N9K-9508# test packet-tracer start

Interpreting test packet-tracer show Output:

N9K-9508# test packet-tracer show
Packet-tracer stats
-----------------------
Module 1:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1
ASIC instance 0:
Entry 0: id = 7425, count = 0, active, fp
Entry 1: id = 7426, count = 0, active, hg
ASIC instance 1:
Entry 0: id = 7425, count = 0, active, fp
Entry 1: id = 7426, count = 0, active, hg
Filter 2 uninstalled:
...
Module 2:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1
ASIC instance 0:
Entry 0: id = 7425, count = 0, active, fp
Entry 1: id = 7426, count = 0, active, hg
...

Configuration Example for Bidirectional Traffic:

N9K-9508# test packet-tracer src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1
N9K-9508# test packet-tracer src-ip 10.2.2.1 dst-ip 1.1.1.1 protocol 1

Starting and Checking Statistics:

N9K-9508# test packet-tracer start
N9K-9508# test packet-tracer show non-zero

Test: Ping from SRC IP to DST IP connected from Module 1 to Module 2.

Router# ping 10.1.1.1 source 10.2.2.1
PING 10.1.1.1 (10.1.1.1) from 10.2.2.1: 56 data bytes
64 bytes from 10.1.1.1: icmp_seq=0 ttl=253 time=0.77 ms
...
--- 10.1.1.1 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.383/0.477/0.77 ms

Verification: Check packet tracer counts.

N9K-9508# test packet-tracer show non-zero
Packet-tracer stats
====================
Module 1:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1
ASIC instance 0:
Entry 0: id = 7425, count = 5, active, fp
Filter 2 installed: src-ip 10.2.2.1 dst-ip 10.1.1.1 protocol 1
...
Module 2:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1
Filter 2 installed: src-ip 10.2.2.1 dst-ip 10.1.1.1 protocol 1
ASIC instance 0:
Entry 0: id = 7457, count = 5, active, fp
...
Module 22:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1
ASIC instance 0:
Entry 0: id = 7425, count = 4, active, hg
...
Module 23:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1
ASIC instance 0:
Entry 0: id = 7425, count = 1, active, hg
...
ASIC instance 0:
Entry 0: id = 7425, count = 3, active, hg
...
Module 24:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1
ASIC instance 0:
Entry 0: id = 7425, count = 2, active, hg
...

Other Useful Commands

• test packet-tracer remove-all: Removes all configured filters.
• test packet-tracer clear <filter #>: Clears counters for all filters or a specified filter.
• test packet-tracer src_ip <.> dst_ip <> l4-dst-port <dst_port> | l4-src-port <src_port> | protocol <protocol>: Matches based on L4 source port, L4 destination port, or protocol.