Cisco Catalyst 6500 Switch Architecture

Cisco Catalyst 6500 Switch Architecture

RST-3465

RST-3465 12523_04_2006_c2

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

1

Session Goal To provide you with a thorough understanding of the Catalyst® 6500 switching architecture, packet flow, and key forwarding engine functions

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

2

2

Agenda • Chassis Architecture • Supervisor Engine and Switch Fabric Architecture • Switching Module Architecture • Layer 2 Forwarding • IPv4 Forwarding • IPv4 Multicast Forwarding • Security and Feature ACLs • QoS • NetFlow

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

3

Chassis Architecture

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

4

Catalyst 6500 Chassis Architecture • Modular chassis in variety of form factors 3, 4, 6, 9, and 13 - slot versions

• Enhanced (“E”) chassis offer higher system power capacity and better signal integrity 3, 4, 6, and -9 slot versions

• Classic switching bus traces/connectors • Crossbar fabric traces/connectors • Redundant power supplies • Fan tray for system cooling 6509 - NEB - A chassis offers redundant fan trays and air filtration

• Redundant voltage termination (VTT)/clock modules • Redundant MAC address EEPROMs

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

5

Catalyst 6503/6503E and 6504E • Slots 1 and 2—Supervisor engine, or switching module Power Supply

• Other slots—Any switching module • 2 fabric channels per slot

Power Supply

VTT/Clock Modules

EEPROMs

• Power supplies in rear

• 950W AC/DC and 1400W AC power supplies for 6503/6503E • 2700W AC/DC power supplies for 6504E

Fan Tray

6503/6503E—Power entry modules (PEMs) in front of chassis provides power connection

Dual Channels

Slot 1

Dual Channels

Slot 2

Dual Channels

Slot 3

Dual Channels

Slot 4

Crossbar

Shared Bus

5 RU 4 RU

Note: CEF720 modules not supported in Catalyst 6503 (non-E) chassis RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

6

Catalyst 6506/6509 and 6506E/6509E • Slots 1 and 2—Supervisor Engine 2, or switching module

VTT/Clock Modules

• Slots 5 and 6—Supervisor Engine 32/720, or switching module • Other slots—Any switching module

• Wide variety of power supplies, from legacy 1000W to new 6000W—E chassis requires at least 2500W PS

Fan Tray

• 2 fabric channels per slot

• NEB-A chassis has vertical slot alignment, dual fan trays, front-to-back air flow, air filtration system

21 RU

EEPROMs

Dual Channels

Slot 1

Dual Channels

Slot 2

Dual Channels

Slot 3

Dual Channels

Slot 4

Dual Channels

Slot 5

Dual Channels

Slot 6

Dual Channels

Slot 7

Dual Channels

Slot 8

Dual Channels Crossbar

Slot 9

Power Supply

Shared Bus Power Supply

12 RU 15 RU RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

7

Catalyst 6513 VTT/Clock Modules

• Slots 1 and 2—Supervisor Engine 2, or switching module • Slots 7 and 8—Supervisor Engine 32/720, or switching module Fan Tray

• Wide variety of power supplies, from 2500W to new 6000W • 1 fabric channel slots 1–8 Dual-fabric modules not supported in slots 1–8!

• 2 fabric channels slots 9–13

19 RU

Any switching module RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

EEPROMs

Single Channel

Slot 1

Single Channel

Slot 2

Single Channel

Slot 3

Single Channel

Slot 4

Single Channel

Slot 5

Single Channel

Slot 6

Single Channel

Slot 7

Single Channel

Slot 8

Dual Channels

Slot 9

Dual Channels

Slot 10

Dual Channels

Slot 11

Dual Channels

Slot 12

Dual Channels Crossbar

Slot 13

Power Supply

Shared Bus Power Supply Cisco Public

8

Agenda • Chassis Architecture • Supervisor Engine and Switch Fabric Architecture • Switching Module Architecture • Layer 2 Forwarding • IPv4 Forwarding • IPv4 Multicast Forwarding • Security and Feature ACLs • QoS • NetFlow

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

9

Supervisor Engine and Switch Fabric Architecture

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

10

Supervisor 2 • PFC2 forwarding engine daughter card

• Internal RP and SP bootflash (32MB each)

• Switch Processor CPU (300MHz R7000)

• External PCMCIA flash slot

• Optional MSFC2 daughter card with Route Processor CPU (300MHz R7000)

• Supports optional Switch Fabric Module (SFM)/SFM2 • 2 x 1GE GBIC uplink ports

• 256MB/256MB (Sup2) or 256MB/512MB (Sup2U) DRAM

Supported from Cisco IOS 12.1(5c)EX and Catalyst OS 6.1(1)/12.1(3a)E1 RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

11

Supervisor 2 / PFC2 Architecture RP CPU runs L3 protocols and maintains control plane state

FIB contains IPv4 prefix entries

QoS TCAM contains QoS ACL entries

Supervisor 2 Baseboard ACL TCAM contains security and feature ACL entries

DRAM

NetFlow table for stats and features

RP (MSFC2) CPU

1 Gbps

Port ASIC

QoS TCAM

FIB TCAM

L2/L4 Engine

GbE Uplinks

MSFC2 Daughter Card

ADJ contains rewrite info

ACL TCAM

SP CPU runs L2 protocols and manages hardware

Layer 3 Engine

DRAM

SP (NMP) 1 Gbps CPU

ADJ LCDBUS LCRBUS

NetFlow L2 CAM

Bus Interface

12523_04_2006_c1

16 Gbps Bus © 2006 Cisco Systems, Inc. All rights reserved.

Fabric Interface 8 Gbps

L2 CAM contains MAC entries RST-3465

PFC2 Daughter Card

Replication Engine

DBUS RBUS

MET

To SFM/SFM2 EOBC

Replication engine for multicast/ SPAN Interface to fabric and bus Cisco Public

12

Supervisor 720 • 720Gbps crossbar fabric • PFC3 forwarding engine daughter card • Integrated RP/SP CPUs on MSFC3 daughter card (600MHz MIPS)

• 512/512MB (3A/B) or 1/1GB (3BXL) DRAM • Internal RP and SP bootflash (64MB each) • Optional 512MB CF bootflash upgrade for SP (WS-CF-UPG=) • Dual external compact flash slots • 2 x GbE uplink ports— 2 x SFP 1 x SFP and 1 x 10/100/1000

Supported from Cisco IOS 12.2(14)SX and Catalyst OS 8.1(1)/12.2(14)SX2 RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

13

Supervisor 720 / PFC3 Architecture ACL and QoS classification move to L3/4 engine

Addition of ACL TCAM counters

GbE Uplinks

Supervisor 720 Baseboard Counter QoS FIB ADJ FPGA TCAM TCAM (B/BXL Only)

ACL TCAM

L3/4 Engine

L2 Engine L2 CAM

L2 CAM moved on-chip for higher performance RST-3465 12523_04_2006_c1

CPU Daughter Card

1 Gbps

RP (MSFC3) CPU

DRAM

1 Gbps

SP (NMP) CPU

DRAM

Port ASIC

NetFlow

PFC3 Daughter Card

16 Gbps Bus © 2006 Cisco Systems, Inc. All rights reserved.

RP and SP both sit on MSFC3 CPU daughter card

Fabric interface and replication engine combined

MET

Fabric Interface/ Replication Engine

20 Gbps

Integrated 720 Gbps Switch Fabric

17 x 20 Gbps Fabric Channels DBUS RBUS EOBC

… Crossbar switch fabric integrated on supervisor baseboard Cisco Public

14

Supervisor 32 • Classic supervisor—no fabric, uses 16Gig bus only • PFC3B forwarding engine daughter card • SP CPU (400MHz Sibyte) • MSFC2a routing engine

2 10GE Xenpak + 1 10/100/1000 RJ-45 uplink ports

• 256MB/256MB DRAM (512MB/512MB with non-$0 feature set) • Internal CF bootdisk (256MB) and MSFC2A bootflash (64MB) • External CF slot • Uplink options: 8 SFP + 1 10/100/1000 2 10GE + 1 10/100/1000

8 1GE SFP + 1 10/100/1000 RJ-45 uplink ports

Supported from Cisco IOS 12.2(18)SXF and Catalyst OS 8.4(1)/12.2(17)SXB7 RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

15

Supervisor 32-GE / PFC3 Architecture PFC3 exactly the same as on Supervisor 720

Supervisor Engine 32 Baseboard Counter FPGA

GbE Uplinks

QoS FIB ADJ TCAM TCAM

1 Gbps

SP CPU

DRAM

RP CPU

DRAM

Port ASIC ACL TCAM

L3/4 Engine

1 Gbps NetFlow

MSFC2a Daughter Card L2 Engine L2 CAM

PFC3 Daughter Card

Replication Engine WS-SUP32-GE-3B

MET

16 Gbps Bus RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

DBUS RBUS EOBC

Bus attached only; no fabric support Cisco Public

16

Supervisor 32-10GE / PFC3 Architecture Dual port ASICs to support two 10GE interfaces

Supervisor Engine 32 Baseboard Counter FPGA

ACL TCAM

QoS FIB ADJ TCAM TCAM

L3/4 Engine

L2 Engine L2 CAM

10GE Uplinks

Port ASIC

Port ASIC

1 Gbps 1 Gbps

NetFlow

FPGA MUX PFC3 Daughter Card

DRAM

RP CPU

DRAM

MSFC2a Daughter Card

Replication Engine WS-SUP32-10GE-3B

MET

16 Gbps Bus RST-3465 12523_04_2006_c1

SP CPU

© 2006 Cisco Systems, Inc. All rights reserved.

DBUS RBUS EOBC Cisco Public

17

Supervisor Chassis Requirements Supervisor 720 and Supervisor 32 require: • Catalyst 6500 or 6500-E chassis • High speed fan tray (FAN2/E-FAN) • 2500W power supply (AC or DC) or greater 3000W supply recommended for new deployments

• Specific chassis slots: Slot 1 or 2 in 3/4 slot Slot 5 or 6 in 6/9 slot Slot 7 or 8 in 13 slot

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

18

Crossbar Switch Fabric • Provides multiple conflict-free paths between switching modules Dedicated bandwidth per slot Compare to system bus which is shared by all bus-attached modules

• 18 fabric channels in total • Two fabric channels per slot in 6503/6504/6506/6509 • In 6513: One fabric channel slots 1–8 Two fabric channels slots 9–13 “Dual-fabric channel” modules not supported in slots 1–8 of 6513

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

19

Switch Fabric Module and SFM2 • 256 Gbps crossbar switch fabric • Works with Supervisor 2 and CEF256/dCEF256 modules • Fabric channels run at 8 Gbps full duplex 8 Gbps in/8 Gbps out per channel

• Fabric module occupies a full slot 6506/6509—Slots 5 and 6 6513—Slots 7 and 8

• SFM—Supports 6506 and 6509 (and E-versions) • SFM2—Supports 6506, 6509, and 6513 (and E-versions) • Not supported in 6503/6504

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

20

Supervisor 720 Switch Fabric • 720 Gbps crossbar switch fabric • Integrated on Supervisor 720 baseboard • Fabric channels run at 20 Gbps full duplex 20 Gbps in/20 Gbps out per channel

• Works with all fabric-capable modules Fabric channels auto-sync speed on per-slot basis (8 Gbps or 20Gbps)

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

21

Monitoring Fabric Status and Utilization • Cisco IOS: show fabric [active | channel-counters | errors | fpoe | medusa | status | switching-mode | utilization] • Catalyst OS: show fabric {channel {counters | switchmode | utilization} | status} 6506#show fabric utilization slot channel speed 1 0 8G 2 0 8G 3 0 20G 3 1 20G 4 0 20G 4 1 20G 6 0 20G 6506# RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Ingress % 22 4 0 11 0 10 0

Egress % 23 9 1 12 1 13 1 Cisco Public

22

Monitoring System Bus Utilization • Monitor the traditional Catalyst 6500 bus when using: Classic modules Centralized forwarding with a fabric • Cisco IOS: show catalyst6000 traffic-meter • Catalyst OS: show traffic 6506#show catalyst6000 traffic-meter traffic meter = peak = 6506#

RST-3465 12523_04_2006_c1

7% 46%

Never cleared reached at 08:07:50 PST Fri Dec 30 2005

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

23

Policy Feature Cards • Mandatory daughter card for supervisor engine • Provides the key components enabling high-performance hardware packet processing • Supervisor 2 supports PFC2 • Supervisor 32 supports PFC3B • Supervisor 720 supports: PFC3A PFC3B PFC3BXL

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

24

Policy Feature Cards (Cont.) Key hardware-enabled features: • Layer 2 switching • IPv4 unicast forwarding • IPv4 multicast forwarding • Security ACLs • QoS/policing • NetFlow accounting PFC3 also supports: • IPv6, MPLS*/VRF-lite, Bidir PIM, NAT/PAT, GRE/v6 tunnels, CoPP RST-3465 12523_04_2006_c1

* MPLS on 3B/3BXL only © 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

25

High-Level Forwarding Engine Logic Frame received

Input Layer 2 lookup Layer 2 Table

Ingress Forwarding Engine

FIB TCAM FIB lookup ACL TCAM

Router MAC?

Yes

No

Input QoS lookup

Input ACL lookup

QoS TCAM

NetFlow Table NetFlow lookup

Input QoS lookup QoS TCAM Input ACL lookup ACL TCAM

Bridged NetFlow NetFlow Table

RST-3465 12523_04_2006_c1

Output QoS lookup*

Output ACL lookup

Output Layer 2 lookup

QoS TCAM

ACL TCAM

Layer 2 Table

Output QoS lookup*

Output ACL lookup

QoS TCAM

ACL TCAM

© 2006 Cisco Systems, Inc. All rights reserved.

Transmit frame *PFC3 only

Cisco Public

26

PFC TCAM Technology • TCAM—Ternary Content Addressable Memory

• Leveraged heavily in Catalyst 6500

Value 1 Value 2 Mask 1

Value 4 Value 5

FIB, ACL, QoS, NetFlow all utilize TCAM memory

• All entries accessed in parallel—fixed performance independent of number of entries • Memory consists of groups of values and associated masks

Value 6 Value 7 Value 8 Value 1 Value 2 Mask 2

Value 5 Value 6

• Masks are used to “wildcard” some portion of values

Value 7 Value 8 Masks

© 2006 Cisco Systems, Inc. All rights reserved.

Value 3 Value 4

8:1 ratio of values to masks

RST-3465 12523_04_2006_c1

Value 3

Values Cisco Public

27

Generic TCAM Lookup Logic 1 Packet

Fields

3 Compare

Generate Lookup Key

01101010 011010xx 0110xxxx

110110xx

Lookup Key

000111xx

1 2

101101xx

3

100111xx

4

000000xx

5

010010xx

6

111111xx

7

2. Lookup key created

001100xx

8

3. As lookup key compared to value entries, associated mask applied

0111xxxx 1011xxxx

1 2

1101xxxx

3

0110xxxx 1110xxxx

4 45

0011xxxx

6

0000xxxx

7

1000xxxx

8

2

11111100

1. Relevant fields read from contents of packet

1=“Compare” 0=“Mask”

4. Longest match returns result

11110000 HIT!

Result format varies depending on lookup type

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Masks

Values

Result

Cisco Public

28

Agenda • Chassis Architecture • Supervisor Engine and Switch Fabric Architecture • Switching Module Architecture • Layer 2 Forwarding • IPv4 Forwarding • IPv4 Multicast Forwarding • Security and Feature ACLs • QoS • NetFlow

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

29

Switching Module Architecture

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

30

Classic Module DBUS RBUS

Example: WS-X6416-GBIC

Classic Module Port ASIC

Port ASIC

Port ASIC

Port ASIC

4xGE

4xGE

4xGE

4xGE

Port ASICs for physical connectivity, buffering, and queueing

DBUS RBUS

Classic Module Port ASIC

Example: WS-X6148A-RJ-45 RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

48x10/100

Cisco Public

31

CEF256 Module Example: WS-X6516-GBIC

8Gbps Fabric Channel

DBUS RBUS

CEF256 Module

Fabric Interface Fabric interface to interface with fabric and bus

LCDBUS LCRBUS

MET

Replication Engine

Replication engine for local SPAN/multicast replication RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Port ASIC

Port ASIC

Port ASIC

Port ASIC

4xGE

4xGE

4xGE

4xGE

Local linecard bus for ASIC interconnection Cisco Public

32

CEF256 Module with DFC Example: WS-X6516-GBIC with WS-F6K-DFC

Layer 2/4 Engine for L2 and ACL/QoS lookups 8Gbps Fabric Channel

Fabric Interface

Layer 2/4 Engine

CEF256 Module with DFC

L3 Engine

Layer 3 Engine for FIB/Adj and NetFlow lookups

DFC LCDBUS LCRBUS

MET

RST-3465 12523_04_2006_c1

Replication Engine

© 2006 Cisco Systems, Inc. All rights reserved.

Port ASIC

Port ASIC

Port ASIC

Port ASIC

4xGE

4xGE

4xGE

4xGE

Cisco Public

33

CEF720 Module Example: WS-X6748-SFP

DBUS RBUS

20Gbps Fabric Channel

Complex A

MET

Combined fabric interface and replication engine

Fabric Interface & Replication Engine

Bus Interface CFC

20Gbps Fabric Channel

CEF720 Module

Complex B Bus Interface

Fabric Interface & Replication Engine

Port ASIC

Port ASIC

Port ASIC

Port ASIC

12xGE

12xGE

12xGE

12xGE

MET

Transparent bus interface

Bus interface for control data only!! RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

34

CEF720 Module with DFC3 Example: WS-X6748-SFP with WS-F6700-DFC3B

20Gbps Fabric Channel

20Gbps Fabric Channel

Complex A

MET

Fabric Interface & Replication Engine Port ASIC

Port ASIC

12xGE

12xGE

Layer 2 Engine

DFC3

Layer3/4 Engine for FIB/Adj, ACL, QoS and NetFlow lookups RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

CEF720 Module with DFC3

Complex B Layer 2 Engine

L3/4 Engine

Fabric Interface & Replication Engine Port ASIC

Port ASIC

12xGE

12xGE

MET

Layer 2 Engine for L2 lookups Cisco Public

35

Distributed Forwarding • One or more modules have local forwarding engine (DFC—Distributed Forwarding Card) • Central engine and distributed engines perform different lookups independently and simultaneously • Implementation is fully distributed All hardware from PFC is present on the DFC Full Layer 2, Layer 3, ACL/QoS information downloaded from Supervisor Ingress DFC performs all lookups locally

• Deterministic, highly scalable—Not flow-based • NOT just for local switching— destination interface irrelevant • DFCs always require Cisco IOS software and a switch fabric RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

36

Distributed Forwarding Cards • DFCs work in conjunction with specific supervisor DFC works with PFC2 on Supervisor 2 DFC3A/3B/3BXL works with PFC3 on Supervisor 720

• PFC/DFC “major” module version must be identical PFC/DFC “minor” module version mismatch supported in lowest common denominator mode Example: System with PFC3B and DFC3As runs in PFC3A mode

• DFC is optional daughter card for CEF256 modules • DFC3 is optional daughter card for CEF256/CEF720 modules Several flavors and form factors available

• WS-X6816-GBIC module REQUIRES either DFC or DFC3 • Local CPU for managing hardware tables • Use remote login module command to access DFC console Commands available on DFC console for troubleshooting use, under direction from Cisco TAC/escalation

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

37

Centralized Forwarding Supervisor Engine 32

L3/4 Engine

Red D

Classic Module B

L2 Engine 2 3 PFC3

Port ASIC

Port4 ASIC

DBUS RBUS

Classic Module A

Port ASIC 1

Port ASIC

Blue S

Source

S

Destination

D

Blue VLAN Red VLAN

Entire Packet Packet Header

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

38

Centralized Forwarding with Fabric Red D

Port ASIC

Supervisor Engine 720

L3/4 Engine

Port ASIC LCRBUS LCDBUS

L2 Engine 3 PFC3

720Gbps Switch Fabric

8Gbps

6 Fabric Interface

CEF256 Module B DBUS RBUS

2 Fabric 5 Interface 4

8Gbps

CEF256 Module A

LCDBUS LCRBUS

Source

S

Destination

D

Blue VLAN Red VLAN

Port ASIC 1

Port ASIC

Entire Packet Packet Header

Blue S RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

39

Distributed Forwarding Red D

Port ASIC

Port ASIC

DFC3

Supervisor Engine 720 5 Fabric Interface/ 720Gbps Replication Switch 20Gbps Engine Fabric

Layer 2 Engine

20Gbps

PFC3

L3/4 Engine

CEF720 Module B w/DFC3

4 Fabric Interface/ 2 Replication Engine Port ASIC

Port ASIC

Layer 2 3Engine

CEF720 Module A w/DFC3

Source

S

Destination

D

Blue VLAN Red VLAN

DFC3

1

L3/4 Engine

Entire Packet Packet Header

Blue S RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

40

Agenda • Chassis Architecture • Supervisor Engine and Switch Fabric Architecture • Switching Module Architecture • Layer 2 Forwarding • IPv4 Forwarding • IPv4 Multicast Forwarding • Security and Feature ACLs • QoS • NetFlow

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

41

Layer 2 Forwarding

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

42

Layer 2 Lookups Frame received

Input Layer 2 lookup Layer 2 Table

Ingress Forwarding Engine

FIB TCAM FIB lookup ACL TCAM

Router MAC?

Yes

No

Input QoS lookup

Input ACL lookup

QoS TCAM

NetFlow Table NetFlow lookup

Input QoS lookup QoS TCAM Input ACL lookup ACL TCAM

Bridged NetFlow NetFlow Table

RST-3465 12523_04_2006_c1

Output QoS lookup

Output ACL lookup

Output Layer 2 lookup

QoS TCAM

ACL TCAM

Layer 2 Table

Output QoS lookup

Output ACL lookup

QoS TCAM

ACL TCAM

© 2006 Cisco Systems, Inc. All rights reserved.

Transmit frame Cisco Public

43

Layer 2 Forwarding • Layer 2 forwarding based on {VLAN, MAC} pairs Same MAC can be learned in multiple VLANs

• MAC learning fully hardware based CPU not involved in learning

• PFC and DFCs have copies of MAC table Refreshing of entries based on “seeing” traffic—forwarding engines age entries independently New learns on one forwarding engine communicated to other engines

• MAC table size: 128K entries on PFC2 (32K effective) 64K entries on PFC3 (32K effective)

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

44

Layer 2 Forwarding Logic Frame received

Layer 2 Table

Learn

SMAC lookup

DMAC lookup

New MAC?

Router MAC? Yes

Yes Layer 2 Table

Layer 2 Table

No

Update entry

L3 forwarding

No

L2 forwarding

Known MAC? Yes

Layer 2 Table

No

L2 flooding RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

45

Layer 2 Forwarding Table Design PFC2

PFC3

16 pages

4096 rows

16384 rows

MAC Table 8 pages RST-3465 12523_04_2006_c1

MAC Table

16K*8=128K entries

© 2006 Cisco Systems, Inc. All rights reserved.

4K*16=64K entries Cisco Public

46

PFC2 Layer 2 Lookup Frame 1

2 VLAN

16384 rows

Lookup Key

10 | 0000.aaaa.aaaa MAC Address

5 Compare

4000 3233.1111.3333 40| 20 | 0000.1111.2222 | 0000.cccc.cccc 111 | 9000.8000.7000 3999 | 9090.9090.9090 100 10| 0000.1111.1111 | 0000.bbbb.bbbb 2101 | 4334.5445.6556 44444 | 6666.6666.6666 | 0100.5e01.0101 10 | 0000.aaaa.aaaa HIT! 44 | 2468.ace0.2468 30 | 0000.dddd.dddd

3 Hash Function

8 pages © 2006 Cisco Systems, Inc. All rights reserved.

DMAC lookup

6 SMAC lookup

Update Entry

Starting Page and Row 4

RST-3465 12523_04_2006_c1

Destination interface(s)

MAC Table Cisco Public

47

PFC3 Layer 2 Lookup Frame 1

2 VLAN

Lookup Key

10 | 0000.aaaa.aaaa MAC Address DMAC lookup

Destination interface(s)

5 Compare

3 Hash Function

20 | 0000.cccc.cccc

MAC Table Row

4

4096 rows

10 | 0000.bbbb.bbbb HIT!

10 | 0000.aaaa.aaaa

6

30 | 0000.dddd.dddd 16 pages

MAC Table Update Entry RST-3465 12523_04_2006_c1

SMAC lookup © 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

48

Displaying the Layer 2 Table • Cisco IOS: show mac-address-table • Catalyst OS: show cam 6509#show mac-address-table dynamic vlan 30 Codes: * - primary entry vlan

mac address

type

learn qos

ports

------+----------------+--------+-----+---+----------------------* 30 0003.a088.c408 dynamic Yes -- Fa3/18 *

30

0012.d949.04d2

dynamic

Yes

--

Gi5/1

*

30

0003.a08a.15f3

dynamic

Yes

--

Fa3/24

*

30

0090.a400.1850

dynamic

Yes

--

Fa3/14

*

30

0003.a08a.15f9

dynamic

Yes

--

Fa3/25

<…> 6509# RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

49

Agenda • Chassis Architecture • Supervisor Engine and Switch Fabric Architecture • Switching Module Architecture • Layer 2 Forwarding • IPv4 Forwarding • IPv4 Multicast Forwarding • Security and Feature ACLs • QoS • NetFlow

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

50

IPv4 Forwarding

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

51

IPv4 Lookups Frame received

Input Layer 2 lookup Layer 2 Table

Ingress Forwarding Engine

FIB TCAM FIB lookup ACL TCAM

Router MAC?

Yes

No

Input QoS lookup

Input ACL lookup

QoS TCAM

NetFlow Table NetFlow lookup

Input QoS lookup QoS TCAM Input ACL lookup ACL TCAM

Bridged NetFlow NetFlow Table

RST-3465 12523_04_2006_c1

Output QoS lookup

Output ACL lookup

Output Layer 2 lookup

QoS TCAM

ACL TCAM

Layer 2 Table

Output QoS lookup

Output ACL lookup

QoS TCAM

ACL TCAM

© 2006 Cisco Systems, Inc. All rights reserved.

Transmit frame Cisco Public

52

Hardware-Based CEF • Catalyst 6500 leverages existing software Cisco Express Forwarding (CEF) model • Supervisor 2, Supervisor 32, Supervisor 720 extend CEF to hardware • What is CEF, in a nutshell? Boil down the routing table = FIB table Boil down the ARP table = adjacency table

• FIB table contains IP prefixes • Adjacency table contains next-hop information

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

53

Hardware-Based CEF (Cont.) • Decouples control plane and data plane Forwarding tables built on control plane Tables downloaded to hardware for data plane forwarding

• Hardware CEF process: FIB lookup based on destination prefix (longest-match) FIB “hit” returns adjacency, adjacency contains rewrite information (next-hop) ACL, QoS, and NetFlow lookups occur in parallel and affect final result

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

54

FIB TCAM and Adjacency Entries FIB: • IPv4 entries logically arranged from most to least specific

172.20.45.1

• 0/0 default entry terminates unicast FIB entries

MASK (/32)

• Overall FIB hardware shared by IPv4 unicast

10.1.1.100 … 10.1.3.0

IF, MACs, MTU

10.1.2.0 MASK (/24)

IPv4 multicast IPv6 unicast

…

IPv6 multicast

10.1.0.0

MPLS

172.16.0.0

Adjacency table: • Hardware adjacency table also shared among protocols • Actual adjacency table entries are NOT shared RST-3465 12523_04_2006_c1

IF, MACs, MTU

© 2006 Cisco Systems, Inc. All rights reserved.

IF, MACs, MTU IF, MACs, MTU

MASK (/16) … 0.0.0.0

Adjacency Table

MASK (/0) FIB TCAM

Cisco Public

55

IPv4 FIB TCAM Lookup Generate Lookup Key

Compare 3

10.1.1.10 10.1.1.xx

2

10.1.1.2 10.1.1.3

1 2

10.1.1.4

3

10.10.0.10

4

10.10.0.100

5

10.10.0.33

6

IF, MACs, MTU

10.100.1.1

7

10.100.1.2

8

10.1.2.xx

Load-Sharing IF, MACs, MTU Hash Offset 5 IF, MACs, MTU 6

10.1.3.xx

1 2

IF, MACs, MTU

10.10.100.xx

3

10.1.1.xx 10.100.1.xx

4 45

10.10.0.xx

6

10.100.1.xx

7

Lookup Key 1

DIP 10.1.1.10

FFFFFFFF

Packet

/32 entries (compare all bits)

FFFFFF00 HIT!

/24 entries (mask last octet)

Flow Data

Adj Index Result

Adjacency Table

8 RST-3465 12523_04_2006_c1

Masks

FIB TCAM

© 2006 Cisco Systems, Inc. All rights reserved.

Values

Cisco Public

56

Displaying IPv4 Forwarding Summary Information • Cisco IOS: show mls cef summary show mls cef statistics

6509-neb#show mls cef summary Total routes:

8309

IPv4 unicast routes:

5948

IPv4 Multicast routes:

2359

MPLS routes:

0

IPv6 unicast routes:

0

show mls cef

IPv6 multicast routes:

0

show mls

EoM routes:

0

show mls statistics show mls cef hardware

• Catalyst OS:

6509-neb#

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

57

Displaying Hardware IPv4 Prefix Entries 6509-neb#show mls cef Codes: decap - Decapsulation, + - Push Label Index

Prefix

Adjacency

64

127.0.0.51/32

receive

65

127.0.0.0/32

receive

66

127.255.255.255/32

receive

67

0.0.0.0/32

receive

68

255.255.255.255/32

receive

75

10.10.1.1/32

receive

76

10.10.1.0/32

receive

77

10.10.1.255/32

receive

78

10.10.1.2/32

Gi1/1,

3200

224.0.0.0/24

receive

3201

10.10.1.0/24

glean

3202

10.100.0.0/24

Gi1/1,

0030.f272.31fe

3203

10.100.1.0/24

Gi1/1,

0030.f272.31fe

3204

10.100.2.0/24

Gi1/1,

0030.f272.31fe

3205

10.100.3.0/24

Gi1/1,

0030.f272.31fe

• Cisco IOS: show mls cef • Catalyst OS: show mls entry cef ip

0030.f272.31fe

<…> RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

58

Displaying Detailed Hardware Entries • Cisco IOS: show mls cef <prefix> [detail] show mls cef adjacency [entry <entry> [detail]]

• Catalyst OS: show mls entry cef ip <prefix/mask> [adjacency] 6509-neb#show mls cef 10.100.20.0 detail <…> M(3222 ): E | 1 FFF 0 0 0 0 255.255.255.0 V(3222 ): 8 | 1 0 0 0 0 0 10.100.20.0

(A:98304

,P:1,D:0,m:0 ,B:0 )

6509-neb#show mls cef adjacency entry 98304 Index: 98304 smac: 000f.2340.5dc0, dmac: 0030.f272.31fe mtu: 1518, vlan: 1019, dindex: 0x0, l3rw_vld: 1 packets: 4203, bytes: 268992 6509-neb#

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

59

Finding the Longest-Match Prefix Entry • Cisco IOS: show

mls cef lookup [detail]

6509-neb#show mls cef 10.101.1.0 Codes: decap - Decapsulation, + - Push Label Index

Prefix

Adjacency

6509-neb#show mls cef lookup 10.101.1.0 Codes: decap - Decapsulation, + - Push Label Index

Prefix

Adjacency

3203

10.101.0.0/16

Gi2/12,

0007.b30a.8bfc

6509-neb#

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

60

IPv4 CEF Load Sharing • Up to 8 hardware load-sharing paths per prefix • Use maximum-paths command in routing protocols to control number of load-sharing paths

10.10.0.0/16 via Rtr-A via Rtr-B

• IPv4 CEF load-sharing is per-IP flow • Per-packet load-balancing NOT supported • Load-sharing based on Source and Destination IP addresses by default

A

B

“Unique ID” in PFC3 prevents polarization

• Configuration option supports inclusion of L4 ports in the hash

10.10.0.0/16

mls ip cef load-sharing full

• Unique ID not included in hash in “full” mode RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

61

Load-Sharing Prefix Entry Example • show mls cef • show mls cef lookup 6509-neb#show mls cef lookup 10.100.20.1 Codes: decap - Decapsulation, + - Push Label Index Prefix Adjacency 3222 10.100.20.0/24 Gi1/1, 0030.f272.31fe Gi1/2, 0008.7ca8.484c Gi2/1, 000e.382d.0b90 Gi2/2, 000d.6550.a8ea 6509-neb#

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

62

Identifying the Load-Sharing Path show mls cef exact-route

6509-neb#show mls cef exact-route 10.77.17.8 10.100.20.199

Interface: Gi1/1, Next Hop: 10.10.1.2, Vlan: 1019, Destination Mac: 0030.f272.31fe

6509-neb#show mls cef exact-route 10.44.91.111 10.100.20.199

Interface: Gi2/2, Next Hop: 10.40.1.2, Vlan: 1018, Destination Mac: 000d.6550.a8ea

6509-neb#

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

63

IPv4 Unicast RPF Check 6500 Routing Table Prefix

Next Hop

Interface

10.255.0.0/16

10.10.1.1 10.20.1.1 10.30.1.1 10.40.1.1 10.20.1.1

gig 1/1 gig 1/2 gig 2/1 gig 2/2 gig 6/3

g1/1 10.20.0.0/16

g1/2

10.255.0.0/16

10.20.0.0/16

g2/1 gig 6/3 g2/2

Supervisor 2:

Supervisor 720/Supervisor 32:

• One RPF interface per prefix in hardware • Enabling uRPF check halves available FIB TCAM (128K entries)

• • • •

Up to 6 RPF interfaces per prefix in hardware Enabling does not affect available FIB entries Two reverse-path interfaces for all prefixes Four user-configurable “multipath interface groups” to define additional interfaces for uRPF

Gotcha: System supports only a global uRPF mode—strict or loose—last configured mode overrides RST-3465 12523_04_2006_c1

Gotcha: uRPF with exception ACL not recommended due to software processing © 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

64

Verifying uRPF Check Configuration • show mls cef ip rpf [<prefix>] (PFC3 only) 6509#show mls cef ip rpf RPF global mode:

strict

RPF mpath mode:

punt

Index

Global uRPF check mode

Interfaces

-------+----------------------------------------

Global uRPF multipath mode

0 1

uRPF interface groups (not configured)

2 3 6509#show mls cef ip rpf 192.168.1.0 RPF information for prefix 192.168.1.0 uRPF check performed in the hardware for interfaces: Vlan776 Vlan777 uRPF check punted to software for interfaces:

uRPF details for specific IP prefix

uRPF check disabled for interfaces:

RST-3465 12523_04_2006_c1

6509# © 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

65

Agenda • Chassis Architecture • Supervisor Engine and Switch Fabric Architecture • Switching Module Architecture • Layer 2 Forwarding • IPv4 Forwarding • IPv4 Multicast Forwarding • Security and Feature ACLs • QoS • NetFlow

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

66

IPv4 Multicast Forwarding

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

67

IPv4 Multicast Lookups Frame received

Input Layer 2 lookup Layer 2 Table

Ingress Forwarding Engine

FIB TCAM FIB lookup ACL TCAM

Router MAC?

Yes

No

Input QoS lookup

Input ACL lookup

QoS TCAM

NetFlow Table NetFlow lookup

Input QoS lookup QoS TCAM Input ACL lookup ACL TCAM

Bridged NetFlow NetFlow Table

RST-3465 12523_04_2006_c1

Output QoS lookup

Output ACL lookup

Output Layer 2 lookup

QoS TCAM

ACL TCAM

Layer 2 Table

Output QoS lookup

Output ACL lookup

QoS TCAM

ACL TCAM

© 2006 Cisco Systems, Inc. All rights reserved.

Transmit frame Cisco Public

68

IPv4 Multicast Forwarding • Central and distributed IPv4 multicast hardware forwarding • Distributed multicast replication with appropriate switching modules† • PIM-SSM and PIM-SM forwarding in hardware • BiDir-PIM forwarding in hardware‡ • Off-loads majority of forwarding tasks from RP CPU

† Supervisor 2/SFM and Supervisor 720 only, with fabric-enabled modules ‡ Supervisor 32 and Supervisor 720 only RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

69

Multicast Forwarding Tables • RP CPU derives 3 key data structures from multicast routing table Multicast FIB—Consists of (S,G) and (*,G) entries, and RPF VLAN Adjacency table—Contains rewrite MAC and MET index Multicast Expansion Table (MET)—Contains output interface lists (OILs), i.e., lists of interfaces requiring replication

• RP CPU downloads tables to SP CPU • SP CPU installs tables in the appropriate hardware Multicast FIB and adjacency tables installed in PFC/DFC hardware MET installed in replication engines

• SP CPU also maintains L2 table for IGMP snooping

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

70

Multicast Hardware Entries • FIB

MAC, MET Index

IPv4 multicast entries arranged logically from most to least specific

• Adjacency table Different format than unicast Key piece of data is MET index

172.21.4.19, 225.3.3.3 10.1.44.199, 240.9.8.1

MAC, MET Index

10.1.1.1, 239.1.1.1

MAC, MET Index

… MASK (S,G) /32

MAC, MET Index

… *, 234.0.1.1

• MET Contains OILs for multicast routes Memory resident on replication engines (not PFC/DFC)

MASK BiDir Entries

…

Adjacency Table

10.1.1.0, 224.0.0.0 MASK IF 224/4 Entries

OIL #1

…

OIL #2

*, 229.0.1.1 MASK PIM-SM (*,G) /32

OIL #3 OIL #4

FIB TCAM RST-3465 12523_04_2006_c1

MET © 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

71

Multicast FIB TCAM Lookup Generate Lookup Key

Compare

10.1.1.10, 239.1.1.1

10.1.1.10, 239.1.1.1 10.1.1.10, 239.1.1.1

1 2

10.1.1.10, 239.1.1.1

3

10.1.1.10, 239.1.1.1

4

10.1.1.10, 239.1.1.1 HIT! 10.1.1.10, 239.1.1.1

5 4 6

10.1.1.10, 239.1.1.1

7

10.1.1.10, 239.1.1.1

8

Lookup Key 2

1

S,G 10.1.1.10, 239.1.1.1

FFFFFFFF FFFFFFFF

Multicast Packet

S,G compares all bits in SIP and GIP

3

FIB TCAM

Masks

Result Adj Index RPF VLAN

Values

Replication Engine(s)

MAC, MET Index OIL #1 6

OIL #2 OIL #3

MAC, MET Index 5 MAC, MET Index

OIL #4 MET RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

MAC, MET Index Adjacency Table

Cisco Public

72

Displaying Summary Hardware Multicast Information • Cisco IOS: show mls ip multicast summary

6506#show mls ip multicast summary

• show mls ip multicast statistics

Number of partial hardware-switched flows: 0

• Catalyst OS: show mls multicast

21210 MMLS entries using 3394656 bytes of memory

Number of complete hardware-switched flows: 21210

Directly connected subnet entry install is enabled Hardware shortcuts for mvpn mroutes supported Current mode of replication is Ingress Auto-detection of replication mode is enabled Consistency checker is enabled Bidir gm-scan-interval: 10 6506#

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

73

Displaying Hardware Multicast Forwarding Entries • Cisco IOS: show mls ip multicast • Catalyst OS: show mls multicast entry 6506#show mls ip multicast Multicast hardware switched flows: (10.3.1.100, 239.1.1.100) Incoming interface: Gi3/1, Packets switched: 720396460 Hardware switched outgoing interfaces: Gi3/2 Vlan100 Vlan150 Gi4/1 Gi4/2 Vlan200 RPF-MFD installed

(10.3.1.103, 230.100.1.1) Incoming interface: Gi3/1, Packets switched: 443201 Hardware switched outgoing interfaces: Gi3/2 Gi4/1 RPF-MFD installed <…>

For more details, attend: “RST-3262: Catalyst 6500 IP Multicast Architecture and Troubleshooting” RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

74

Agenda • Chassis Architecture • Supervisor Engine and Switch Fabric Architecture • Switching Module Architecture • Layer 2 Forwarding • IPv4 Forwarding • IPv4 Multicast Forwarding • Security and Feature ACLs • QoS • NetFlow

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

75

Security and Feature ACLs

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

76

ACL Lookups Frame received

Input Layer 2 lookup Layer 2 Table

Ingress Forwarding Engine

FIB TCAM FIB lookup ACL TCAM

Router MAC?

Yes

No

Input QoS lookup

Input ACL lookup

QoS TCAM

NetFlow Table NetFlow lookup

Input QoS lookup QoS TCAM Input ACL lookup ACL TCAM

Bridged NetFlow NetFlow Table

RST-3465 12523_04_2006_c1

Output QoS lookup

Output ACL lookup

Output Layer 2 lookup

QoS TCAM

ACL TCAM

Layer 2 Table

Output QoS lookup

Output ACL lookup

QoS TCAM

ACL TCAM

© 2006 Cisco Systems, Inc. All rights reserved.

Transmit frame Cisco Public

77

Security ACLs • Enforce security policies based on Layer 2, Layer 3, and Layer 4 information • Dedicated ACL TCAM ensures security ACLs do not affect system performance • Router ACL (RACL)—Enforced for all traffic crossing a Layer 3 interface in a specified direction IPv4, IPX†, IPv6‡ RACLs supported

• VLAN ACLs (VACLs)—Enforced for all traffic in the VLAN IPv4, IPX†, MAC VACLs supported

• Port ACLs (PACLs)††—Enforced for all traffic input on a Layer 2 interface IPv4, MAC PACLs supported † IPX ACLs in Supervisor 2 only ‡ IPv6 ACLs on Supervisor 720 and Supervisor 32 only †† PACLs in Supervisor 720 and Supervisor 32 in CatOS only RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

78

Feature ACLs • Classify traffic that requires additional or special handling Policy-Based Routing (PBR) Reflexive ACLs Network Address Translation (NAT/PAT) WCCP redirection

• Programmed in ACL TCAM to preserve performance • Override FIB forwarding decision to allow alternative processing • Typically paired with NetFlow table and/or Adjacency table

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

79

ACL Merge • Sophisticated feature merge algorithm allows multiple security and feature ACLs to be applied to a single interface/VLAN • What is merging? PFC/DFC hardware supports limited number of ACL lookups on a single packet May need two or more ACL features on a single interface (e.g., RACL and PBR) Merge produces ACEs that return correct result in a single lookup

• Downside: Can cause TCAM blowup ACE intersection/interrelations can require lots of TCAM entries

• Two algorithms: ODM and BDD (Supervisor 2 only) • If using Supervisor 2, USE ODM! (mls aclmerge algorithm odm) • PFC3 dual-bank TCAM architecture can avoid merge entirely

White Paper on ACL Merge Algorithms and ACL Hardware Resources: http://www.cisco.com/warp/public/cc/pd/si/casi/ca6000/tech/65acl_wp.pdf RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

80

ACL TCAM Entry Population Protocol

Dest IP

Source IP 00000000 FFFFFFFF 00 0000 0000

Dest Port

xxxxxxxx 10.1.2.100 xx xxxx xxxx

1

Permit

xxxxxxxx 10.1.68.101 xx xxxx xxxx

2

Deny

xxxxxxxx 10.33.2.25 xx xxxx xxxx

3

Deny

4

Source Port

5 6

1=“Compare” 0=“Mask”

7 8

ip access-list extended example

xxxxxxxx xxxxxxxx 06 xxxx 0016

1

Permit

permit ip any host 10.1.2.100

xxxxxxxx xxxxxxxx 06 xxxx 0017

2

Deny

xxxxxxxx xxxxxxxx 11 xxxx 0202

3

Deny

xxxxxxxx xxxxxxxx 06 xxxx 0080

4

Permit

xxxxxxxx xxxxxxxx 11 xxxx 00A1

5

Permit

deny ip any host 10.1.68.101 00000000 00000000 FF 0000 FFFF

deny ip any host 10.33.2.25 permit tcp any any eq 22 deny tcp any any eq 23 deny udp any any eq 514

6

permit tcp any any eq 80

7

permit udp any any eq 161

8

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Masks

Values

Cisco Public

81

ACL TCAM Lookup Generate Lookup Key

Compare 3 10.1.1.10 xxxxxxxx xxxxxxxx | 10.1.2.11 xxxxxxxx 10.1.2.11| 06 xx 06|xxxx xxxx 84C8xxxx 0050 | 0050

2

Lookup Key

1

00000000 FFFFFFFF 00 0000 0000

SIP=10.1.1.10 DIP=10.1.2.11 Protocol=TCP (6) SPORT=33992 DPORT=80

xxxxxxxx 10.1.2.100 xx xxxx xxxx

1

xxxxxxxx 10.1.68.101 xx xxxx xxxx

2

xxxxxxxx 10.33.2.25 xx xxxx xxxx

3 4 5 6

Entries matching only destination IP

Packet

7 8

ip access-list extended example

xxxxxxxx xxxxxxxx 06 xxxx 0016

1

permit ip any host 10.1.2.100

xxxxxxxx xxxxxxxx 06 xxxx 0017

2

xxxxxxxx xxxxxxxx 11 xxxx 0202

3

HIT! xxxxxxxx xxxxxxxx 06 xxxx 0050

4 45

deny ip any host 10.1.68.101 00000000 00000000 FF 0000 FFFF

deny ip any host 10.33.2.25 permit tcp any any eq 22

xxxxxxxx xxxxxxxx 11 xxxx 00A1

deny tcp any any eq 23 deny udp any any eq 514 permit tcp any any eq 80

Entries matching only protocol and destination port

6

© 2006 Cisco Systems, Inc. All rights reserved.

Result

7

permit udp any any eq 161 RST-3465 12523_04_2006_c1

Permit

8 Masks

Values

Cisco Public

82

Monitoring ACL TCAM Utilization 6509 -

• Cisco IOS: show tcam counts

neb#show tcam counts Used

Free

Percent Used

Reserved

- - --

- ---

- - ----------

- - ------

23

4073

0

Masks:

2902

1194

70

72

Entries:

15261

17507

46

576

Masks:

7

4089

0

18

Entries:

32

32736

0

144

LOU:

47

81

36

ANDOR:

1

15

6

ORAND:

0

16

0

ADJ:

0

2048

0

Labels:

• Catalyst OS: show security acl resource-usage

ACL_TCAM --------

QOS_TCAM --------

6509 RST-3465 12523_04_2006_c1

neb# © 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

83

Verifying Hardware ACL Enforcement • show fm summary 6509-neb#show fm summary Interface: Vlan199 is up TCAM screening for features: ACTIVE inbound Interface: Vlan400 is up TCAM screening for features: ACTIVE inbound TCAM screening for features: ACTIVE outbound Interface: Vlan402 is up TCAM screening for features: ACTIVE inbound TCAM screening for features: ACTIVE outbound Interface: Vlan404 is up TCAM screening for features: INACTIVE inbound Interface: Vlan405 is up TCAM screening for features: ACTIVE inbound 6509-neb# fm = “Feature Manager” ACTIVE = ACL policy is installed in hardware INACTIVE = ACL policy is NOT installed in hardware RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

84

Displaying Hardware ACL “Hit Counters” Cisco IOS: show tcam interface acl {in | out} ip 6509-neb#show tcam interface vlan199 acl in ip <…> permit

udp any 10.89.210.0 0.0.0.255 (234265 matches)

permit

udp any 10.90.143.0 0.0.0.255 (6860 matches)

permit

udp any 10.91.25.0 0.0.0.255 (23 matches)

permit

udp any 10.92.82.0 0.0.0.255 (23662 matches)

permit

udp any 10.93.154.0 0.0.0.255 (3232 matches)

permit

udp any 10.94.1.0 0.0.0.255 (12113 matches)

permit

udp any 10.95.109.0 0.0.0.255 (247878 matches)

permit

udp any 10.96.201.0 0.0.0.255 (33234 matches)

permit

udp any 10.97.16.0 0.0.0.255 (6855 matches)

permit

udp any 10.98.43.0 0.0.0.255 (89745 matches)

permit

udp any 10.1.1.0 0.0.0.255 (7893485 matches)

deny

ip any any (448691555 matches)

6509-neb# ACL Hit Counters Supported on PFC3B/BXL Only! RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Global or per-ACL entry (use [no] mls acl tcam share-global to toggle) Cisco Public

85

Agenda • Chassis Architecture • Supervisor Engine and Switch Fabric Architecture • Switching Module Architecture • Layer 2 Forwarding • IPv4 Forwarding • IPv4 Multicast Forwarding • Security and Feature ACLs • QoS • NetFlow

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

86

QoS

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

87

Catalyst 6500 QoS Model

Receive Interface

Input Queue Schedule QoS Actions at Ingress Port ASIC

RST-3465 12523_04_2006_c1

Classify

Mark

Ingress Police

QoS Actions at PFC/DFC

© 2006 Cisco Systems, Inc. All rights reserved.

Egress Police

Congestion Avoidance

Output Queue Schedule

QoS Actions at Egress Port ASIC

Transmit Interface

Cisco Public

88

QoS Lookups Frame received

Input Layer 2 lookup Layer 2 Table

Ingress Forwarding Engine

FIB TCAM FIB lookup ACL TCAM

Router MAC?

Yes

No

Input QoS lookup

Input ACL lookup

QoS TCAM

NetFlow Table NetFlow lookup

Input QoS lookup QoS TCAM Input ACL lookup ACL TCAM

Bridged NetFlow NetFlow Table

RST-3465 12523_04_2006_c1

Output QoS lookup*

Output ACL lookup

Output Layer 2 lookup

QoS TCAM

ACL TCAM

Layer 2 Table

Output QoS lookup*

Output ACL lookup

QoS TCAM

ACL TCAM

© 2006 Cisco Systems, Inc. All rights reserved.

Transmit frame *PFC3 only

Cisco Public

89

Classification • Selects traffic for further QoS processing Marking Policing

• Based on— Port trust QoS ACLs

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

90

QoS ACLs • Support standard and extended IPv4, IPv6,† and MAC ACLs for classification • Use QoS TCAM to classify traffic for marking and policing • Leverage dedicated QoS TCAM 32K entries/4K masks

• Share other resources (LOUs and labels) with security ACLs

† PFC3 only RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

91

QoS ACL Lookup Results • QoS TCAM lookups behave exactly the same as ACL TCAM lookups • But, returned result differs: Index into Aggregate table (identifies aggregate policer to use) Index into Microflow table (identifies microflow policer to use) Remarked DSCP/IP precedence value

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

92

Marking • Untrusted port—Set a default QoS value • Trusted port—Use the marking (COS, precedence, DSCP) provided by upstream device • QoS ACLs / service-policies—Set QoS values based on standard or extended ACL match

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

93

Policing • Enforces a policy on a port or VLAN for traffic matching classification policy Markdown Police (drop)

• Two types of policers: Aggregate Microflow

• Based on a classic token bucket scheme Add tokens to bucket at constant rate (equivalent to policed rate) Packets are “in profile” if enough tokens exist in the bucket to transmit the packet Packets without adequate tokens are dropped or marked down

• Note! PFC2 uses Layer 3 packet size, PFC3 uses Layer 2 frame size, when determining rate RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

94

Aggregate Policing • Bandwidth limit applied cumulatively to all flows that match the associated class Example—All FTP flows in a VLAN limited in aggregate to configured rate

• Ingress policing performed on per-switchport, per-Layer 3 interface, or per-VLAN basis PFC2 and PFC3 both support ingress policing

• Egress policing on a performed on per-Layer 3 interface or perVLAN basis NOT possible on a per-switchport basis PFC3 support only

• Dual-rate policers allow for combined markdown and drop policies Normal rate and excess rate are configurable PFC3 support only

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

95

Microflow Policing • Bandwidth limit applied separately to each individual flow that matches the associated class Every individual FTP flow limited to configured rate

• User-based rate limiting using source-only and destination-only flow masks All FTP from a given source IP limited to configured rate PFC3 only

• Leverages NetFlow table • Microflow policing performed on ingress only

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

96

Remarking Traffic with Policers • Policing action may remark certain traffic For example, transmit with marked-down DSCP

• Dual-rate aggregate policer can mark-down traffic exceeding the normal rate and drop traffic exceeding the excess rate • Use markdown maps to configure marked-down DSCP values mls qos map policed-dscp (Cisco IOS) or set qos policed-dscp-map (CatOS)

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

97

Monitoring Service Policies (Marking and Policing) • Cisco IOS: show policy-map interface*

6506#show policy- map interface vlan 100 Service class -

policy input: VLAN map: NET -

Match: access -

100

44 - TCP (match- all) group name POL -

44 - TCP

police : 100000000 bps 100000 limit 100000 extended limit Earl in slot 6 :

Policed Class

2940073472 bytes

• Catalyst OS: show qos statistics {aggregatepolicer | l3stats}

5 minute offered rate 358172704 bps aggregate -

forwarded 608631808 bytes action: transmit

exceeded 2331441664 bytes action: drop aggregate class -

map: NET -

Match: access -

forward 100352000 bps exceed 384495616 bps 55 (match -

all)

group name MARK -

55

set precedence 5:

Marked Class

Earl in slot 6 : 2940069888 bytes 5 minute offered rate 358172616 bps aggregate -

RST-3465 12523_04_2006_c1

forwarded 2940069888 bytes

6506# © 2006 Cisco Systems, Inc. All rights reserved.

* Shows aggregate policer stats only; Cisco Publicpolicing 98 use NetFlow table to monitor microflow

Agenda • Chassis Architecture • Supervisor Engine and Switch Fabric Architecture • Switching Module Architecture • Layer 2 Forwarding • IPv4 Forwarding • IPv4 Multicast Forwarding • Security and Feature ACLs • QoS • NetFlow

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

99

NetFlow

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

100

NetFlow Lookups Frame received

Input Layer 2 lookup Layer 2 Table

Ingress Forwarding Engine

FIB TCAM FIB lookup ACL TCAM

Router MAC?

Yes

No

Input QoS lookup

Input ACL lookup

QoS TCAM

NetFlow Table NetFlow lookup

Input QoS lookup QoS TCAM Input ACL lookup ACL TCAM

Bridged NetFlow NetFlow Table

RST-3465 12523_04_2006_c1

Output QoS lookup

Output ACL lookup

Output Layer 2 lookup

QoS TCAM

ACL TCAM

Layer 2 Table

Output QoS lookup

Output ACL lookup

QoS TCAM

ACL TCAM

© 2006 Cisco Systems, Inc. All rights reserved.

Transmit frame Cisco Public

101

IPv4 NetFlow • Tracks statistics for traffic flows through the system • Entries created in NetFlow table when new flows start Flow mask determines format of entries

• Entries removed when flows expire Timer and session based expiration

• Full collection by default when NetFlow enabled Also support time- and packet-based NetFlow sampling

• Flow statistics can be exported using NetFlow Data Export (NDE) Supported export formats include NetFlow v5 and v7 NetFlow v9 export format supported in Supervisor 720 and Supervisor 32

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

102

Displaying NetFlow Statistics Entries • Cisco IOS: show mls netflow ip • Catalyst OS: show mls statistics entry

Which fields are populated depends on the configured flow mask

6506#show mls netflow ip Displaying Netflow entries in Supervisor Earl DstIP

SrcIP

Prot:SrcPort:DstPort

Src i/f

:AdjPtr

--------------------------------------------------------------------------Pkts

Bytes

Age

LastSeen

Attributes

--------------------------------------------------10.102.130.213 7

3766

10.230.215.148 25

9

47

10.17.64.177 7664

10.90.33.185

17

10.155.22.221

21329

10.97.36.200

2569654

10.214.39.79

17

10.46.13.211 1269409076

17

tcp :46528 15:47:37 tcp :51813 15:47:39 tcp :65211 15:47:38 tcp :27077 15:47:38

:www L3-

Vl39 Dynamic

:45912 L3-

Vl144

:0x0

:0x0

Dynamic

:60425 L3-

Vl22

Dynamic

:www L3-

:0x0

Vl13

:0x0

Dynamic

<…> RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

103

NetFlow Table Utilization • PFC2 NetFlow table contains 128K entries Hash ~25% efficient (32K entries) Probability of collision increases after 32K entries

• PFC3 NetFlow table size varies • PFC3A/B—128K entries • PFC3BXL—256K entries

Hash ~50–90% efficient (64/96/230K entries for PFC3A/B/BXL) Probability of collision increases after 64K/96K/230K entries Alias CAM handles hash collision cases RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

104

PFC2 NetFlow Table Architecture Packet 1 Flow Key 2 | 1044 | 80 SIP 10.10.20.1 DIP | 10.20.2.2 Proto | 6SPort DPort

16K rows

5 Compare 3 Hash Function

172.16.8.2 | 192.168.1.2 | 6 | 1025 | 80 172.16.1.1 | 172.16.2.2 17 | 2334 | 23 10.1.1.1 | 10.1.1.2 | 6 | 1030 || 80 10.10.10.1 6 | 2334 | 80 10.1.1.1 | 10.1.1.2| |10.20.1.1 6 | 1030 || 80 10.1.1.1 | 10.1.1.2 | 6 | 1030 | 80 |0|0 10.1.1.1 | 10.1.1.2 |80 6 | 1030| 1 | 80 10.1.1.2 |192.168.1.1 10.1.1.1 | 6 || 10.1.1.2 10.4.4.4 | 172.16.8.8 17 | 1025 10.1.1.1 | 10.1.1.2 6 | |1030 | 80 | 514 10.1.1.2 | 10.1.1.1 | 6 | |80 10.1.1.1 | 10.1.1.2 | 6 | 1030 | 80 10.1.1.2 | 10.1.1.1 | 6 | 80 10.1.1.1 | 239.1.1.1 10.1.1.2 | 10.1.1.1 | 6 | 17 80 | 5000 1030 | 5000 10.10.20.1 | 10.20.2.2 | 1044 | 80 10.1.1.2 | 10.1.1.1 | 6 | 80| 6 | 1030 10.1.1.2 | 10.1.1.1 | 6 | 80 | 1030 HIT! 6 10.1.1.2 | 10.1.1.1 | 6 | 22 | 3245 10.99.1.1 | 10.99.100.1 | 6 | 4444 | 25

Update Statistics

10.99.100.1 | 10.4.5.6 | 6 | 25 | 1080

Starting Page 4 and Row

8 pages RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

NetFlow Table

Cisco Public

105

PFC3 NetFlow Lookups Packet 1 Mask

2 Flow Flow Key Key

HIT!

3 Hash Function

Key Key Key Key Key Key Key Key Key Key Key

Compare

5

NetFlow Table Index Result

Mask Hash Key Key Hash

4 Compare

6

Flow Data Flow Data Flow Data Flow Data HIT! Flow Data 7 Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data

128K/256K entries

Netflow TCAM

Statistics

128K/256K rows

Netflow Table

Key 128 entries RST-3465 12523_04_2006_c1

Alias CAM

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

106

Monitoring NetFlow Table Usage • Cisco IOS: show mls netflow table-contention • Catalyst OS: show mls debug 6506#show mls netflow table -

contention detailed

Earl in Module 6 Detailed Netflow CAM (TCAM and ICAM) Utilization ================================================ TCAM Utilization

:

100%

ICAM Utilization

:

82%

Netflow TCAM count

:

131072

Netflow ICAM count

:

105

Netflow Creation Failures

:

3432605

Netflow CAM aliases

:

8

6506#show mls netflow table -

contention aggregate

Current utilization

Clear on read

Earl in Module 6 Aggregate Netflow CAM Contention Information =============================================

RST-3465 12523_04_2006_c1

Netflow Creation Failures

:

222917949

Netflow Hash Aliases

:

834

6506# © 2006 Cisco Systems, Inc. All rights reserved.

Cumulative Cisco Public

107

NetFlow Aging • Process of removing stale NetFlow entries • Types of aging Normal—Fixed idle time for flows Fast—Threshold-based aging of flows Long—Maximum lifetime for flows Session-based—Based on TCP FIN/RST flags

• Default timers are conservative Tuning is recommended! Start with more aggressive normal aging timer— Reduce until no creation failures seen or CPU is at threshold Enable fast aging to remove short-lived flows—Adjust until creation failures cease or CPU is at threshold RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

108

Changing and Viewing the NetFlow Aging Configuration • Cisco IOS: mls aging {normal | fast | long} show mls netflow aging

• Catalyst OS: set mls agingtime [fast | long-duration] show mls 6506#show mls netflow aging enable timeout

packet threshold

------ -------

----------------

normal aging true

300

N/A

fast aging

false

32

100

long aging

true

1920

N/A

6506# RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

109

Conclusion • You should now have a thorough understanding of the Catalyst 6500 switching architecture, packet flow, and key forwarding engine functions… ANY QUESTIONS?

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

110

Related Networkers Sessions • RST-3262: IP Multicast Architecture and Troubleshooting for the Cisco Catalyst 6500 Series • RST-3143: Troubleshooting Catalyst 6500 Series Switches • RST-2031: Multilayer Campus Architectures and Design Principles • RST-3466: Cisco IOS Software Modularity—Architecture and Deployment • TECRST-3101: Troubleshooting Cisco Catalyst Switches • TECRST-2001: Enterprise High Availability • BoF-06: Enterprise Switching

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

111

Q and A

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

112

Recommended Reading • Continue your Cisco Networkers learning experience with further reading from Cisco Press • Check the Recommended Reading flyer for suggested books

Available Onsite at the Cisco Company Store RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

113

Complete Your Online Session Evaluation • Win fabulous prizes; Give us your feedback • Receive ten Passport Points for each session evaluation you complete • Go to the Internet stations located throughout the Convention Center to complete your session evaluation • Drawings will be held in the World of Solutions Tuesday, June 20 at 12:15 p.m. Wednesday, June 21 at 12:15 p.m. Thursday, June 22 at 12:15 p.m. and 2:00 p.m.

RST-3465 12523_04_2006_c1

© 2006 Cisco Systems, Inc. All rights reserved.

Cisco Public

114

RST-4501 11366_06_2005_x

© 2005 Cisco Systems, Inc. All rights reserved.

115