Brkcrs-3147-advanced-troubleshooting-of-the-asr1k-and-asr4400-made-easy-2014-milan-90-mins.pdf

ASR1K and ISR445x Troubleshooting Made Easy BRKCRS-3147

Frederic Detienne

Agenda           

Platform and Hardware Architecture Software Architecture Day in the the Life of a Normal Packet Advanced Example: IPsec Control Plane Programming Debugging strategies Road to Simplification: Part I, Data Plane Debugging Understanding and Extracting ESP Logs Road to Simplification: Part II, Control Plane Unified Show Commands Road to Simplification: Part III, Deep Data Plane Debugging Future: Resource Consumption Monitoring Wrapping up... BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

3

Session Objectives  Understand the ASR 1K and ISR 445x architecture – software – hardware – relationship between the two

 Understand how features process packets through IOS-XE  Understand how to easily debug the platform – long journey – presentation of recent serviceability enhancements

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

4

Platforms and Hardware Architecture

Cisco ASR 1000 Series Routers: Overview Compact, Powerful Router

Instant-on Services Delivery Instant On Service Delivery

Business-Critical Resiliency

 Line-rate performance 2.5G to 200G+ with services enabled

 Fully separated control and forwarding planes

 Integrated firewall, VPN, encryption, DPI, CUBE

 Investment protection with modular engines, IOS CLI and SPAs for I/O

 Hardware and software redundancy

 Hardware based QoS engine with up to 472K queues

 In-service software upgrades

 Scalable on-chip service provisioning through software licensing

One IOS-XE Feature Set ASR 1013

ASR 1001

2.5-5 Gbps

ASR 1002-X

5-36 Gbps BRKCRS-3147

ASR 1004

10-40 Gbps © 2014 Cisco and/or its affiliates. All rights reserved.

ASR 1006

10-100 Gbps

40-200 Gbps Cisco Public

6

Chassis Options: ASR 1002-X

4 x 1GE

SPAs

ESP

2RU RP/SI P

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

7

Chassis Options: ASR 1004 SPAs

SIP

4RU

ESP

RP

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

8

Rack Mount & Cable Management

Chassis Options: ASR 1006 SPAs

SIP 6RU ESP

RP

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

9

ASR1K Building Blocks RP

ESP

RP

Active

interconn.

QFP Crypto Assist.

GE switch

interconn.

FECP

GE switch

QFP

PPE

QFP Crypto Assist.

BQS

Route Processor Handles control plane traffic Manages system

interconn. Embedded Service Processor Handles forwarding plane traffic

SIP

CPU

Stby

CPU

Stby

Active

FECP

ESP

PPE

BQS

interconn.

Midplane

SIP

interconn.

SIP

interconn.

interconn.

SPA Aggreg.

IOCP

SPA Aggreg.

IOCP

SPA Aggreg.

IOCP

SPA

SPA

SPA

SPA

SPA

SPA

SPA Interface Processor Houses SPA’s Queues packets in & out (FIFO) BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

10

System Architecture Control Plane RP

ESP

RP

Active

interconn.

QFP Crypto Assist.

PPE

CPU

Stby

CPU

GE switch

BQS

interconn.

FECP

Stby

Active

FECP

ESP

GE switch

QFP Crypto Assist.

EOBC switch in RP

interconn.

PPE

BQS

interconn.

Midplane

Inter Integrated Circuit (I2C) Bus SIP Slow (few kbps) Used for system monitoring (temp., OIR, fan speed,…)

SIP

interconn.

SIP

interconn.

Ethernet Out of Band Channel (aka EOBC) 1Gbps Ethernet bus Used by RP to program system Used by system to notify RP

interconn.

SPA Aggreg.

IOCP

SPA Aggreg.

IOCP

SPA Aggreg.

IOCP

SPA

SPA

SPA

SPA

SPA

SPA

SPA Control Link Works between the SPA’s and SIP

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

11

System Architecture Forwarding Plane Hypertransport 10 Gbps Ethernet RP

ESP

RP

Active

interconn.

QFP Crypto Assist.

PPE

GE switch

GE switch

BQS

QFP

Embedded Service Interconnect aka ESI Bus 11.2 – 40 Gbps Forwarding Bus

PPE

BQS

interconn.

Midplane

Centralized Architecture All traffic flows through ESP

BRKCRS-3147

interconn.

FECP

Crypto Assist.

interconn.

SIP

CPU

Stby

CPU

Stby

Active

FECP

ESP

SIP

interconn.

SIP

interconn.

interconn.

SPA Aggreg.

IOCP

SPA Aggreg.

IOCP

SPA Aggreg.

IOCP

SPA

SPA

SPA

SPA

SPA

SPA

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

12

RP

CPU

Route Processor Architecture interconn.

Highly Scalable Control Plane Processor Route Processor Manages all chassis functions Runs IOS

Not a traffic interface! Management only

USB

Mgmt Ethernet

GE switch

System Logging Core Dumps

Console & Aux

BITS (input & output)

2.5’’ Hard disk

Card Infrastructure Runs IOS, Linux OS Manages boards and chassis IOS Memory: RIB, FIB & other processes Determines BGP routing table size RP1: 4GB RP2: 8&16GB

NVRAM

CPU Memory

CPU

RP1: 1GB RP2: 2GB

Bootdisk

(1.5 – 2.66 GHz Dual-core)

Stratum-3 Network clock circuit I2C

Chassis Management Bus

SIPs ESPs RP

BRKCRS-3147

33MB

ESI Interconnect

Misc Ctrl

ESPs

EOBC Gig Eth Switch

SIPs

ESPs

© 2014 Cisco and/or its affiliates. All rights reserved.

Output clocks

RP

SIPs

Input clocks

SIPs

Cisco Public

RP

GE, 1Gbps I 2C SPA Control SPA Bus ESI, 11.2-40 Gbps SPA-SPI, 11.2Gbps Hypertransport, 10Gbps Other

13

ESP FECP

ESP10 Block Diagram

QFP

Crypto Assist.

PP E

BQ S

intercon.

Reset / Pwr Ctrl

TCAM

Resource DRAM

(10Mbit)

(512MB)

Packet Buffer DRAM (128MB)

Temp Sensor EEPROM

QFP Packet Processor Engine

DDRAM Boot Flash (OBFL,…)

FECP

BQS

PPE1

PPE2

PPE3

PPE4

PPE5

PPE6

PPE7

PPE8

…

PPE40

E-CSR

JTAG Ctrl

PCI*

Part Len / BW SRAM

E-RP*

Dispatcher Packet Buffer

GE, 1Gbps I2C SPA Control SPA Bus

Crypto (Nitrox-II CN2430)

Reset / Pwr Ctrl

RPs

SA table DRAM

RPs BRKCRS-3147

SPI Mux

Interconnect

ESP

ESI, 11.2Gbps SPA-SPI, 11.2Gbps Hypertransport, 10Gbps Other

Interconnect

RPs

© 2014 Cisco and/or its affiliates. All rights reserved.

SIPs Cisco Public

14

ESP FECP

ESP10 Block Diagram (comments) Forwarding Engine Control ProcessorReset / Pwr Ctrl Manages board Programs QBS, Crypto TempPPE, Sensor Linux Kernel

(10Mbit)

QFP Packet Processor Engine

DDRAM Boot Flash (OBFL,…)

FECP

BQS

PPE1

PPE2

PPE3

PPE4

PPE5

PPE6

PPE7

PPE8

…

PPE40

E-CSR

JTAG Ctrl

PCI*

GE, 1Gbps I2C SPA Control SPA Bus

Crypto Reset / Pwr Ctrl

RPs

SA table DRAM

RPs BRKCRS-3147

Buffering Queuing & Scheduling Executes complex QoS scheduling (shapers, LLQ’s,…) Queues and schedules packets in due time

E-RP*

Dispatcher Packet Buffer

(Nitrox-II CN2430)

BQ S

Part Len / BW SRAM

(128MB)

EEPROM

PP E

intercon.

Quantum Flow Processor Subsystem Responsible for forwarding packets Packet Buffer Resource DRAM DRAM (512MB)

TCAM

QFP

Crypto Assist.

SPI Mux

Interconnect

ESP

ESI, 11.2Gbps SPA-SPI, 11.2Gbps Hypertransport, 10Gbps Other

Interconnect

RPs

© 2014 Cisco and/or its affiliates. All rights reserved.

SIPs Cisco Public

15

ESP200 Block Diagram TCAM (80Mbit)

Resource DRAM (2GB)

Reset / Pwr Ctrl

Packet Buffer DRAM (512MB)

QFP Packet Processor Engine

Temp Sensor

QFP BQS

Packet Processor Engine

PPE PPE PPE PPE PPE 1

EEPROM

2

3

PPE PPE PPE 6

7

4

5

…

PPE

8

DDRAM

1

2

FECP

6

7

4

5

…

PPE

8

40

Dispatcher Packet Buffer Resource DRAM (2GB)

JTAG Ctrl

Packet Buffer DRAM (512MB)

Packet Buffer DRAM (512MB)

Resource DRAM (2GB)

QFP Packet Processor Engine 1

2

3

PPE PPE PPE 6

7

4

5

…

PPE

8

BQS

Packet Processor Engine

Memory

1

Dispatcher

Memory

RPs BRKCRS-3147

2

3

PPE PPE PPE

40

Crypto

BQS

PPE PPE PPE PPE PPE

6

Dispatcher Packet Buffer

Reset / Pwr Ctrl

GE, 1Gbps I 2C

QFP

PPE PPE PPE PPE PPE

RPs

3

PPE PPE PPE

40

TCAM (80Mbit)

BQS

PPE PPE PPE PPE PPE

Dispatcher Packet Buffer

Boot Flash (OBFL,…)

Packet Buffer DRAM (512MB)

Resource DRAM (2GB)

7

4

5

…

PPE

8

SPA Control SPA Bus ESI, 11.5 or 23Gbps SPA-SPI, 11.2Gbps

Hypertransport, 10Gbps Other

40

Dispatcher Packet Buffer

Interconnect

Pkt Reorder Logic

Crypto ESP RPs

© 2014 Cisco and/or its affiliates. All rights reserved.

SIPs

Cisco Public

16

ESI Capacity by ESP-xxx and SIP-xxx  Enhanced SerDes Interconnect (ESI) links over midplane carry

QFP Complex 11.2Gbps SPI4.2

ESP-10G Interc.

11.2Gbps SPI4.2

25.6Gbps eSPI

40+G I/L

– – – –

packets between ESP and other cards (SIPs, RP & other ESP) network traffic to/from SPA SIP’s punt/inject traffic to/from RP state synchronization to/from standby

 Additional full set of ESI links to/from standby ESP (not shown)

ESP-10G Interc.

 CRC protection of packet contents

ESP-20G Interconnect

 ESP-10G: 1x11.5G ESI to each SIP slot

ESP-40 G Interconnect

 ESP-20G: 2x11.5G ESI to two SIP slots; 1x11.5G to third SIP slot  ESP-40G: – –

–

2x23G ESI* to all three SIP slots could also support a 6-SIP chassis with 1 ESI to each (e.g. voice application) also 23G between two ESP-40G’s

 SIP-10G: supports 1x11.5G mode only  SIP-40G: supports 1x11.5G, 2x11.5G, 2x23G Other ESP

RP1

RP0

SIP0

SIP1 ASR1004

BRKCRS-3147

SIP2 ASR1006

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

17

Embedded Services Processor – The Real Thing Interconnect ASIC

SPI MUX TCAM

Crypto Engine

FECP CPU

QFP Subsystem PPE + BQS

FECP DRAM

PPE DRAM

BRKCRS-3147

BQS Packet DRAM

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

18

Cisco “Quantum Flow Processor” Feature Summary • Packet Processing Engine (QFP-PPE) – 40 Packet Processors with 4 Contexts (threads) each; 160 simultaneous threads – Up to 1.2GHz Tensilica ISA processors + DRAM packet memory – Single TCAM4 I/F; can cascade 1-4 devices Multi-Core (40) Packet Processor – C-language for feature development; extensive development support tools – HW assist for flow-locks, look-ups, stats, WRED, policers, range lookup, crypto, CRC • Buffer/queue subsystem (QFP-BQS) – HW hierarchical 3-parameter (min, max & excess) scheduler – Fully configurable # of layers based on HQF – Priority propagation through the multiple layers BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Traffic Manager (BQS)

Cisco Public

19

ESP FECP

Generic ESP Block Diagram

QFP

Crypto Assist.

PP E

BQ S

intercon.

Reset / Pwr Ctrl TCAM

Resource DRAM

Temp Sensor EEPROM

Part Len / BW SRAM

QFP Complex Packet Processor Engine

DDRAM Boot Flash (OBFL,…)

Packet Buffer DRAM

FECP

JTAG Ctrl

BQS

PPE1

PPE2

PPE3

PPE4

PPE5

PPE6

PPE7

PPE8

…

PPEN

Dispatcher Packet Buffer

Crypto Reset / Pwr Ctrl

RPs

SPI Mux

SA table DRAM

RPs BRKCRS-3147

GE, 1Gbps I2C SPA Control SPA Bus

ESI, 11.2Gbps SPA-SPI, 11.2Gbps Hypertransport, 10Gbps Other

Interconnect

ESP

RPs

© 2014 Cisco and/or its affiliates. All rights reserved.

SIPs Cisco Public

20

SIP intercon.

SIP10 Block Diagram RPs

ESPs Reset / Pwr Ctrl

SPA Aggreg.

IOCP

SPA

SPA

RPs

Interconnect

EV-RP

Temp Sensor

EV-FC

In ref clocks

EEPROM DDRAM

Boot Flash (OBFL,…)

Egress Buffer Status

Ingress Scheduler

IOCP (SC854x SOC)

JTAG Ctrl

SPA Aggregation ASIC (Marmot) …

Network clock distribution

…

Ingress buffers

Egress buffers

(per port)

(per port)

Network clocks

Ingress Classifier

SPA Agg.

Reset / Pwr Ctrl

RPs BRKCRS-3147

RPs

ESI, 11.2 Gbps SPA-SPI, 11.2Gbps Hypertransport, 10Gbps Other

4 SPAs

4 SPAs

© 2014 Cisco and/or its affiliates. All rights reserved.

C2W

GE, 1Gbps I2C SPA Control SPA Bus

4 SPAs 4 SPAs 4 SPAs

Cisco Public

21

SIP intercon.

SIP10 Block Diagram (comments) RPs

ESPs Reset / Pwr Ctrl

SPA Aggreg.

IOCP

SPA

SPA

RPs

Interconnect

EV-RP

Temp Sensor

EV-FC

In ref clocks

EEPROM DDRAM

Boot Flash (OBFL,…)

IOCP (SC854x SOC)

JTAG Ctrl

IO Control Processor Manages SPA OIR & drivers Linux Kernel

Egress Buffer Status

Ingress Scheduler

SPA Aggregation ASIC (Marmot) …

SPA Aggregation Forwards and queues packets (FIFO) Network clock distribution

…

Ingress buffers

Egress buffers

(per port)

(per port)

Network clocks

Ingress Classifier

SPA Agg.

Reset / Pwr Ctrl

RPs BRKCRS-3147

RPs

ESI, 11.2 Gbps SPA-SPI, 11.2Gbps Hypertransport, 10Gbps Other

4 SPAs

4 SPAs

© 2014 Cisco and/or its affiliates. All rights reserved.

C2W

GE, 1Gbps I2C SPA Control SPA Bus

4 SPAs 4 SPAs 4 SPAs

Cisco Public

22

SPA Interface Processor – SIP-10G  Physical termination of SPA  Supports up to 4 SPA's – 4 half-height, 2 full-height, 2 HH+1FH – full OIR support  Does not participate in forwarding  Limited QoS – Ingress packet classification – high/low – Ingress over-subscription buffering (low priority) until ESP can service them. Up to 128MB of ingress oversubscription buffering  Capture stats on dropped packets  Network clock distribution to SPA's, reference selection from SPA's  IOCP manages Midplane links, SPA OIR, SPA drivers

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

23

ISR 4451-X Hardware Diagram

DDR3

DRAM

Control Plane (4 cores)

Ctrl

4xPCIe

SVC1

4xSGMI

Data Plane

FPGE

(10 core)

PPE1

PPE2

PPE3

PPE4

PPE5

DDR3 SVC2

SVC3

PPE6

PPE7

PPE8

PPE9

DRAM

PPE10

10 Gbps XAUI System FPGA Mgmt Ethernet Console / Aux

USB Flash

1xSGMI Multi Gigabit Fabric

Peripheral Interconnect

10 Gbps/slot

DSP

SM-X SM-X

2Gb/slot

NIM NIM

NIM BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

24

ISR 4451-X Hardware Diagram (comments) Inline Cryptography No Crypto Assist chip Crypto “locks” core True run-to-completion

10 Cores, 1 thread / core 5 fwd cores by default 4 remaining cores license activated DDR3

DRAM

Control Plane (4 cores)

1 Control Plane Core RP and FECP-like roles

Ctrl

4xPCIe

SVC1

4xSGMI

Data Plane (10 core)

PPE1

PPE2

PPE3

PPE4

PPE5

BQS onFPGE a core One Core dedicated to BQS Always active DDR3(5+1 or 9+1 cores) DRAM

SVC2

SVC3

PPE6

PPE7

PPE8

PPE9

PPE10

3 Services Core No hardware TCAM

10 Gbps XAUI System FPGA Mgmt Ethernet Console / Aux

USB Flash

1xSGMI Multi Gigabit Fabric

Peripheral Interconnect

10 Gbps/slot

DSP

SM-X SM-X

2Gb/slot

NIM NIM

NIM BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

25

ISR 4451 System Layout (2RU Platform) Dataplane DIMM External Serviceable CF

Dataplane CPU

DSP Slot

Control & Services Dual DIMM Control & Services CPU

1 SW-NIM or Dual HDD Configurable Slot (@ factory only)

Airflow – Front to back 2RU, ~18” depth MGMT

Dual USB Type-A

AUX

Console: Mini-USB / RJ45 BRKCRS-3147

30W PoE converter for onboard GE’s

4-GE (SFP)

2-GE (RJ-45) 2-GE (RJ-45)

© 2014 Cisco and/or its affiliates. All rights reserved.

Service Modules and Network Interface Modules

Cisco Public

26

Acronyms  MCP – Midrange Converged Platform (codename for ASR1000 during development)  RP – Route Processor  FP – Forwarding Processor = ESP (Embedded Service Processor)

 CPP – Cisco Packet Processor Compex= QFP (Quantum Flow Processor)  PPE – Packet Processing Engine  IOCP – I/O Control Processor  FECP – Forwarding Engine Control Processor

 SPA – Shared Port Adapter  SIP – SPA Interface Processor  IOSd – IOS image that runs as a process on the RP  FMAN – Forwarding manager (FMAN-RP, FMAN-FP)

 Scbac – FW Session Control Block  EOBC = Ethernet Out of Band Channels – Packet Interface for Card to Card Control Traffic  IOS-XE (BinOS) = Linux Based Software Infrastructure That Executes on MCP

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

27

Software Architecture

ASR1K Software Architecture RP

CPU RP

Chassis Manager CPU

IOS

ESI (10-40 Gbps)

Forwarding Manager interconn.Linux GE switch Kernel

ESP ESP FECP

I2C

EOBC (1 Gbps)

Drivers Drivers Drivers Crypto Assist. QFP

µ µµ µ µ µ

ESI (10-40 Gbps)

SIP

Linux Kernel QFP

Crypto BQS Assist. interconn.

SIP IOCP interconn. SPA Driver Chassis SPA Driver Manager SPA Driver SPA IOCP Aggreg. Linux Kernel SPA SPA

BRKCRS-3147

Chassis Manager FECP Forwarding Manager

SPA

SPA

© 2014 Cisco and/or its affiliates. All rights reserved.

SPA

Cisco Public

29

Chassis Manager (CM) RP

 CM on RP communicates with CM processes on ESP and SIP

CPU Chassis Manager

– Distributed function

IOS

ESI (10-40 Gbps)

Forwarding Manager

– CM on SIP queries SPA type and load SPA drivers

Linux Kernel

ESP

 Manages hardware components FECP

Chassis Manager

EOBC (1 Gbps)

I2C

Drivers Drivers Drivers

Forwarding Manager Linux Kernel

QFP µ µµ µ µ µ

BQS

Crypto Assist.

– – – –

Manages EOBC on RP Manages ESI links on RP/ESP/SIP Manages timing circuitry on RP Reset and power-down on RP/ESP/SIP

 Communicates IOS hardware components – Static & OIR

ESI (10-40 Gbps)

 Monitors environmental variables and alarms SIP

IOCP

SPA Driver SPA Driver SPA Driver

Chassis Manager

Linux Kernel SPA BRKCRS-3147

 Initializes hardware and boots other processes

SPA

 Selects active/standby RP or ESP – Coordinates switchover in case of failure or operator command

SPA

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

30

Forwarding Manager (FMAN) RP

CPU FMAN-RP Chassis Manager IOS

ESI (10-40 Gbps)

Forwarding Manager Linux Kernel

ESP

FMAN-FP ESP aka Forwarding Plane Chassis Manager

FECP

EOBC (1 Gbps)

I2C

Drivers Drivers Drivers

Forwarding Manager Linux Kernel

QFP µ µµ µ µ µ

ESI (10-40 Gbps)

SIP

IOCP

BQS

SPA Driver SPA Driver SPA Driver

Crypto Assist.

Chassis Manager

Linux Kernel SPA BRKCRS-3147

SPA

SPA

 FMAN on RP communicates with FMAN process on ESP – Distributed function

 Propagates control plane ops. to ESP – CEF tables, ACL’s, NAT, SA’s,…

 FMAN-FP communicates information back to FMAN-RP – e.g. statistics – FMAN-RP pushes info back to IOS

 FMAN on active RP maintains state for both active & standby ESP’s – Facilitates NSF after re-start with bulk download of state information

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

31

PPE Microcode RP

 Written in C

CPU Chassis Manager IOS

ESI (10-40 Gbps)

Forwarding Manager

 Runs on each thread of the PPC

Linux Kernel

 Processes packets ESP

FECP

Chassis Manager

EOBC (1 Gbps)

I2C

Drivers Drivers Drivers

Forwarding Manager Linux Kernel

QFP QFP µ µµ µ … µ µ Packet Processor Engine

BQS

PPE PPE PPE PPE PPE 1

2

3

4

PPE PPE PPE 6

7

5

PPE

8

BQS

N

Crypto Assist.

ESI (10-40 Gbps)

Dispatcher Packet Buffer

SIP

IOCP

SPA Driver SPA Driver SPA Driver

Chassis Manager

Linux Kernel SPA BRKCRS-3147

– proper features, no hack

SPA

– run to completion – assisted by various memories – TCAM, DRAM,… various speeds

 Features applied via FIA – Feature Invocation Array

 FIA per interface – input FIA, output FIA – drop FIA (Null interface)

SPA

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

32

show platform hardware qfp active interface if-name GigabitEthernet 0/0/0

Feature Invocation Array (FIA)  A per protocol array of functions/features to be executed in sequence  The FIA is executed in PPE for every packet  Input interface  Input FIA ; Output interface  Output FIA Example Input FIA

Example Output FIA

Dst Lookup Consume

Output Inspect

For Us Martian

WCCP

RPF

NAT

Security ACL (in)

Refragment

RPF Checks

MQC Classify

IPsec Classify (crypto map)

Lawful Intercept

NAT PBR WCCP

Example Punt FIA

Output Inspect VFR Refrag

RP seens as an external device from the ESP… connected to a special interface. This interface has its own FIA.

Drop Policy Internal Transmit Pkt

Security ACL (out) Tunnel Encapsulation Crypto (tunnel protection)

Input Lookup Process

These are simple examples. Real FIA’s can be somewhat arcane...

IP Options Process BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

33

Day in Life of Normal Packet

SIP intercon.

SIP10 Block Diagram

SPA Aggreg.

IOCP

SPA

SPA

ESPs Reset / Pwr Ctrl

Interconnect

EV-RP

Temp Sensor

EV-FC

In ref clocks

EEPROM DDRAM

Boot Flash (OBFL,…)

Egress Buffer Status

Ingress Scheduler

IOCP (SC854x SOC)

…

JTAG Ctrl

SPA Aggregation ASIC (Marmot) …

Network clock distribution

…

Ingress buffers

Egress buffers

(per port)

(per port)

Network clocks

Ingress Classifier Reset / Pwr Ctrl

SPA Agg.

C2W

SPA BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

35

ESP FECP

Ingress packet through SIP

Crypto Assist.

QFP PP E

BQ S

intercon.

Reset / Pwr Ctrl TCAM

Resource DRAM

Temp Sensor EEPROM

Part Len / BW SRAM

QFP Complex Packet Processor Engine

DDRAM Boot Flash (OBFL,…)

Packet Buffer DRAM

FECP

JTAG Ctrl

BQS

PPE1

PPE2

PPE3

PPE4

PPE5

PPE6

PPE7

PPE8

…

PPEN

Dispatcher Packet Buffer

Crypto Reset / Pwr Ctrl

RPs

SPI Mux

SA table DRAM

RPs BRKCRS-3147

Interconnect

ESP

RPs

© 2014 Cisco and/or its affiliates. All rights reserved.

SIPs Cisco Public

36

ESP FECP

Packet dispatched to PPE core

Crypto Assist.

QFP PP E

BQ S

intercon.

Reset / Pwr Ctrl TCAM

Resource DRAM

Temp Sensor EEPROM

Packet Processor Engine PPE2

PPE3

PPE4

PPE5

PPE6

PPE7

PPE8

…

PPEN

Thread 4

JTAG Ctrl

BQS

PPE1

Thread 3

Thread 1

FECP

Thread 2

Boot Flash (OBFL,…)

Part Len / BW SRAM

QFP Complex

PPE2

DDRAM

Packet Buffer DRAM

Dispatcher Packet Buffer

Crypto Reset / Pwr Ctrl

RPs

SPI Mux

SA table DRAM

RPs BRKCRS-3147

Interconnect

ESP

RPs

© 2014 Cisco and/or its affiliates. All rights reserved.

SIPs Cisco Public

37

ESP FECP

Packet dispatched to PPE thread

Crypto Assist.

QFP PP E

BQ S

intercon.

Reset / Pwr Ctrl TCAM

Resource DRAM

Temp Sensor EEPROM

Packet Processor Engine PPE2

PPE3

PPE4

PPE5

PPE6

PPE7

PPE8

…

PPEN

Thread 4

JTAG Ctrl

BQS

PPE1

Thread 3

Thread 1

FECP

Thread 2

Boot Flash (OBFL,…)

Part Len / BW SRAM

QFP Complex

PPE2

DDRAM

Packet Buffer DRAM

Dispatcher Packet Buffer

Crypto Reset / Pwr Ctrl

RPs

SPI Mux

SA table DRAM

RPs BRKCRS-3147

Interconnect

ESP

RPs

© 2014 Cisco and/or its affiliates. All rights reserved.

SIPs Cisco Public

38

ESP FECP

Packet processed by PPE thread

Crypto Assist.

QFP PP E

BQ S

intercon.

X-Connect Reset / Pwr Ctrl L2 Switch

IPv4 TCAM

IPv6 Resource DRAM

Temp Sensor EEPROM

Packet Processor Engine

NBAR Classify

IP Unicast PPE6

PPE7

PBR

Packet Us

Dialer IDLE Rst

Reset / Pwr Ctrl

RPs

BRKCRS-3147

…

Dispatcher For Packet Buffer

NAT

BQS

PPE5 NBAR Classify

… PPEN

MQC Policing MAC Accounting WRED Queuing

SPI Mux

SA table DRAM

RPs

PPE8

PPE4

IP Multicast

NAT

URD Crypto

PPE3

Thread 4

PPE2

Thread 3

…

JTAG Ctrl

PPE1

Thread 2

Thread 1

FECP MQC Classify

Part Len / BW SRAM

QFP Complex Netflow

PPE2

BGP Accounting

Boot Flash (OBFL,…)

MPLS

Output FIA

Input FIA

Netflow DDRAM

Packet Buffer DRAM

PPE2 Thread 3

Interconnect

ESP

RPs

© 2014 Cisco and/or its affiliates. All rights reserved.

SIPs Cisco Public

39

ESP FECP

Generic ESP Block Diagram

Crypto Assist.

QFP PP E

BQ S

intercon.

X-Connect Reset / Pwr Ctrl L2 Switch

IPv4 TCAM

IPv6 Resource DRAM

Temp Sensor EEPROM

Packet Processor Engine

NBAR Classify

IP Unicast PPE6

PPE7

PBR

Packet Us

Dialer IDLE Rst

Reset / Pwr Ctrl

RPs

BRKCRS-3147

…

Dispatcher For Packet Buffer

NAT

BQS

PPE5 NBAR Classify

… PPEN

MQC Policing MAC Accounting WRED Queuing

SPI Mux

SA table DRAM

RPs

PPE8

PPE4

IP Multicast

NAT

URD Crypto

PPE3

Thread 4

PPE2

Thread 3

…

JTAG Ctrl

PPE1

Thread 2

Thread 1

FECP MQC Classify

Part Len / BW SRAM

QFP Complex Netflow

PPE2

BGP Accounting

Boot Flash (OBFL,…)

MPLS

Output FIA

Input FIA

Netflow DDRAM

Packet Buffer DRAM

PPE2 Thread 3

Interconnect

ESP

RPs

© 2014 Cisco and/or its affiliates. All rights reserved.

SIPs Cisco Public

40

ESP FECP

Packet proceeding to BQS then SIP

Crypto Assist.

QFP PP E

BQ S

intercon.

Reset / Pwr Ctrl TCAM

Resource DRAM

Temp Sensor EEPROM

Part Len / BW SRAM

QFP Complex Packet Processor Engine

DDRAM Boot Flash (OBFL,…)

Packet Buffer DRAM

FECP

JTAG Ctrl

BQS

PPE1

PPE2

PPE3

PPE4

PPE5

PPE6

PPE7

PPE8

…

PPEN

Dispatcher Packet Buffer

Crypto Reset / Pwr Ctrl

RPs

SPI Mux

SA table DRAM

RPs BRKCRS-3147

Interconnect

ESP

RPs

© 2014 Cisco and/or its affiliates. All rights reserved.

SIPs Cisco Public

41

SIP intercon.

Egress packet through SIP

SPA Aggreg.

IOCP

SPA

SPA

ESPs Reset / Pwr Ctrl

Interconnect

EV-RP

Temp Sensor

EV-FC

In ref clocks

EEPROM DDRAM

Boot Flash (OBFL,…)

Egress Buffer Status

Ingress Scheduler

IOCP (SC854x SOC)

JTAG Ctrl

SPA Aggregation ASIC (Marmot) …

Network clock distribution

…

Ingress buffers

Egress buffers

(per port)

(per port)

Network clocks

Ingress Classifier Reset / Pwr Ctrl

SPA Agg.

C2W

SPA BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

42

An Advanced Example: IPsec control plane programming

IPsec SA – from IOS to FMAN-FP show crypto ipsec sa interface virtual-access 1002

RP

interface: Virtual-Access1002 Crypto map tag: Virtual-Access1002-head-0, local addr 172.18.0.1

protected vrf: (none) local ident (addr/mask/prot/port): (172.18.0.1/255.255.255.255/47/0) remote ident (addr/mask/prot/port): (17.0.0.26/255.255.255.255/47/0) current_peer 17.0.0.26 port 500 … inbound esp sas: spi: 0x956A1B11(2506758929) … conn id: 30214, flow_id: HW:28214, sibling_flags FFFFFFFF80000008, crypto map: Virtual-Access1002-head-0

CPU Chassis Manager

IOS Forwarding Manager Linux Kernel show platform software ipsec fp active flow identifier SPD id: 1008 … QFP SA handle: 1892 … Crypto SA ctx id: 0x000000002e02b9b6

EOBC (1 Gbps)

outbound esp sas: spi: 0x51E3FC8E(1373895822) … conn id: 30213, flow_id: HW:28213, sibling_flags FFFFFFFF80000008, crypto map: Virtual-Access1002-head-0 …

ESP FECP

Chassis Manager

Drivers Drivers Drivers

Forwarding Manager Linux Kernel

QFP µ µµ µ µ µ TCAM BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

BQS

Crypto Assist.

DRAM

DRAM

Cisco Public

44

IPsec SA – from FMAN-FP to QFP TCAM show crypto ipsec sa interface virtual-access 1002 interface: Virtual-Access1002 Crypto map tag: Virtual-Access1002-head-0, local addr 172.18.0.1

protected vrf: (none) local ident (addr/mask/prot/port): (172.18.0.1/255.255.255.255/47/0) remote ident (addr/mask/prot/port): (17.0.0.26/255.255.255.255/47/0) current_peer 17.0.0.26 port 500 … inbound esp sas: spi: 0x956A1B11(2506758929) … conn id: 30214, flow_id: HW:28214, sibling_flags FFFFFFFF80000008, crypto map: Virtual-Access1002-head-0

RP

CPU Chassis Manager

IOS Forwarding Manager Linux Kernel show platform software ipsec fp active flow identifier SPD id: 1008 … QFP SA handle: 1892 … Crypto SA ctx id: 0x000000002e02b9b6

show platform hardware qfp active classification feature-manager class-group tcam ipsec <SPD-id> global detail class-group [ipsec-cg:1008] (classes: 1, total number of vmrs: 4) key name: 160_03 value size: 160 result size: 16 region id: 1 vmr id: 698 number of vmrs: 4 tcam id: TCAM0 Value: : ac120001 2f000000 00000000 1100001a 12d70000 Mask: : ffffffff ff000000 00000000 ffffffff ffff0000 Result: : 40000000 905a0400 00000000 00000000 Value: : ac120001 2f000000 00000000 1100001a 12d70000 Mask: : ffffffff ff000000 00000000 ffffffff ffff0000 Result: : 20000000 8d458860 00000000 00000000 … BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved.

EOBC (1 Gbps)

outbound esp sas: spi: 0x51E3FC8E(1373895822) … conn id: 30213, flow_id: HW:28213, sibling_flags FFFFFFFF80000008, crypto map: Virtual-Access1002-head-0 …

ESP FECP

Chassis Manager

Drivers Drivers Drivers

Forwarding Manager Linux Kernel

QFP µ µµ µ µ µ TCAM

BQS

Crypto Assist.

DRAM

DRAM

Cisco Public

45

IPsec SA – from FMAN-FP to Crypto Engine show crypto ipsec sa interface virtual-access 1002

RP

interface: Virtual-Access1002 Crypto map tag: Virtual-Access1002-head-0, local addr 172.18.0.1

Chassis Manager

IOS

protected vrf: (none) local ident (addr/mask/prot/port): (172.18.0.1/255.255.255.255/47/0) remote ident (addr/mask/prot/port): (17.0.0.26/255.255.255.255/47/0) current_peer 17.0.0.26 port 500 … inbound esp sas: spi: 0x956A1B11(2506758929) … conn id: 30214, flow_id: HW:28214, sibling_flags FFFFFFFF80000008, crypto map: Virtual-Access1002-head-0

Forwarding Manager Linux Kernel show platform software ipsec fp active flow identifier SPD id: 1008 … QFP SA handle: 1892 … Crypto SA ctx id: 0x000000002e02b9b6

class-group [ipsec-cg:1008] (classes: 1, total number of vmrs: 4) key name: 160_03 value size: 160 result size: 16 region id: 1 vmr id: 698 number of vmrs: 4 tcam id: TCAM0 Value: : ac120001 2f000000 00000000 1100001a 12d70000 Mask: : ffffffff ff000000 00000000 ffffffff ffff0000 Result: : 40000000 905a0400 00000000 00000000 Value: : ac120001 2f000000 00000000 1100001a 12d70000 Mask: : ffffffff ff000000 00000000 ffffffff ffff0000 Result: : 20000000 8d458860 00000000 00000000 … BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved.

EOBC (1 Gbps)

outbound esp sas: spi: 0x51E3FC8E(1373895822) … conn id: 30213, flow_id: HW:28213, sibling_flags FFFFFFFF80000008, crypto map: Virtual-Access1002-head-0 …

show platform hardware qfp active classification feature-manager class-group tcam ipsec <SPD-id> global detail

CPU

show platform software ipsec fp active encryption-processor context 2e02b9b6 =======

ESP

Context id: 0x02b249 … SA word 0: 0x5ae0460fc201aa5 action bits: 0x001f84 FECP direction: outbound Chassis Manager mode: transport protocol: esp Drivers Forwarding Manager authentication: SHA-1 Drivers confidentiality: AES-128 Drivers … Linux Kernel mfs: 1454 … QFP sequence number: 306 µ µµ Crypto … BQS µ Assist. byte count: 25704 µ µ packet count: 306

TCAM

DRAM Cisco Public

DRAM 46

IPsec SA – from FMAN-FP to QFP DRAM show crypto ipsec sa interface virtual-access 1002

RP

interface: Virtual-Access1002 Crypto map tag: Virtual-Access1002-head-0, local addr 172.18.0.1

CPU Chassis Manager

IOS

protected vrf: (none) local ident (addr/mask/prot/port): (172.18.0.1/255.255.255.255/47/0) remote ident (addr/mask/prot/port): (17.0.0.26/255.255.255.255/47/0) current_peer 17.0.0.26 port 500 … inbound esp sas: spi: 0x956A1B11(2506758929) … conn id: 30214, flow_id: HW:28214, sibling_flags FFFFFFFF80000008, show platform hardware qfp active feature ipsec sa crypto map: Virtual-Access1002-head-0

Forwarding Manager Linux Kernel show platform software ipsec fp active flow identifier SPD id: 1008 FMAN-FP knows … everything QFP SA handle: 1892 … Crypto SA ctx id: 0x000000002e02b9b6

show platform hardware qfp active classification feature-manager class-group tcam ipsec <SPD-id> global detail class-group [ipsec-cg:1008] (classes: 1, total number of vmrs: 4) key name: 160_03 value size: 160 result size: 16 region id: 1 vmr id: 698 number of vmrs: 4 tcam id: TCAM0 Value: : ac120001 2f000000 00000000 1100001a 12d70000 Mask: : ffffffff ff000000 00000000 ffffffff ffff0000 Result: : 40000000 905a0400 00000000 00000000 Value: : ac120001 2f000000 00000000 1100001a 12d70000 Mask: : ffffffff ff000000 00000000 ffffffff ffff0000 Result: : 20000000 8d458860 00000000 00000000 … BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved.

ESP

EOBC (1 Gbps)

outbound esp sas: QFP ipsec sa Information Also indexed by spi: 0x51E3FC8E(1373895822) class-group … QFP sa id: 3623 conn id: 30213, flow_id: HW:28213, sibling_flags FFFFFFFF80000008, pal sa id: 32085 crypto map: Virtual-Access1002-head-0 QFP spd id: 3398 … QFP sp id: 1066 QFP spi: 0x51E3FC8E(1373895822) crypto ctx: 0x000000002e02b9b6 …

show platform software ipsec fp active encryption-processor context 2e02b9b6 =======

Context id: 0x02b249 … SA word 0: 0x5ae0460fc201aa5 action bits: 0x001f84 FECP direction: outbound Chassis Manager mode: transport protocol: esp Drivers Forwarding Manager authentication: SHA-1 Drivers confidentiality: AES-128 Drivers … Linux Kernel mfs: 1454 … QFP sequence number: 306 µ µµ Crypto … BQS µ Assist. byte count: 25704 µ µ packet count: 306

TCAM

DRAM Cisco Public

DRAM 47

ESP FECP

Egress IPsec Packet Flow (I)

Crypto Assist.

Lookup SA Handler by class-group ID Obtain Crypto SA ctx ID

Look up IPsec proxy-identities Obtain class-group ID

Reset / Pwr Ctrl TCAM

Resource DRAM

Temp Sensor EEPROM

Packet Processor Engine

FECP

JTAG Ctrl

Uses Crypto Context identified by Context ID

Crypto Reset / Pwr Ctrl

RPs

BRKCRS-3147

BQ S

intercon.

Part Len / BW SRAM

BQS

PPE1

PPE2

PPE3

PPE4

PPE5

PPE6

PPE7

PPE8

…

PPEN

Dispatcher Packet Buffer

SPI Mux

SA table DRAM

RPs

PP E

QFP Complex

DDRAM Boot Flash (OBFL,…)

Packet Buffer DRAM

QFP

Interconnect

ESP

RPs

© 2014 Cisco and/or its affiliates. All rights reserved.

SIPs Cisco Public

48

ESP FECP

Egress IPsec Packet Flow (II)

Crypto Assist.

QFP PP E

BQ S

intercon.

Reset / Pwr Ctrl TCAM

Resource DRAM

Temp Sensor EEPROM DDRAM Boot Flash (OBFL,…)

FECP

Packet Buffer DRAM

Part Len / BW SRAM

QFP Complex PPE may be different butPacket packetProcessor Engine processing continues where it stopped (right after crypto)

JTAG Ctrl

BQS

PPE1

PPE2

PPE3

PPE4

PPE5

PPE6

PPE7

PPE8

…

PPEN

Dispatcher Packet Buffer

Crypto Reset / Pwr Ctrl

RPs

SPI Mux

SA table DRAM

RPs BRKCRS-3147

Interconnect

ESP

RPs

© 2014 Cisco and/or its affiliates. All rights reserved.

SIPs Cisco Public

49

ESP FECP

General Feature Dependencies Class/Policy Maps: QoS, DPI, FW ACL/ACE storage Reset / Pwr Ctrl IPSec Traffic Selectors, classes, rules NAT Tables Temp Sensor

QoS Mark/Police NAT sessions IPSec SA Netflow Cache Per session data (FW, NAT, Netflow)

TCAM

FECP

Reset / Pwr Ctrl

RPs

BRKCRS-3147

PPE2

PPE3

PPE4

PPE5

PPE6

PPE7

PPE8

…

PPEN

Dispatcher Packet Buffer

BQ S

intercon.

Part Len / BW SRAM

BQS

BQS offloads queuing and scheduling from cores. 16000 Queues on ASR1001 & ESP 5 127000 Queues on ESP10+ 1Gbps 470000 Queues on ESP 100+GE, 2 System Bandwidth 5, 10, 20, 40, 100, 200 Gbps

SPI Mux

SA table DRAM

RPs

PPE1

Crypto Assist chip offloads crypto from the PPE cores

Crypto

QoS Queuing NAT VFR re-assembly IPSec headers

PP E

QFP Complex

DDRAM

QFP client / driver JTAG Ctrl OBFL QoS Class maps FM FP Statistics ACL ACEs copy NAT config objects IPSec/IKE SA NF config data ZB-FW config objects

Packet Buffer DRAM

Cores execute packet processing All features handled from here CPU horsepower is here… Packet Processor Engine

EEPROM

Boot Flash (OBFL,…) Memory for FECP

Resource DRAM

QFP

Crypto Assist.

ESI, 11.2Gbps SPA-SPI, 11.2Gbps Hypertransport, 10Gbps Other

Interconnect

ESP

RPs

© 2014 Cisco and/or its affiliates. All rights reserved.

IC SPA Control SPA Bus

SIPs Cisco Public

50

Debugging strategies

Everyday situations

Traffic did not reach its target ! What happened to that packet ? Why did that happen ?

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

52

Using statistics for troubleshooting packet drops ESP

SPA 

show interfaces



show interfaces accounting



show interfaces stats

SIP  

show platform hardware port <slot/card/port> plim statistics show platform hardware subslot {slot/card} plim statistics



show platform hardware slot {slot} plim statistics



show platform hardware slot {0|1|2} plim status internal



show platform hardware slot {0|1|2} serdes statistics



show platform hardware slot {f0|f1} serdes statistics



show platform hardware slot {f0|f1} serdes statistics internal



show platform hardware qfp active bqs 0 ipm mapping



show platform hardware qfp active bqs 0 ipm statistics channel all



show platform hardware qfp active bqs 0 opm mapping



show platform hardware qfp active bqs 0 opm statistics channel all



show platform hardware qfp active statistics drop [detail]



show platform hardware qfp active interface if-name statistics



show platform hardware qfp active infrastructure punt statistics type per-cause | exclude _0_



show platform hardware qfp active infrastructure punt statistics type punt-drop | exclude _0_



show platform hardware qfp active infrastructure punt statistics type inject-drop | exclude _0_



show platform hardware qfp active infrastructure punt statistics type global-drop | exclude _0_



show platform hardware qfp active infrastructure bqs queue output default all



show platform hardware qfp active infrastructure bqs queue output recycle all

RP  

Not easy… not very practical either. Let’s dig deeper before making it simpler

show platform hardware slot {r0|r1} serdes statistics show platform software infrastructure lsmpi

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

53

Debugging Strategies to Date Well Known

IOS Control Plane • show interface • show ip route, show bgp …

• ESP “stuff” • e.g. show platform …

Top Down

Very Difficult

Bottom Up

Platform Control Plane

Let’s change that!!

Data Plane • ESP “stuff” • e.g. show platform … BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

54

The Road to Simplification: Part I, Data Plane Debugging 55

IOS 3.7

The Embedded Packet Capture One way of capturing packets… Device# Device# Device# Device# Device# Device# Device# Device#

monitor monitor monitor monitor monitor monitor monitor monitor

capture capture capture capture capture capture capture capture

mycap mycap mycap mycap mycap mycap mycap mycap

start access-list v4acl limit duration 1000 interface GigabitEthernet 0/0/1 both buffer circular size 10 start export tftp://10.1.88.9/mycap.pcap stop

Shows whether packets have been received or sent Shows what packets look like Requires hex dump analysis or export to decoder (sniffer) Does not tell us what happened to the packet

Device# show monitor capture mycap buffer dump 0 0000: 0010: 0020: 0030:

01005E00 00300000 000207C1 1D006369

080045C0 0002E000 10030AFA 0001

..^...........E. .0.............. .........*...... ..example.......

1 0000: 0010: 0020:

01005E00 0002001B 2BF69280 080046C0 00200000 00000102 44170000 0000E000 00019404 00001700 E8FF0000 0000

..^.....+.....F. . ......D....... ..............

2 0000: 0010: 0020: 0030:

01005E00 00300000 000207C1 1D006369

..^.....+.....E. .0.............. ...............n ..example.......

00020000 00000111 07C1001C 73636F00

0002001B 00000111 07C1001C 73636F00

BRKCRS-3147

0C07AC1D CFDC091D 802A0000 0000091D

2BF68680 CFDB091D 88B50000 0000091D

080045C0 0003E000 08030A6E 0001

© 2014 Cisco and/or its affiliates. All rights reserved.

Excellent tool but insufficient in many cases

http://www.cisco.com/en/US/docs/iosxml/ios/epc/configuration/xe-3s/asr1000/nm-packet-capturexe.html

Cisco Public

56

IOS 3.10

The Packet Tracer and FIA Debugger X-Connect Reset / Pwr Ctrl L2 Switch

IPv4 TCAM

Condition determines packets Temp Sensor to be traced

Packet Buffer DRAM

Resource DRAM

Packet Processor Engine

DDRAM

NAT

BRKCRS-3147

BQS

PPE8

…

Dispatcher Packet Buffer

SPI Mux

MQC Classify NAT

PPE5

NAT

PBR

PPEN

Output ACL

Encaps Statistics and final action will be

SA table DRAM

RPs

PPE7

PPE4

IP Unicast

PBR

Reset / Pwr Ctrl

PPE6

PPE3

Thread 4

JTAG Ctrl

PPE2

Thread 3

MQC Classify

PPE1

Thread 2

Thread 1

FECP

Packet # 16 Input ACL

Output ACL

Input ACL

Crypto

Part Len / BW SRAM

QFP Complex

PPE2

Pak Trace ?

Boot Flash (OBFL,…)

MPLS

Output FIA

Input FIA

EEPROM

RPs

IPv6

Crypto collected (matched packets dropped, punted to RP, forwarded to output PPE2 interface …) Thread 3

NAT Encaps Crypto

Interconnect

ESP

RPs

© 2014 Cisco and/or its affiliates. All rights reserved.

SIPs

Optionally, FIA actions can logged per packet System can capture several packets flows Packet flows can be reviewed in show commands Cisco Public

57

Packet Tracer Demonstration

Packet-Trace: Configuration Commands  The Pactrac (Packet Tracer) shows us what happens to a series of packets – True inspection of IOS XE packet forwarding flow

 debug platform packet-trace enable – Enables accounting – Required for all levels of inspection

 debug platform packet-trace packet \ [fia-trace | summary-only] [circular] [data-size ] – – – – – –

Required for any per-packet data capture (e.g. necessary for packet copy to function) Specifies maximum number of packets maintained at one time () Always enables capture of summary data or only summary data (summary-only) Captures feature path data by default Optionally performs FIA trace (fia-trace) in addition to path data capture Allows specifying the size of the path data buffers (defaults to 2048) BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

59

Packet-Trace: Configuration Commands  debug platform packet-trace copy packet {in | out | both} [L2 | L3 | L4] [size ] – Enables copy of the ingress and/or egress packets – Optionally allows specifying where to start copy of the packet (L2 is default) – Optionally allows specifying the maximum number of octets to copy (64 is default)

Available XE3.11 and forward  debug platform packet-trace drop [code ] – Enables retention only for dropped packets – Optionally allows specifying retaining packets for a specific drop code – Can be used without global/interface conditions to capture drop events*

*Drop event capture means the only the drop itself is traced not the life of the packet, but, it still allows capture of summary data, tuple data and the packet to help refine conditions or provide clues to the next debug step. BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

60

Packet-Trace: Configuration Commands  clear platform packet-trace statistics – Clears any collected statistics and data buffers – Tracing must be stopped first (debug platform condition stop)

 clear platform packet-trace configuration – Removes all debug platform packet-trace commands

 clear platform condition all – Removes all debug platform condition and debug platform packet-trace commands

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

61

Packet-Trace: Configuration Commands Packet-trace relies on the conditional infra to determine which packets are interesting. The condition infra provides the ability to filter by protocol, IP address and mask, ACL, interface and direction. A complete discussion of conditions is not made here but some illustrative examples are:  debug platform condition ingress – Checks all incoming packets on all interfaces for all protocols

 debug platform condition interface g0/0/0 ipv4 ingress – Checks all IPv4 packets arriving on interface g0/0/0

 debug platform condition interface g0/0/0 ipv4 access-list FOO ingress – Checks incoming IPv4 packets on interface g0/0/0 that match access-list FOO

Conditions are activated or de-activated using debug platform condition start or debug platform condition stop respectively. BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

62

Packet-Trace: Configuration Commands NOTA BENE!!!!! Conditions define what the filters are and when the filters are applied to a packet. For example, debug platform condition interface g0/0/0 egress means that a packet will be identified as a match when it reaches the output FIA on interface g0/0/0 so any packet-processing that took place from ingress up to that point is missed. Best Practice It is highly recommended to use ingress conditions for pactrac to get the most complete and meaningful data. Egress conditions can be used but just be aware of the limitation above.

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

63

Packet-Trace: Configuration Example The following shows how one would trace the most recent 128 packets entering GigabitEthernet0/0/0 including FIA trace and a copy of up to the first 2048 octets of the input packet. debug platform condition interface g0/0/0 ingress debug platform packet-trace enable debug platform packet-trace packet 128 fia-trace circular debug platform packet-trace copy packet input size 2048 debug platform condition start <…wait until you’ve captured the packets you think you want…> debug platform condition stop

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

64

Packet-Trace: Configuration Highlights  Pactrac buffers consume QFP DRAM – Be mindful of how much memory a config needs and how much memory is available

 Configure as much detail as you want…more detail…more performance impact for matched packets  Each pactrac “config” change will temporarily disables pactrac and clears counts/buffers – “Cheap” way of ‘debug plat cond stop’, ‘clear plat pack stats’ and ‘debug plat cond start’

 Some configs require a ‘stop’ in order to display summary or per packet data – Currently circular and drop tracing

 Conditions define where and when filters are applied to a packet

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

65

Packet-Trace: Show Commands Show commands are used to display pactrac configuration and each level of data:  show platform packet-trace configuration – Displays packet-trace configuration including any defaults

 show platform packet-trace statistics – Displays accounting data for all pactrac packets

 show platform packet-trace summary – Displays summary data for the number of packets specified by debug platform packet-trace packet

 show platform packet-trace packet { all | } [decode]* – Displays all path data for all packets or the packet specified – Decode attempts to display packets captured by debug platform packet-trace copy in user friendly way – * decode was introduced in XE3.11

 NOTE: only a few protocol headers are supported initially (ARPA, IP, TCP, UDP, ICMP) BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

66

Example of Packet-Trace Configuration

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

67

Example of Packet-Trace Accounting

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

68

Example of Packet-Trace Summary

in0/0/rp:0 is how the ESP sees the RP

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

69

Example of Packet-Trace Packet Details

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

70

Example of Clearing Packet-Trace Stats

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

71

Understanding and Extracting ESP Logs 72

ESP Tracing aka Logging RP

TEMP RAM FS RP logs are first written here (efficiency)

CPU

Chassis Manager IOS

ESI (10-40 Gbps)

NFS Shared Disk Hard disk is really here

Forwarding Manager Linux Kernel

ESP

FECP

Chassis Manager

I2C

EOBC (1 Gbps)

Drivers Drivers Drivers

Forwarding Manager Linux Kernel

QFP µ µµ µ µ µ

ESI (10-40 Gbps)

SIP

IOCP

TEMP RAM FS ESP logs are first written here (efficiency)

BQS

SPA Driver SPA Driver SPA Driver

Mounted NFS ESP logs are committed here at regular intervals

Crypto Assist.

Chassis Manager

Linux Kernel SPA

BRKCRS-3147

SPA

© 2014 Cisco and/or its affiliates. All rights reserved.

SPA

Cisco Public

73

Important logs RP

CPU

fman_rp_R[0|1]-0.log

Chassis Manager IOS

ESI (10-40 Gbps)

Under /harddisk/tracelogs fman_rp_R[0|1]-0.log. fman-fp_R0.log. cpp_cp_F[0|1]-0.log.

Forwarding Manager Linux Kernel

ESP

FECP

Chassis Manager

I2C

EOBC (1 Gbps)

Drivers Drivers Drivers

Forwarding Manager Linux Kernel

QFP µ µµ µ µ µ

ESI (10-40 Gbps)

SIP

IOCP

fman_rp_R[0|1]-0.log fman_fp_F[0|1]-0.log cpp_cp_F[0|1]-0.log

BQS

SPA Driver SPA Driver SPA Driver

Under /harddisk/tracelogs/ fman-fp_R0.log. cpp_cp_F[0|1]-0.log.

Crypto Assist.

Chassis Manager

Linux Kernel SPA

BRKCRS-3147

SPA

© 2014 Cisco and/or its affiliates. All rights reserved.

SPA

Cisco Public

74

What log files are important?  Important log files to get for security issues: – fman_rp_R[0|1].log (under /tmp/rp/trace directory on RP) – fman-fp_F[0|1]-0.log (under /tmp/fp/trace directory on ESP – cpp_cp_F[0|1]-0.log (under /tmp/fp/trace directory on ESP)

 All these logs get rotated and are copied to /harddisk/tracelogs directory on active RP.  Look for the relevant log files depending on the time of the failure  By default, all ERR messages are logged, these should be the first things to look for

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

75

Example log files

The timestamp…

My-ASR1000-2#dir harddisk:/tracelogs/cpp_cp_F0* Directory of harddisk:/tracelogs/cpp_cp_F0* Directory of harddisk:/tracelogs/ 3768365 -rwx 1048934 Jan 6 2014 18:20:16 +00:00 cpp_cp_F0-0.log.7133.20140106182015 3768330 -rwx 551643 Jan 7 2014 09:27:51 +00:00 cpp_cp_F0-0.log.7133.20140107092751 3768335 -rwx 1048901 Jan 7 2014 08:56:44 +00:00 cpp_cp_F00.log.7133.2014010708564339313059840 bytes total (30680653824 bytes free)

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

76

Rotating the log files My-ASR1000-2#dir harddisk:/tracelogs/cpp_cp_F0* Directory of harddisk:/tracelogs/cpp_cp_F0* Directory of harddisk:/traceMy-ASR1000-2#test platform software trace slot rp active forwardingmanager rotate Rotated file from: /tmp/rp/trace/stage/fman_rp_R0-0.log.13836.20140107094754, Bytes: 0, Messages: 6535 My-ASR1000-2#test platform software trace slot FP active cpp-control-process rotate Rotated file from: /tmp/fp/trace/stage/cpp_cp_F0-0.log.7133.20140107093650, Bytes: 154027, Messages: 786 My-ASR1000-2#test platform software trace slot FP active forwarding-manager rotate Rotated file from: /tmp/fp/trace/stage/fman-fp_F0-0.log.8247.20140107093738, Bytes: 20170, Messages: 210

OR use My-ASR1000-2#request platform software trace rotate all

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Does not show the rotated file names w/ time stamp  have to hunt them down

Cisco Public

77

The Road to Simplification – Part II Control Plane Unified Show Commands 78

Simplifying the IPsec show commands One show command to rule them all

interface: Virtual-Access1002 Crypto map tag: Virtual-Access1002-head-0, local addr 172.18.0.1

protected vrf: (none) local ident (addr/mask/prot/port): (172.18.0.1/255.255.255.255/47/0) remote ident (addr/mask/prot/port): (17.0.0.26/255.255.255.255/47/0) current_peer 17.0.0.26 port 500 PERMIT, flags={origin_is_acl,} #pkts encaps: 25227, #pkts encrypt: 25227, #pkts digest: 25227 #pkts decaps: 25237, #pkts decrypt: 25237, #pkts verify: 25237 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 0, #pkts compr. failed: 0 #pkts not decompressed: 0, #pkts decompress failed: 0 #send errors 0, #recv errors 0 local crypto endpt.: 172.18.0.1, remote crypto endpt.: 17.0.0.26 plaintext mtu 1458, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0/2 current outbound spi: 0xA7B61FE5(2813730789) PFS (Y/N): N, DH group: none inbound esp sas: spi: 0xA222F391(2720199569) transform: esp-aes esp-sha-hmac , in use settings ={Transport, } conn id: 36130, flow_id: HW:34130, sibling_flags FFFFFFFF80000008, crypto map: Virtual-Access1002-head-0 sa timing: remaining key lifetime (k/sec): (4607974/2137) IV size: 16 bytes replay detection support: Y replay window size: 512 Status: ACTIVE(ACTIVE)

------------------ show platform software ipsec fp active flow identifier 34130 -----------------… ------------------ show platform hardware qfp active feature ipsec sa 1427 -----------------… ------------------ show platform software ipsec fp active encryption-processor context 6502aa4f -----------------… ------------------ show platform software ipsec fp active flow identifier 34129 -----------------… ------------------ show platform hardware qfp active feature ipsec sa 1867 -----------------… ------------------ show platform software ipsec fp active encryption-processor context 2e02aa4e ----------------…

show crypto ipsec sa interface virtual-access 1002 platform or show crypto ipsec sa peer 17.0.0.26 platform

… outbound esp sas: spi: 0xA7B61FE5(2813730789) transform: esp-aes esp-sha-hmac , in use settings ={Transport, } conn id: 36129, flow_id: HW:34129, sibling_flags FFFFFFFF80000008, crypto map: Virtual-Access1002-head-0 sa timing: remaining key lifetime (k/sec): (4607974/2137) IV size: 16 bytes replay detection support: Y replay window size: 512 BRKCRS-3147 © 2014 Cisco and/or its affiliates. All rights reserved. Status: ACTIVE(ACTIVE)

Cisco Public

79

Simplifying the ZBF show commands Three Commands for ZBF under the sky show policy-firewall config platform --show platform software firewall FP active bindings---show platform software firewall RP active bindings---show platform software firewall FP active pairs---show platform software firewall RP active pairs---show platform software firewall FP active parameter-maps---show platform software firewall RP active parameter-maps---show platform software firewall FP active zones---show platform software firewall RP active zones--

show policy-firewall sessions platform | i show platform

--show platform hardware qfp active feature firewall datapath scb any any any any any all any --

show policy-firewall stats platform | i show platform --show platform software firewall FP active statistics---show platform software firewall RP active statistics---show platform hardware qfp active feature firewall runtime---show platform hardware qfp active feature firewall memory---show platform hardware qfp active feature firewall drop---show platform hardware qfp active feature firewall client statistics-BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

80

The Road to Simplification Part III, Deep Data Plane Debugging 81

IOS 3.11

The Packet Tracer and FIA Debugger X-Connect Reset / Pwr Ctrl L2 Switch

IPv4

IPv6

TCAM

Resource DRAM

Temp Sensor

Part Len / BW SRAM

Packet Processor Engine

DDRAM

JTAG Ctrl

NAT

Reset / Pwr Ctrl

BRKCRS-3147

PPE8

PPE4

…

Dispatcher Packet Buffer

SPI Mux

SA table DRAM

RPs

PPE7

PPE3

IP Unicast

PBR Crypto

PPE6

PPE2

Thread 4

MQC Classify

PPE1

Thread 3

Thread 1

FECP

Thread 2

Boot Flash (OBFL,…)

BQS

Output ACL

Input ACL

Packet # 16 Input ACL

QFP Complex

PPE2

Pak Trace ?

RPs

MPLS

Output FIA

Input FIA

EEPROM

Packet Buffer DRAM

MQC Classify NAT

PPE5

NAT

PBR

PPEN

Output ACL

Encaps

NAT

Crypto Cond Dbg ?

If Conditional Debugging is on for feature AND if packet needs to be traced… feature will log its action step by step in cpp_cp_f0-0.log !!

PPE2 Thread 3

Encaps Crypto

Interconnect

ESP

RPs

© 2014 Cisco and/or its affiliates. All rights reserved.

SIPs Cisco Public

82

Platform Conditional Debugging BGL.D.16-ASR1000-1# debug platform condition feature ?

atm

ATM feature

atom

ATOM feature

bridge-domain

Layer2 bridging feature

cft

CFT feature

cxsc

CXSC feature

evc

EVC feature

fw

FW feature

ipsec

IPSEC feature

nbar

NBAR feature

otv

OTV feature

subscriber

Subscriber feature

vpls

VPLS feature

Debugs get populated in cpp_cp_F0-0.log

Same match statement as packet tracer…

BGL.D.16-ASR1000-1#debug platform condition ipv4 172.19.2.1/32 ingress BGL.D.16-ASR1000-1#debug platform condition feature ipsec dataplane submode cce level info BGL.D.16-ASR1000-1#debug platform condition start

Start and stop debugging BRKCRS-3147

Tells which feature to debug

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

83

Conditional Debugger Demonstration

Checking Resource Usage Coming your way in an IOS-XE near you… 85

Unified show CPU platform summary show processes cpu platform Core 0: CPU utilization for five seconds: 1%; one minute: 1%; five minutes: 1% Core 1: CPU utilization for five seconds: 1%; one minute: 1%; five minutes: 1% PID Runtime(ms) uSecs 5Sec 1Min 5Min TTY Process 1 1102 1800 0.20% 0.50% 0.30% 0 init 3 100 1000 0.00% 0.00% 0.05% 0 events/0 4 100 200 0.00% 0.00% 0.00% 0 khelper 6 200 200 0.70% 0.10% 0.00% 0 kthread …

 Simplified CPU usage visualization – system wide  Will display all relevant CPU’s

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

86

Unified show memory platform summary show memory platform summary Total number of processes: 134 Virtual memory : 2822197248 Pages resident : 360197 Major page faults: 1921

 Simplified memory consumption

Minor page faults: 1290831

 Will display all relevant memory

Memory (kB)

 Including TCAM consumption…

Physical Total

: 4127744 : 3874992

Used

: 2231964

Free

: 1643028

Active

: 1438412

Inactive

: 694176

… BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

87

Wrapping up… 88

New Debugging Strategy Well Known

IOS Control Plane • show interface, show ip route, show bgp … • Feature debugging

Platform Control Plane Still Difficult (not overly)

• Unified show commands • Platform show commands • Future: control plane conditional debugging

Data Plane Easy!!

BRKCRS-3147

• Packet Tracer • Forwarding plane conditional debugging • Embedded Packet Capture © 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

89

Call to Action… Visit the World of Solutions: Cisco Campus  Walk-in Labs  Technical Solutions Clinics

 Meet the Engineer

 Lunch Time Table Topics, held in the main Catering Hall  Recommended Reading: For reading material and further resources for this session, please visit www.pearson-books.com/CLMilan2014 BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

90

Complete Your Online Session Evaluation  Complete your online session evaluation  Complete four session evaluations and the overall conference evaluation to receive your Cisco Live T-shirt

BRKCRS-3147

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

91