Troubleshooting Bgp (2210)

2210 1351_06_2000_c2

1

© © 2000, 2000, Cisco Cisco Systems, Systems, Inc. Inc.

Troubleshooting BGP in Large IP Networks Session 2210

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

2

BGP in Large Scale Networks

Scalable Stable Simple 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

3

Avoid the Problem in the First Place • Use simple configurations maintain a consistent policy throughout the AS

• Promote stable networks nail-down your routes use loopback interfaces

• Grow into your network use peer-groups and RRs for scalability 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

4

Agenda

• Basic Tools • Peer Establishment • UPDATE Exchange • Selection Algorithm • Route Reflectors 2210 1351_06_2000_c2

5

© 2000, Cisco Systems, Inc.

Tool Time Basic Tools

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

6

BGP Troubleshooting Tools

• show commands • debug output • Log messages

2210 1351_06_2000_c2

7

© 2000, Cisco Systems, Inc.

show Commands router#show ip bgp ? A.B.C.D IP prefix /, e.g., 35.0.0.0/8 A.B.C.D Network in the BGP routing table to display cidr-only Display only routes with non-natural netmasks community Display routes matching the communities community-list Display routes matching the community-list dampened-paths Display paths suppressed due to dampening filter-list Display routes conforming to the filter-list flap-statistics Display flap statistics of routes inconsistent-as Display only routes with inconsistent origin ASs neighbors Detailed information on TCP and BGP neighbor connections paths Path information peer-group Display information on peer-groups quote-regexp Display routes matching the AS path "regular expression" regexp Display routes matching the AS path regular expression summary Summary of BGP neighbor status | Output modifiers 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

8

show Commands (Cont.)

router#show ip bgp neighbors x.x.x.x ? advertised-routes Display the routes advertised to a BGP neighbor dampened-routes Display the dampened routes received from neighbor flap-statistics Display flap statistics of the routes learned from neighbor paths Display AS paths learned from neighbor received Display information received from a BGP neighbor received-routes Display the received routes from neighbor routes Display routes learned from neighbor | Output modifiers

2210 1351_06_2000_c2

9

© 2000, Cisco Systems, Inc.

The BGP Table router#show ip bgp BG P table version is 9,localrouter ID is 7.72.6.1 Status codes:s suppressed,d dam ped,h history,* valid,> best,i-internal O rigin codes:i-IG P,e -EG P,? -incom plete N etw ork *> 3.0.0.0 *> 5.0.0.0 *> 6.0.0.0 *i *> 7.0.0.0 *> 8.0.0.0/5 *> 17.0.0.0 *i *> 23.0.0.0 *i *> 35.0.0.0 *i 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

N extH op M etric LocPrfW eightPath 0.0.0.0 0 32768 i 0.0.0.0 0 32768 i 6.72.6.2 4294967294 02i 6.72.6.2 4294967294 100 0 2 i 0.0.0.0 0 32768 i 0.0.0.0 0 32768 i 6.72.6.2 4294967294 02i 6.72.6.2 4294967294 100 0 2 i 6.72.6.2 4294967294 02i 6.72.6.2 4294967294 100 0 2 i 6.72.6.2 4294967294 02i 6.72.6.2 4294967294 100 0 2 i

10

The BGP Table (Cont.)

router#show ip bgp 6.0.0.0 BG P routing table entry for 6.0.0.0/8,version 2 Paths:(2 available,best#1) A dvertised to non peer-group peers: 7.25.14.4 7.72.6.3 7.75.7.1 2 6.72.6.2 from 6.72.6.2 (7.72.6.2) O rigin IG P,m etric 4294967294,localpref100,valid, 2 6.72.6.2 from 7.75.7.1 (7.75.7.1) O rigin IG P,m etric 4294967294,localpref100,valid,

2210 1351_06_2000_c2

external,best

internal

© 2000, Cisco Systems, Inc.

11

show ip bgp Summary

router#show ip bgp sum m ary BG P router identifier 7.72.6.1,localA S num ber 1 BG P table version is 9,m ain routing table version 9 8 netw ork entries and 12 paths using 1176 bytes ofm em ory 3 BG P path attribute entries using 144 bytes ofm em ory 1 BG P A S-PA TH entries using 24 bytes ofm em ory BG P activity 8/0 prefixes,12/0 paths N eighbor V A S M sgR cvd M sgSent TblV er InQ O utQ U p/D ow n State/PfxR cd 6.72.6.2 4 2 6885 6882 9 0 0 4d18h 4 7.25.14.4 4 3 6882 6883 9 0 0 4d18h 0 7.72.6.3 4 1 6880 6886 9 0 0 4d18h 0 7.75.7.1 4 1 6884 6885 9 0 0 4d18h 4

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

12

show ip bgp neighbors router#show ip bgp neighbors 6.72.6.2 BG P neighbor is 6.72.6.2,rem ote A S 2,externallink Index 1,O ffset0,M ask 0x2 BG P version 4,rem ote router ID 7.72.6.2 BG P state = Established,table version = 9,up for 4d21h Lastread 00:00:56,lastsend 00:00:48 H old tim e 180,keepalive interval60 seconds N eighbor N LR I negotiation: C onfigured for unicastroutes only Peer negotiated unicastand m ulticastroutes Exchanging unicastroutes only R eceived route refresh capability from peer M inim um tim e betw een advertisem entruns is 30 seconds R eceived 7044 m essages,0 notifications,0 in queue Sent7041 m essages,0 notifications,0 in queue Prefix advertised 4,suppressed 0,w ithdraw n 0 R oute refresh request:received 0,sent0 Inbound path policy configured R oute m ap for incom ing advertisem ents is k C onnections established 1;dropped 0 Lastresetnever N um ber ofunicast/m ulticastprefixes received 4/0 ExternalBG P neighbor m ay be up to 255 hops aw ay. C onnection state is ESTA B,I/O status:1,unread inputbytes:0 Localhost:3.72.6.1,Localport:179 Foreign host:6.72.6.2,Foreign port:11014 2210 1351_06_2000_c2

13

© 2000, Cisco Systems, Inc.

debug ip bgp router#debug ip bgp ? A.B.C.D BGP neighbor address dampening BGP dampening events BGP events keepalives BGP keepalives updates BGP updates

• Remember—can be dangerous! Use only in the lab or If advised by the TAC

• To make a little safer: logging buffered <size> no logging console 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

14

Session Establishment (debug ip bgp ) 16:06:30: BGP: 7.72.6.1 sending OPEN, version 4 16:06:31: BGP: 7.72.6.1 OPEN rcvd, version 4 16:06:31: BGP: 7.72.6.1 rcv OPEN w/ OPTION parameter len: 12 16:06:31: BGP: 7.72.6.1 rcv OPEN w/ option parameter type 2 (Capability) len 6 16:06:31: BGP: 7.72.6.1 OPEN has CAPABILITY code: 1, length 4 16:06:31: BGP: 7.72.6.1 OPEN has MP_EXT CAP for afi/safi: 1/1 16:06:31: BGP: 7.72.6.1 rcv OPEN w/ option parameter type 2 (Capability) len 2 16:06:31: BGP: 7.72.6.1 OPEN has CAPABILITY code: 128, length 0 16:06:31: BGP: 7.75.7.1 passive open 16:06:31: BGP: 7.75.7.1 OPEN rcvd, version 4 16:06:31: BGP: 7.75.7.1 sending OPEN, version 4 16:06:31: BGP: 7.75.7.1 rcv OPEN w/ OPTION parameter len: 12 16:06:31: BGP: 7.75.7.1 rcv OPEN w/ option parameter type 2 (Capability) len 6 16:06:31: BGP: 7.75.7.1 OPEN has CAPABILITY code: 1, length 4 16:06:31: BGP: 7.75.7.1 OPEN has MP_EXT CAP for afi/safi: 1/1 16:06:31: BGP: 7.75.7.1 rcv OPEN w/ option parameter type 2 (Capability) len 2 16:06:31: BGP: 7.75.7.1 OPEN has CAPABILITY code: 128, length 0 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

15

Session Establishment (debug ip bgp events) 17:31:39: BGP: 7.72.6.1 went from Idle to Active 17:32:00: BGP: 7.72.6.1 went from Active to OpenSent 17:32:00: BGP: 7.72.6.1 went from OpenSent to OpenConfirm 17:32:00: BGP: 7.72.6.1 went from OpenConfirm to Established

17:31:59: BGP: 7.75.7.1 went from Idle to Active 17:32:00: BGP: 7.75.7.1 went from Active to Idle 17:32:00: BGP: 7.75.7.1 went from Idle to Connect 17:32:00: BGP: 7.75.7.1 went from Connect to OpenSent 17:32:00: BGP: 7.75.7.1 went from OpenSent to OpenConfirm 17:32:00: BGP: 7.75.7.1 went from OpenConfirm to Established

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

16

Looking at the Updates router#debug ip bgp updates? <1-199> Access list <1300-2699> Access list (expanded range) router#debug ip bgp x.x.x.x updates? <1-199> Access list <1300-2699> Access list (expanded range)

Use an access-list to limit the output! 2210 1351_06_2000_c2

17

© 2000, Cisco Systems, Inc.

debug ip bgp Updates

Peer Address

Prefix Being Advertised

BG P:6.72.6.2 com puting updates,neighbor version 0,table version at0.0.0.0 BG P:6.72.6.2 send U PD A TE 3.0.0.0/8,next3.72.6.1 BG P:,m etric 0,path 1 BG P:6.72.6.2 send U PD A TE 5.0.0.0/8 (chgflags:0x0),next3.72.6.1 BG P:6.72.6.2 send U PD A TE 7.0.0.0/8 (chgflags:0x0),next3.72.6.1 BG P:6.72.6.2 1 updates enqueued (average=56,m axim um =56) BG P:6.72.6.2 update run com pleted,ran for 0m s,neighbor version 0, throttled to 13,check pointnet0.0.0.0 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

NEXT_HOP

13,starting

startversion 13,

18

debug ip bgp Updates (Cont.) BG P:6.72.6.2 rcv U PD A TE 494,path 2 BG P:6.72.6.2 rcv U PD A TE BG P:6.72.6.2 rcv U PD A TE BG P:6.72.6.2 rcv U PD A TE BG P:6.72.6.2 rcv U PD A TE

w /attr:nexthop 6.72.6.2,origin i,m etric about6.0.0.0/8 about17.0.0.0/8 about23.0.0.0/8 about35.0.0.0/8

Peer Address Prefixes in the Same UPDATE

Attributes Apply to All Prefixes

BG P:6.72.6.2 rcv U PD A TE w /attr:nexthop 6.72.6.2,origin i,m etric 294,path 2 1 BG P:6.72.6.2 rcv U PD A TE about3.0.0.0/8 --D EN IED due to:as-path contains our ow n A S; BG P:6.72.6.2 rcv U PD A TE about7.0.0.0/8 --D EN IED due to:as-path contains our ow n A S; 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

19

Logging Neighbor Changes • Generate a log message whenever a BGP neighbor changes state, also indicate reason for reset • Syntax (router subcommand): [no] bgp log-neighbor-changes Typical log messages: %BGP-5-ADJCHANGE: neighbor x.x.x.x Up %BGP-5-ADJCHANGE: neighbor x.x.x.x Down-Remote AS changed

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

20

show ip bgp neighbors x.x.x.x

router#show ip bgp neighbors 7.75.7.1 BGP neighbor is 7.75.7.1, remote AS 2, external link ... Received 194 messages, 1 notifications, 0 in queue Sent 194 messages, 0 notifications, 0 in queue Prefix advertised 0, suppressed 0, withdrawn 0 Route refresh request: received 0, sent 0 Connections established 7; dropped 7 Last reset 00:04:11, due to BGP Notification received, hold time expired Number of unicast/multicast prefixes received 0/0 External BGP neighbor may be up to 255 hops away. No active TCP connection

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

21

Come Meet the Neighbors! Peer Establishment

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

22

Peer Establishment • Routers establish a TCP session Port 179—permit in ACLs IP connectivity (route from IGP)

• OPEN messages are exchanged Peering addresses must match the TCP session Local AS configuration parameters Capabilities negotiation 2210 1351_06_2000_c2

23

© 2000, Cisco Systems, Inc.

Common Problems

• Sessions are not established No IP reachability Incorrect configuration Peering addresses OPEN parameters 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

24

Can’t Establish Session Symptoms routerA #show ip bgp sum m ary BG P router identifier 7.72.6.1,localA S num ber 1 BG P table version is 4,m ain routing table version 4 6 netw ork entries and 6 paths using 774 bytes ofm em ory 2 BG P path attribute entries using 96 bytes ofm em ory 1 BG P A S-PA TH entries using 24 bytes ofm em ory BG P activity 6/0 prefixes,6/0 paths N eighbor V A S M sgR cvd M sgSentTblV er InQ O utQ U p/D ow n State/PfxR cd 6.72.6.2 4 2 0 0 0 0 0 never Idle 7.25.14.4 4 3 4 5 4 0 0 00:01:43 0 7.72.6.3 4 1 0 0 0 0 0 never A ctive 7.75.7.1 4 1 7 5 4 0 0 00:01:55 3

• The peering session is not established! State may change between active, idle and connect 2210 1351_06_2000_c2

25

© 2000, Cisco Systems, Inc.

Can’t Establish Session— Troubleshooting I

• Is the remote-as assigned correctly? Local AS router bgp 1 neighbor 6.72.6.2 rem ote-as 2 neighbor 7.72.6.3 rem ote-as 1

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

eBGP Peer

iBGP Peer

26

Can’t Establish Session— Troubleshooting I (Cont.) • Verify IP connectivity check the routing table use ping/trace to verify two way reachability inspect for ACLs in the path to the neighbor routerA #show ip route 7.72.6.3 R outing entry for 7.72.6.3/32 K now n via "ospf123”,distance 110,m etric 87,type intra area Lastupdate from 27.27.27.254 on PO S5/0,00:09:33 ago R outing D escriptor Blocks: * 27.27.27.254,from 7.72.6.3,00:09:33 ago,via PO S5/0 R oute m etric is 87,traffic share countis 1 routerA #ping 7.72.6.3 Sending 5,100-byte IC M P Echos to 7.72.6.3,tim eoutis 2 seconds: !!!!! Success rate is 100 percent(5/5),round-trip m in/avg/m ax = 28/30/32 m s 2210 1351_06_2000_c2

27

© 2000, Cisco Systems, Inc.

Can’t Establish Session— Troubleshooting I (Cont.) routerA #debug ip bgp BG P debugging is on 10:51:02:BG P:7.72.6.3 open active,delay 6864m s 10:51:09:BG P:7.72.6.3 open active,localaddress 27.27.27.253 10:51:09:BG P:7.72.6.3 open failed:C onnection refused by rem ote host

• Is the remote router configured for BGP? What IP address is the remote router configured to receive? router bgp 1 no synchronization bgp log-neighbor-changes neighbor 7.72.6.1 rem ote-as 1 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

28

Can’t Establish Session— Troubleshooting I (Cont.) The TCP session is always sourced from the closest IP address to the destination!

C

A

27.27.27.254 27.27.27.253

• Configuration: Router A router bgp 1 neighbor 27.27.27.254 remote-as 1 Router C

If redundant paths exist, use loopback interfaces to establish the session.

router bgp 1 neighbor 27.27.27.253 remote-as 1 2210 1351_06_2000_c2

29

© 2000, Cisco Systems, Inc.

Can’t Establish Session— Troubleshooting I (Cont.) router bgp 1 neighbor 7.72.6.3 rem ote-as 1 neighbor 7.72.6.3 update-source Loopback0

Information sourced from the IP address in interface Loopback0

routerA #debug ip tcp transactions 11:19:48:BG P:7.72.6.3 open active,delay 9916m s 11:19:53:TC P:sending R ST,seq 0,ack 3098129121 11:19:53:TC P:sentR ST to 7.7.7.6:11719 from 7.72.6.1:179

• Solution: make sure both routers source the information from the appropriate interface 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

30

Can’t Establish Session—Symptoms routerA #show ip bgp sum m ary BG P router identifier 7.72.6.1,localA S num ber 1 BG P table version is 4,m ain routing table version 4 6 netw ork entries and 6 paths using 774 bytes ofm em ory 2 BG P path attribute entries using 96 bytes ofm em ory 1 BG P A S-PA TH entries using 24 bytes ofm em ory BG P activity 6/0 prefixes,6/0 paths N eighbor V A S M sgR cvd M sgSent TblV er InQ O utQ U p/D ow n State/PfxR cd 6.72.6.2 4 2 0 0 0 0 0 never Idle 7.25.14.4 4 3 385 385 4 0 0 06:22:17 0 7.72.6.3 4 1 42 49 4 0 0 00:00:15 0 7.75.7.1 4 1 388 385 4 0 0 06:22:30 3

• The eBGP session is still having trouble! 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

31

Can’t Establish Session Troubleshooting II • Verify IP connectivity check the routing table use ping/trace to verify two way reachability routerA #show ip route 6.72.6.2 % N etw ork notin table routerA #configure term inal Enter configuration com m ands,one per line. End w ith C N TL/Z. routerA (config)#ip route 6.72.6.2 255.255.255.255 1.1.1.5 routerA #ping 6.72.6.2 Type escape sequence to abort. Sending 5,100-byte IC M P Echos to 6.72.6.2,tim eoutis 2 seconds: !!!!! Success rate is 100 percent(5/5),round-trip m in/avg/m ax = 1/1/1 m s 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

32

Can’t Establish Session— Troubleshooting II (Cont.) • Peering with a loopback interface Advantages Interface is always up Multiple physical paths may exist to reach it Disadvantages Physical link failure may take longer to detect 2210 1351_06_2000_c2

33

© 2000, Cisco Systems, Inc.

Can’t Establish Session— Troubleshooting II (Cont.) routerA #debug ip bgp routerA #debug ip tcp transactions 13:25:30:TC P:sending R ST,seq 0,ack 2030100669 13:25:30:TC P:sentR ST to 6.72.6.2:11041 from 3.72.6.1:179

router bgp 1 neighbor 6.72.6.2 rem ote-as 2 neighbor 6.72.6.2 update-source Loopback1

Neighbor is trying to peer with this IP address

• The debug output indicates the neighbor’s configured peering address 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

34

Can’t Establish Session— Troubleshooting II (Cont.) 13:33:30:TC P:sending R ST,seq 0,ack 2510129645 13:33:30:TC P:sentR ST to 6.72.6.2:11045 from 3.72.6.1:179

• Hint: by default, eBGP peers should be directly connected in this case, the peering address doesn’t match a connected interface in the local router 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

35

Can’t Establish Session— Troubleshooting II (Cont.) routerA #show ip bgp neighbors 6.72.6.2 BG P neighbor is 6.72.6.2,rem ote A S 2,externallink Index 1,O ffset0,M ask 0x2 BG P version 4,rem ote router ID 0.0.0.0 BG P state = Idle,table version = 0 Lastread 00:00:06,lastsend never H old tim e 180,keepalive interval60 seconds N eighbor N LR I negotiation: C onfigured for unicastroutes only M inim um tim e betw een advertisem entruns is 30 seconds R eceived 0 m essages,0 notifications,0 in queue Sent0 m essages,0 notifications,0 in queue Prefix advertised 0,suppressed 0,w ithdraw n 0 R oute refresh request:received 0,sent0 C onnections established 0;dropped 0 Lastresetnever N um ber ofunicast/m ulticastprefixes received 0/0 ExternalBG P neighbor notdirectly connected. N o active TC P connection 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

36

Can’t Establish Session— Troubleshooting II (Cont.) router bgp 1 neighbor 6.72.6.2 rem ote-as 2 neighbor 6.72.6.2 ebgp-m ultihop 255 neighbor 6.72.6.2 update-source Loopback1

• At this point, the session should come up

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

37

Can’t Establish Session— Symptoms routerA #show ip bgp sum m ary BG P router identifier 7.72.6.1,localA S num ber 1 … N eighbor V A S M sgR cvd M sgSent TblV er InQ O utQ U p/D ow n State/PfxR cd 6.72.6.2 4 2 10 26 0 0 0 never A ctive

router bgp 1 neighbor 6.72.6.2 rem ote-as 2 neighbor 6.72.6.2 ebgp-m ultihop 255 neighbor 6.72.6.2 update-source Loopback1

• Still having trouble! Connectivity issues have already been checked and corrected. 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

38

Can’t Establish Session— Troubleshooting II (Cont.) 14:06:37:BG P:6.72.6.2 open active,localaddress 3.72.6.1 14:06:37:BG P:6.72.6.2 w entfrom A ctive to O penSent 14:06:37:BG P:6.72.6.2 sending O PEN ,version 4 14:06:37:BG P:6.72.6.2 received N O TIFIC A TIO N 2/2 (peer in w rong A S)2 bytes 0001 14:06:37:BG P:6.72.6.2 rem ote close,state C LO SEW A IT 14:06:37:BG P:service resetrequests 14:06:37:BG P:6.72.6.2 w entfrom O penSentto Idle 14:06:37:BG P:6.72.6.2 closing

• If an error is detected, a notification is sent and the session is closed In this case the remote router had a bad configuration 2210 1351_06_2000_c2

39

© 2000, Cisco Systems, Inc.

OPEN Message 0

1

2

3

4

5

6

7

8

9

10

11 12 13 14 15 16 17 18 19

20

21 22 23 24 25 26 27 28 29

30

31

Version My Autonomous System Hold Time BGP Identifier Opt. Parm. Len. Optional Parameters

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

40

Capabilities Negotiation • Allows for the advertisement of capabilities (type 2) • Backwards compatible New error subcode introduced to indicate which capabilities are not supported—the session must be reset 2210 1351_06_2000_c2

Capability Code (1 Octet) Capability Length (1 Octet) Capability Value (Variable)

draft-ietf-idr-bgp4-cap-neg, Mar. 2000

© 2000, Cisco Systems, Inc.

41

Where’s the Beef? UPDATE Exchange

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

42

UPDATE Exchange

• Once the session has been established, UPDATEs are exchanged all the locally known routes only the bestpath is advertised

• Incremental UPDATE messages are exchanged afterwards 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

43

Propagation Decisions

• bestpath received from eBGP peer advertise to all peers

• bestpath received from iBGP peer advertise only to eBGP peers a full iBGP mesh must exist 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

44

Common Problems

• Missing routes No iBGP full mesh Filters: routes are not received/sent

• Slow convergence

2210 1351_06_2000_c2

45

© 2000, Cisco Systems, Inc.

UPDATE Filters

• Type of filters Prefix filters AS_PATH filters Community filters Any attribute may be used in a route-map

• Applied incoming and/or outgoing 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

46

Missing Routes— Troubleshooting Steps • Determine which filters are applied to the BGP session show ip bgp neighbors x.x.x.x Look at the configuration

• Examine the route and pick out the relevant attributes show ip bgp x.x.x.x 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

47

Missing Routes— Troubleshooting Steps (Cont.)

• Compare the route against the filters • If no match is found Use route-refresh or soft-reconfiguration Filter the updates through an ACL to determine where the problem is

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

48

Missing Routes—Symptoms • Missing 4.0.0.0/8 in 7.75.7.1 (routerA) not received from 7.72.6.3 (routerB) routerB#sh ip bgp nei 7.75.7.1 advertised-routes | include 4.0.0.0 *> 4.0.0.0 0.0.0.0 0 32768 i

routerB shows that the route was advertised to routerA!

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

49

Missing Routes—Troubleshooting

routerA#show access-lists 10 Standard IP access list 10 permit 4.0.0.0 routerA#debug ip bgp 7.72.6.3 updates 10 BGP updates debugging is on for access list 10 for neighbor 7.72.6.3 routerA#clear ip bgp 7.72.6.3 in 01:22:41: BGP: 7.72.6.3 rcv UPDATE w/ attr: nexthop 7.72.6.3, origin i, metric 0, path 2 01:22:41: BGP: 7.72.6.3 rcv UPDATE about 4.0.0.0/8 -- DENIED due to: distribute/prefix-list;

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

50

Missing Routes— Troubleshooting (Cont.) router bgp 1 no synchronization bgp log-neighbor-changes neighbor 7.72.6.3 rem ote-as 2 neighbor 7.72.6.3 ebgp-m ultihop 255 neighbor 7.72.6.3 update-source Loopback0 neighbor 7.72.6.3 prefix-listfilter in ! ip prefix-listfilter seq 5 deny 4.0.0.0/8 ip prefix-listfilter seq 10 perm it0.0.0.0/0 le 32

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

51

Slow Convergence—Symptoms • The eBGP peering is established, but convergence is not complete even after several hours routerA #show ip bgp sum m ary ... N eighbor V A S M sgR cvd M sgSentTblV er InQ O utQ U p/D ow n State/PfxR cd 150.10.10.1 4 1 3550 3570 847 0 206 05:53:51 100

• Possible causes Remote router is not healthy (OutQ) Lower layer problems (IP) 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

52

Slow Convergence— Troubleshooting Reply to request 0 Record route: (156.1.1.2) (195.5.5.1) (10.105.1.134) (150.10.10.1) (10.105.1.76) (195.5.5.2) (156.1.1.1) (211.211.211.1) <*>

router bgp 1 neighbor 150.10.10.1 remote-as 2 neighbor 150.10.10.1 ebgp-multihop 2 neighbor 150.10.10.1 update-source Loopback0 routerA#show ip route 150.10.10.1

Reply to request 1 Record route: (10.105.1.69) (140.10.50.5) (150.10.10.1) (140.10.50.6) (10.105.1.71) (211.211.211.1) <*>

Routing entry for 150.10.10.1/32 Routing Descriptor Blocks: 10.105.1.71, from 150.20.20.1, 00:06:14 ago, via POS2/1/0 * 156.1.1.1, from 150.20.20.1, 00:06:14 ago, via POS2/1/1 routerA#ping 150.10.10.1 Sending 5, 100-byte ICMP Echos to 150.10.10.1:

!!!!!

Success is 100 percent, round-trip min/avg/max = 4/64/296 ms 2210 1351_06_2000_c2

Ping with route record option. 53

© 2000, Cisco Systems, Inc.

Slow Convergence— Troubleshooting (Cont.) eBGP Peering

OC-3

OC-3

A

B OC-3

OC-3

OC-3

T3 router bgp 1 neighbor 150.10.10.1 remote-as 2 neighbor 150.10.10.1 ebgp-multihop 2 neighbor 150.10.10.1 update-source Loopback0 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

Longest path has more than 2 hops to the destination. Use higher TTL! 54

Pick One, Only One! Route Selection Process

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

55

Route Selection Process

• A common policy should be maintained across the AS to guarantee loop-free operation Not all routers may select the same path

• Filters may be used to modify or add attributes, affecting the selection algorithm 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

56

Common Problems • Inconsistent decision/policy MED External paths Communities By default, communities are not propagated neighbor x.x.x.x send-community 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

57

Inconsistent Decision— Symptom I • The bestpath changes every time the peering is reset. routerA #sh ip bgp 160.100.0.0 BG P routing table entry for 160.100.0.0/16,version 40 Paths:(3 available,best#3,advertised over IBG P,EBG P) 1 204.146.33.10 from 204.146.33.10 (204.146.33.1) O rigin IG P,m etric 0,localpref100,valid,internal 3 204.146.33.66 from 204.146.33.66 (204.146.33.2) O rigin IG P,m etric 20,localpref100,valid,internal 3 204.146.33.6 from 204.146.33.6 (10.4.1.1) O rigin IG P,m etric 30,valid,external,best 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

58

Inconsistent Decision— Symptom I (Cont.) routerA #sh ip bgp 160.100.0.0 BG P routing table entry for 160.100.0.0/16,version 2 Paths:(3 available,best#3,advertised over EBG P) 1 204.146.33.10 from 204.146.33.10 (204.146.33.1) O rigin IG P,m etric 0,localpref100,valid,internal 3 204.146.33.6 from 204.146.33.6 (10.4.1.1) O rigin IG P,m etric 30,valid,external 3 204.146.33.66 from 204.146.33.66 (204.146.33.2) O rigin IG P,m etric 20,localpref100,valid,internal,best

• Same paths, but different result! 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

59

Inconsistent Decision— Symptom I (Cont.) routerA #sh ip bgp 160.100.0.0 BG P routing table entry for 160.100.0.0/16,version 12 Paths:(3 available,best#3,advertised over EBG P) 3 204.146.33.6 from 204.146.33.6 (10.4.1.1) O rigin IG P,m etric 30,valid,external 3 204.146.33.66 from 204.146.33.66 (204.146.33.2) O rigin IG P,m etric 20,localpref100,valid,internal 1 204.146.33.10 from 204.146.33.10 (204.146.33.1) O rigin IG P,m etric 0,localpref100,valid,internal,best

• Different result…again!! 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

60

Deterministic MED

• By default, the prefixes are compared in order of arrival it may result in inconsistent decisions use bgp deterministic-med the bestpath is recalculated as soon as the command is entered enable in all the routers in the AS 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

61

Deterministic MED—Operation

• The paths are ordered by peer-AS • The bestpath for each group is selected • The overall bestpath results from comparing the winners in each group 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

62

Deterministic MED—Result routerA #sh ip bgp 160.100.0.0 BG P routing table entry for 160.100.0.0/16,version 15 Paths:(3 available,best#1,advertised over EBG P) 1 204.146.33.10 from 204.146.33.10 (204.146.33.1) O rigin IG P,m etric 0,localpref100,valid,internal,best 3 204.146.33.66 from 204.146.33.66 (204.146.33.2) O rigin IG P,m etric 20,localpref100,valid,internal 3 204.146.33.6 from 204.146.33.6 (10.4.1.1) O rigin IG P,m etric 30,valid,external

• The bestpath will always be the same! 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

63

Inconsistent Decision— Symptom II • The bestpath changes every time the peering is reset routerA #show ip bgp 7.0.0.0 BG P routing table entry for 7.0.0.0/8,version 15 Paths:(2 available,best#2) N otadvertised to any peer 2 1.1.1.5 from 1.1.1.5 (1.1.1.1) O rigin IG P,m etric 0,localpref100,valid,external 2 21.21.21.254 from 21.21.21.254 (7.75.7.1) O rigin IG P,m etric 0,localpref100,valid,external,best

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

64

Inconsistent Decision— Symptom II (Cont.) routerA #show ip bgp 7.0.0.0 BG P routing table entry for 7.0.0.0/8,version 17 Paths:(2 available,best#2) N otadvertised to any peer 2 21.21.21.254 from 21.21.21.254 (7.75.7.1) O rigin IG P,m etric 0,localpref100,valid,external 2 1.1.1.5 from 1.1.1.5 (1.1.1.1) O rigin IG P,m etric 0,localpref100,valid,external,best

• The “oldest” external is the bestpath. All other attributes are the same Stability enhancement! 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

65

Route Reflectors Playing with Mirrors

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

66

Route Reflectors

• Provide additional control to allow router to advertise (reflect) iBGP learned routes to other iBGP peers Method to reduce the size of the iBGP mesh

• Normal BGP speakers can coexist Only the RR has to support this feature 2210 1351_06_2000_c2

67

© 2000, Cisco Systems, Inc.

Route Reflectors—Terminology Route Reflector

Non-Client

Clusters Clients

Clients

Lines Represent Both Physical Links and BGP Logical Connections 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

68

Reflection Decisions

• Once the best path is selected: From non-client reflect to all clients From client → reflect to all non-clients AND other clients From eBGP peer → reflect to all clients and non-clients 2210 1351_06_2000_c2

69

© 2000, Cisco Systems, Inc.

Common Problems

• Missing routes • Routing loops and “close calls”

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

70

Missing Routes—Symptoms

• At least one route is missing from at least one router in the network. routerA #show ip bgp 4.0.0.0 % N etw ork notin table routerA #show ip bgp sum m ary BG P router identifier 7.25.14.4,localA S num ber 1 BG P table version is 1,m ain routing table version 1 … N eighbor V A S M sgR cvd M sgSentTblV er InQ O utQ U p/D ow n State/PfxR cd 7.72.6.2 4 1 7 7 1 0 0 00:04:18 0

2210 1351_06_2000_c2

71

© 2000, Cisco Systems, Inc.

Missing Routes—Troubleshooting • Check routers for filters routerA # router bgp 1 no synchronization neighbor 7.72.6.2 rem ote-as 1

routerB# router bgp 1 no synchronization bgp cluster-id 0.0.0.5 neighbor 7.25.14.4 rem ote-as 1 neighbor 7.25.14.4 route-reflector-client neighbor 7.72.6.1 rem ote-as 1

routerC

routerB#show ip bgp 4.0.0.0 % N etw ork notin table 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

72

Missing Routes— Troubleshooting I • Follow the path where the routes should have been learned routerC # router bgp 1 no synchronization bgp cluster-id 0.0.0.5 neighbor 7.72.6.2 rem ote-as 1 neighbor 7.75.7.1 rem ote-as 1 neighbor 7.75.7.1 route-reflector-client

Same Cluster-ID!

routerC #show ip bgp |include 4.0.0.0 *>i4.0.0.0 7.72.6.3 0 100 0 2 i 2210 1351_06_2000_c2

73

© 2000, Cisco Systems, Inc.

Missing Routes— Troubleshooting II • Alternative way to find duplicate cluster-id use route-refresh + debug ip bgp updates ACL routerB#clear ip bgp 7.72.6.1 in 21:45:40:BG P:7.72.6.1 rcv U PD A TE w /attr:nexthop origin i,localpref100,m etric 0,path 2 21:45:40:BG P:7.72.6.1 rcv U PD A TE about4.0.0.0/8 -to:reflected from the sam e cluster;

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

7.72.6.3, D EN IED due

74

Missing Routes— Troubleshooting (Cont.) • Clusters with multiple RRs If the RRs have the same cluster-id, all the clients must peer with all the reflectors

RR

B

RR

C

A

D

Lines Represent Both Physical Links and BGP Logical Connections

If not needed for administration, don’t assign a cluster-id for added flexibility! 2210 1351_06_2000_c2

E

75

© 2000, Cisco Systems, Inc.

Routing Loop—Symptom

A

routerD#traceroute 7.1.1.1 1 1.1.1.2 24 msec 24 msec 40 msec rtrB

2 156.1.1.1 28 msec 48 msec 24 msec

rtrC

3 156.1.1.2 24 msec 24 msec 24 msec

Loop!

B

C

4 156.1.1.1 28 msec 28 msec 24 msec 5 156.1.1.2 28 msec 28 msec 28 msec

D

6 156.1.1.1 28 msec 28 msec 32 msec Lines Represent Physical Connections 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

76

Routing Loop—Troubleshooting Verify Routing Information routerC#show ip bgp 7.0.0.0

routerB#show ip bgp 7.0.0.0

BGP routing table entry for 7.0.0.0/8

BGP routing table entry for 7.0.0.0/8

1

1 150.10.10.1 (metric 115) from 150.10.10.1 (150.20.20.1)

156.1.1.2 from 156.1.1.2 (212.212.212.1) Origin IGP, valid, internal, best

Origin IGP, valid, external, best routerC#show ip route 150.10.10.1

routerB#show ip route 156.1.1.2

Routing entry for 150.10.10.1/32

Routing entry for 156.1.1.0/24

Routing Descriptor Blocks:

Routing Descriptor Blocks:

* 156.1.1.1, from 150.20.20.1, via Ethernet2/1/1

* directly connected, via Ethernet1

Addresses on Same Subnet! 2210 1351_06_2000_c2

77

© 2000, Cisco Systems, Inc.

Routing Loop— Troubleshooting (Cont.)

• Check configuration routerC# router bgp 134 neighbor 150.10.10.1 remote-as 1 neighbor 150.10.10.1 ebgp-multihop 255 neighbor 150.10.10.1 update-source Loopback0 neighbor 156.1.1.1 remote-as 134 neighbor 156.1.1.1 route-reflector-client neighbor 156.1.1.1 next-hop-self ! ip route 150.10.10.1 255.255.255.255 s0 250 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

A-RR

B

C-RR D

Lines Represent Physical Connections 78

Routing Loop— Troubleshooting (Cont.)

• Solution Establish the eBGP peering permanently through the “backup” link

A-RR

B

Use LOCAL_PREF or MED to break any tie!

C-RR D

Lines Represent Physical Connections 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

79

Close Call—Symptom • The bestpath is not being followed to an external destination routerA #show ip route 4.4.4.4 R outing entry for 4.0.0.0/8 K now n via "bgp 1",distance 200,m etric 0 Tag 2,type internal Lastupdate from 6.72.6.3 00:25:45 ago R outing D escriptor Blocks: * 6.72.6.3,from 7.75.7.1,00:25:45 ago R oute m etric is 0,traffic share countis 1 A S H ops 1,BG P netw ork version 0 Expected to go out though the NEXT_HOP in the update. 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

80

Close Call—Symptom (Cont.) routerA #traceroute 4.4.4.4 1 1.1.1.4 0 m sec 0 m sec 0 m sec 2 8.25.14.3 8 m sec 4 m sec 4 m sec 3 172.18.176.1 8 m sec 4 m sec 4 m sec 4 161.44.0.56 8 m sec 8 m sec 8 m sec 5 161.44.0.18 8 m sec 8 m sec 4 m sec 6 4.4.4.4 4 m sec 5 m sec 4 m sec

• All eBGP peers are configured with their interface address The NEXT_HOP is expected in the trace 2210 1351_06_2000_c2

81

© 2000, Cisco Systems, Inc.

Close Call— Troubleshooting (Cont.) • Verify configuration Check for alternate routes router bgp 1 no synchronization neighbor 7.75.7.1 rem ote-as 1 neighbor 7.75.7.1 update-source Loopback0

routerC

routerA #show ip bgp 4.0.0.0 BG P routing table entry for 4.0.0.0/8,version 2 Paths:(1 available,best#1) N otadvertised to any peer 2 6.72.6.3 (m etric 103)from 7.75.7.1 (7.75.7.1) O rigin IG P,m etric 0,localpref100,valid,internal,best 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

82

Close Call— Troubleshooting (Cont.) router bgp 1 no synchronization bgp log-neighbor-changes neighbor 7.72.6.1 rem ote-as 1 neighbor 7.72.6.1 update-source Loopback0 neighbor 7.72.6.1 route-reflector-client neighbor 7.72.6.2 rem ote-as 1 neighbor 7.72.6.2 update-source Loopback0 neighbor 7.72.6.2 route-reflector-client neighbor 6.72.6.3 rem ote-as 2

routerA routerB

• routerC is a RR with two clients 2210 1351_06_2000_c2

83

© 2000, Cisco Systems, Inc.

Close Call— Troubleshooting (Cont.) routerC #show ip bgp 4.0.0.0 BG P routing table entry for 4.0.0.0/8,version 2 2,(R eceived from a R R -client) 8.25.14.3 (m etric 3)from 7.72.6.1 (7.72.6.1) O rigin IG P,m etric 0,localpref100,valid,internal 2 6.72.6.3 (m etric 2)from 6.72.6.3 (6.72.6.3) O rigin IG P,m etric 0,localpref100,valid,external,best

routerC #traceroute 4.4.4.4 1 6.72.6.3 0 m sec 4 m sec 0 m sec 2 161.44.0.56 0 m sec 4 m sec 0 m sec 3 161.44.0.18 0 m sec 4 m sec 0 m sec 4 4.4.4.4 0 m sec 4 m sec 0 m sec

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

Expected Path!

84

Close Call— Troubleshooting (Cont.) routerB#traceroute 4.4.4.4 1 8.25.14.3 8 m sec 16 m sec 8 m sec 2 172.18.176.1 16 m sec 12 m sec 16 m sec 3 161.44.0.48 12 m sec 16 m sec 12 m sec 4 161.44.0.15 16 m sec 12 m sec 16 m sec 5 4.4.4.4 8 m sec 8 m sec 8 m sec

• Output from the other client follows the same exit as routerA 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

85

Close Call— Troubleshooting (Cont.) routerB#show ip bgp 4.0.0.0 BGP routing table entry for 4.0.0.0/8, version 13 2 8.25.14.3 (metric 2) from 8.25.14.3 (8.25.14.3) Origin IGP, metric 0, localpref 100, valid, external, best 2 6.72.6.3 (metric 50) from 7.75.7.1 (7.75.7.1) Origin IGP, metric 0, localpref 100, valid, internal

• routerB is following the correct path! 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

86

Close Call— Troubleshooting (Cont.) • The logical connection between routerC (RR) and routerA provides the route

A

B CC

The physical path is followed D

E

Logical Connection 4.0.0.0/8 2210 1351_06_2000_c2

87

© 2000, Cisco Systems, Inc.

Summary/Tips • Isolate the problem! • Use ACLs when enabling debug commands • Enable bgp log-neighbor-changes • IP reachability must exist for sessions to be established Learned from IGP Make sure the source and destination addresses match the configuration 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

88

Summary/Tips • Use loopback interfaces for stability and where multiple paths exist • Use common filters Keep them simple!

• Maintain a consistent policy throughout the AS • Use deterministic-med 2210 1351_06_2000_c2

89

© 2000, Cisco Systems, Inc.

Summary/Tips

• Select the appropriate knob/attribute for the job Learn the decision algorithm

• Route reflectors Follow the physical topology Define a cluster-id only if administratively needed 2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

90

Troubleshooting BGP in Large IP Networks Session 2210

2210 1351_06_2000_c2

91

© 1999, 2000, Cisco Systems, Inc.

Please Complete Your Evaluation Form Session 2210

2210 1351_06_2000_c2

© 1999, 2000, Cisco Systems, Inc.

92

2210 1351_06_2000_c2

© 2000, Cisco Systems, Inc.

93