Task
Refer to the diagram, we need to design iBGP to meet the following requirements:
- P2 and P5 routers work as Route-Reflectors, with different cluster ID of 2.2.2.2 and 5.5.5.5
- Each RR will have all other routers, except the other RR, as its clients.
- The two RR peer with each other via a normal iBGP session.
Configuration
Route Reflectors:
p2@vr-device:p2> show configuration protocols bgp
group Cluster-2222 {
type internal;
local-address 192.168.5.2;
export static-to-bgp;
cluster 2.2.2.2;
neighbor 192.168.5.1;
neighbor 192.168.5.3;
neighbor 192.168.5.4;
neighbor 192.168.5.6;
}
group Core {
type internal;
local-address 192.168.5.2;
export static-to-bgp;
neighbor 192.168.5.5;
}
p5@vr-device:p5> show configuration protocols bgp
group Cluster-5555 {
type internal;
local-address 192.168.5.5;
export static-to-bgp;
cluster 5.5.5.5;
neighbor 192.168.5.1;
neighbor 192.168.5.3;
neighbor 192.168.5.4;
neighbor 192.168.5.6;
}
group Core {
type internal;
local-address 192.168.5.5;
export static-to-bgp;
neighbor 192.168.5.2;
}
RR clients, with p1 router config as an example:
p1@vr-device:p1> show configuration protocols bgp
group IBGP {
type internal;
local-address 192.168.5.1;
export static-to-bgp;
neighbor 192.168.5.2;
neighbor 192.168.5.5;
}
Note that each RR client needs only to peer with the two RR. This solves the n^2 scalability problem associated with the total number of ibgp peers required (n x (n-1)/2 to be exact). In fact, for BGP to function, a client only need to peer with one RR, but two are recommended for redundancy. With redundant RR design, in case one RR fails, all clients can still maintain route exchange via the other RR.
Verification
Check BGP status summary
p1@vr-device:p1> show bgp summary
Groups: 1 Peers: 2 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0 10 5 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.168.5.2 65000 70 61 0 3 25:55 4/5/5/0 0/0/0/0
192.168.5.5 65000 72 67 0 3 25:48 1/5/5/0 0/0/0/0
p2@vr-device:p2> show bgp summary
Groups: 2 Peers: 5 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0 9 5 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.168.5.1 65000 69 97 0 1 26:11 1/1/1/0 0/0/0/0
192.168.5.3 65000 69 96 0 1 26:09 1/1/1/0 0/0/0/0
192.168.5.4 65000 60 68 0 1 25:56 1/1/1/0 0/0/0/0
192.168.5.5 65000 61 61 0 1 24:18 1/5/5/0 0/0/0/0
192.168.5.6 65000 63 88 0 1 24:07 1/1/1/0 0/0/0/0
Check routing table
p1@vr-device:p1> show route protocol bgp
inet.0: 27 destinations, 32 routes (27 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.123.2.0/24 *[BGP/170] 00:23:37, localpref 100, from 192.168.5.2
AS path: I
> to 172.22.201.2 via em3.12
[BGP/170] 00:21:44, localpref 100, from 192.168.5.5
AS path: I
> to 172.22.201.2 via em3.12
10.123.3.0/24 *[BGP/170] 00:22:33, localpref 100, from 192.168.5.2
AS path: I
> to 172.22.201.2 via em3.12
[BGP/170] 00:22:33, localpref 100, from 192.168.5.5
AS path: I
> to 172.22.201.2 via em3.12
10.123.4.0/24 *[BGP/170] 00:22:38, localpref 100, from 192.168.5.2
AS path: I
> to 172.22.202.2 via em3.14
[BGP/170] 00:22:38, localpref 100, from 192.168.5.5
AS path: I
> to 172.22.202.2 via em3.14
10.123.5.0/24 *[BGP/170] 00:23:30, localpref 100, from 192.168.5.5
AS path: I
> to 172.22.201.2 via em3.12
to 172.22.202.2 via em3.14
[BGP/170] 00:21:44, localpref 100, from 192.168.5.2
AS path: I
> to 172.22.201.2 via em3.12
to 172.22.202.2 via em3.14
10.123.6.0/24 *[BGP/170] 00:21:33, localpref 100, from 192.168.5.2
AS path: I
> to 172.22.201.2 via em3.12
to 172.22.202.2 via em3.14
[BGP/170] 00:21:48, localpref 100, from 192.168.5.5
AS path: I
> to 172.22.201.2 via em3.12
to 172.22.202.2 via em3.14
Tip: A quick way to check the number of routes using command pipe
p1@vr-device:p1> show route protocol bgp 10/8 | match 10.123 | count
Count: 5 lines
Look closer to a specific route, we see that P1 router receives the prefix behind P6 router via the two cluster RRs (P2 and P5)
p1@vr-device:p1> show route protocol bgp 10.123.6.0
inet.0: 27 destinations, 32 routes (27 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.123.6.0/24 *[BGP/170] 00:25:44, localpref 100, from 192.168.5.2
AS path: I
> to 172.22.201.2 via em3.12
to 172.22.202.2 via em3.14
[BGP/170] 00:25:59, localpref 100, from 192.168.5.5
AS path: I
> to 172.22.201.2 via em3.12
to 172.22.202.2 via em3.14
p1@vr-device:p1> show route protocol bgp 10.123.6.0 detail
inet.0: 27 destinations, 32 routes (27 active, 0 holddown, 0 hidden)
10.123.6.0/24 (2 entries, 1 announced)
*BGP Preference: 170/-101
Next hop type: Indirect
Address: 0x8f939a0
Next-hop reference count: 4
Source: 192.168.5.2
Next hop type: Router, Next hop index: 131092
Next hop: 172.22.201.2 via em3.12, selected
Next hop: 172.22.202.2 via em3.14
Protocol next hop: 192.168.5.6
Indirect next hop: 8fae000 131099
State:
Local AS: 65000 Peer AS: 65000
Age: 26:06 Metric2: 3
Task: BGP_65000.192.168.5.2+62938
Announcement bits (2): 0-KRT 5-Resolve tree 1
AS path: I (Originator) Cluster list: 2.2.2.2
AS path: Originator ID: 192.168.5.6
Accepted
Localpref: 100
Router ID: 192.168.5.2
BGP Preference: 170/-101
Next hop type: Indirect
Address: 0x8f939a0
Next-hop reference count: 4
Source: 192.168.5.5
Next hop type: Router, Next hop index: 131092
Next hop: 172.22.201.2 via em3.12, selected
Next hop: 172.22.202.2 via em3.14
Protocol next hop: 192.168.5.6
Indirect next hop: 8fae000 131099
State:
Inactive reason: Not Best in its group - Update source
Local AS: 65000 Peer AS: 65000
Age: 26:21 Metric2: 3
Task: BGP_65000.192.168.5.5+179
AS path: I (Originator) Cluster list: 5.5.5.5
AS path: Originator ID: 192.168.5.6
Accepted
Localpref: 100
Router ID: 192.168.5.5
Verify with tracing
p1@vr-device:p1> traceroute 10.123.6.1
traceroute to 10.123.6.1 (10.123.6.1), 30 hops max, 40 byte packets
1 172.22.201.2 (172.22.201.2) 0.281 ms 0.323 ms 0.292 ms
2 172.22.206.2 (172.22.206.2) 0.455 ms 0.448 ms 0.424 ms
3 172.22.207.2 (172.22.207.2) 0.535 ms 0.611 ms 0.599 ms
4 172.22.207.2 (172.22.207.2) 0.561 ms !H 0.937 ms !H 0.567 ms !H
Cluster ID
What happen if P2 and P5 share the same Cluster ID?
Cluster ID is originally defined to identify and stop routing loops. A RR does not accept route tagged with its own cluster ID. In case that P5 also use a cluster ID of 2.2.2.2 for its own cluster, then routes learnt by P5 from its clients and advertised to P2 will not be accepted by P2. P2 think that there’s a loop and just simply ignore the route updates. On the other hand, those routes originated by P5 not tagged with a cluster ID of 2.2.2.2, such as 10.123.5.0/24, is accepted by P2.
See the log below when P2 and P5 shares Cluster ID of 2.2.2.2
p2@vr-device:p2> show configuration protocols bgp
traceoptions {
file trace-bgp-p2;
flag route receive;
flag update receive;
}
Aug 8 22:38:44.874564
Aug 8 22:38:44.874564 BGP RECV 192.168.5.5+57694 -> 192.168.5.2+179
Aug 8 22:38:44.874599 BGP RECV message type 2 (Update) length 62
Aug 8 22:38:44.874618 bgp_read_v4_update: peer 192.168.5.5 (Internal AS 65000) UPDATE with local cluster-id (0.0.2.2) in clusterlist
Aug 8 22:38:44.874636 bgp_rcv_nlri: 10.123.3.0/24
Aug 8 22:38:44.874645 bgp_read_v4_message: done with 192.168.5.5 (Internal AS 65000) received 62 octets 1 update 1 route
Aug 8 22:38:48.876036
Aug 8 22:38:48.876036 BGP RECV 192.168.5.5+57694 -> 192.168.5.2+179
Aug 8 22:38:48.876067 BGP RECV message type 2 (Update) length 62
Aug 8 22:38:48.876080 bgp_read_v4_update: peer 192.168.5.5 (Internal AS 65000) UPDATE with local cluster-id (0.0.2.2) in clusterlist
Aug 8 22:38:48.876104 bgp_rcv_nlri: 10.123.4.0/24
Aug 8 22:38:48.876113 bgp_read_v4_message: done with 192.168.5.5 (Internal AS 65000) received 62 octets 1 update 1 route
Aug 8 22:38:52.877937
Aug 8 22:38:52.877937 BGP RECV 192.168.5.5+57694 -> 192.168.5.2+179
Aug 8 22:38:52.877961 BGP RECV message type 2 (Update) length 62
Aug 8 22:38:52.877976 bgp_read_v4_update: peer 192.168.5.5 (Internal AS 65000) UPDATE with local cluster-id (0.0.2.2) in clusterlist
Aug 8 22:38:52.877988 bgp_rcv_nlri: 10.123.6.0/24
Aug 8 22:38:52.877996 bgp_read_v4_message: done with 192.168.5.5 (Internal AS 65000) received 62 octets 1 update 1 route
p2@vr-device:p2> show route receive-protocol bgp 192.168.5.5
inet.0: 26 destinations, 26 routes (26 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 10.123.5.0/24 192.168.5.5 100 I
In this design, clients will maintain connectivity between them if either P2 or P5 failure, as well as one single link between a client and one of its RR. However, client will lose connectivity to the RR if the link between the two fails. Similarly, if P1 loses the link to P5 AND P6 loses the link to P2, then P1 will not have reachablity to P6 !
As a best practice, keep the Cluster ID unique to maximise the redundancy (although there may be redundant routing updates). The routing update loops does not exist, even if there is a group of RRs that form a fully or partly meshed peering.
Consider the following topologies:
In this topology, P1, P2, P5 form a RR fully meshed group. Each one is clients of other one. P4 form iBGP with P1 to inject a route (10.123.4.0/24) for demonstration purpose.
In the second topology, P2 and P5 are client of P1 (Cluster ID 1.1.1.1).
P1 is client of P2 (Cluster ID 2.2.2.2). P1 is client of P5 (Cluster ID 5.5.5.5). P2 and P5 peer via a normal iBGP, to form a mesh of mix RR and iBGP between P1, P2, P5. Finally P4 form a normal iBGP peering with P1 to inject 10.123.4.0/24 for the demonstration purpose.
In both cases, there is no loop, as both Originator ID, and Cluster List (of all Cluster ID) are included in ALL iBGP updates between RRs and RR clients, as well as between normal iBGP peers. They are used for routing (and routing updates) loop prevention. Cluster ID list is also used for path selection (similar to AS-path), where a shorter cluster list length is preferred.
Below is the debug messages on P1 for both cases. It receives updates for 10.123.4.0/24 from both P2 and P5, but reject the route, because it see its own Cluster ID in the updates.
Aug 11 13:36:13.865086 BGP RECV 192.168.5.2+179 -> 192.168.5.1+49556
Aug 11 13:36:13.865102 BGP RECV message type 2 (Update) length 70
Aug 11 13:36:13.865107 BGP RECV Update PDU length 70
Aug 11 13:36:13.865110 BGP RECV flags 0x40 code Origin(1): IGP
Aug 11 13:36:13.865113 BGP RECV flags 0x40 code ASPath(2) length 0:
Aug 11 13:36:13.865116 BGP RECV flags 0x40 code NextHop(3): 192.168.5.4
Aug 11 13:36:13.865119 BGP RECV flags 0x40 code LocalPref(5): 100
Aug 11 13:36:13.865122 BGP RECV flags 0x80 code Originator_Id(9) 192.168.5.4
Aug 11 13:36:13.865125 BGP RECV flags 0x80 code Cluster_List(10): 2.2.2.2 5.5.5.5 1.1.1.1
Aug 11 13:36:13.865130 BGP RECV 10.123.4.0/24
Aug 11 13:36:13.865142 bgp_read_v4_update: peer 192.168.5.2 (Internal AS 65000) UPDATE with local cluster-id (0.0.1.1) in clusterlist
Aug 11 13:36:13.865151 bgp_rcv_nlri: 10.123.4.0/24
Aug 11 13:36:13.867383 BGP RECV 192.168.5.5+179 -> 192.168.5.1+62193
Aug 11 13:36:13.867392 BGP RECV message type 2 (Update) length 27
Aug 11 13:36:13.866815 BGP RECV Update PDU length 70
Aug 11 13:36:13.866818 BGP RECV flags 0x40 code Origin(1): IGP
Aug 11 13:36:13.866820 BGP RECV flags 0x40 code ASPath(2) length 0:
Aug 11 13:36:13.866823 BGP RECV flags 0x40 code NextHop(3): 192.168.5.4
Aug 11 13:36:13.866826 BGP RECV flags 0x40 code LocalPref(5): 100
Aug 11 13:36:13.866829 BGP RECV flags 0x80 code Originator_Id(9) 192.168.5.4
Aug 11 13:36:13.866832 BGP RECV flags 0x80 code Cluster_List(10): 5.5.5.5 2.2.2.2 1.1.1.1
Aug 11 13:36:13.866836 BGP RECV 10.123.4.0/24
Aug 11 13:36:13.866843 bgp_read_v4_update: peer 192.168.5.5 (Internal AS 65000) UPDATE with local cluster-id (0.0.1.1) in clusterlist
Aug 11 13:36:13.866850 bgp_rcv_nlri: 10.123.4.0/24
Aug 11 13:36:13.866856 bgp_read_v4_message: done with 192.168.5.5 (Internal AS 65000) received 70 octets 1 update 1 route
Aug 11 13:36:13.867383
Below are the best BGP routes seen on P1, P2, P5 for 10.123.4/24
p1@vr-device:p1> show route protocol bgp 10.123.4.0/24 detail
inet.0: 25 destinations, 27 routes (25 active, 0 holddown, 0 hidden)
10.123.4.0/24 (1 entry, 1 announced)
*BGP Preference: 170/-101
Next hop type: Indirect
Address: 0x8f93b98
Next-hop reference count: 3
Source: 192.168.5.4
Next hop type: Router, Next hop index: 878
Next hop: 172.22.202.2 via em3.14, selected
Protocol next hop: 192.168.5.4
Indirect next hop: 900b000 131079
State:
Local AS: 65000 Peer AS: 65000
Age: 5:54 Metric2: 1
Task: BGP_65000.192.168.5.4+179
Announcement bits (3): 0-KRT 4-BGP RT Background 5-Resolve tree 1
AS path: I
Accepted
Localpref: 100
Router ID: 192.168.5.4
p2@vr-device:p2> show route protocol bgp 10.123.4.0/24 detail
inet.0: 25 destinations, 28 routes (25 active, 0 holddown, 0 hidden)
10.123.4.0/24 (2 entries, 1 announced)
*BGP Preference: 170/-101
Next hop type: Indirect
Address: 0x8f93838
Next-hop reference count: 4
Source: 192.168.5.1
Next hop type: Router, Next hop index: 131084
Next hop: 172.22.205.2 via em3.25, selected
Next hop: 172.22.201.1 via em4.12
Protocol next hop: 192.168.5.4
Indirect next hop: 8faea50 131080
State:
Local AS: 65000 Peer AS: 65000
Age: 6:50 Metric2: 2
Task: BGP_65000.192.168.5.1+49556
Announcement bits (3): 0-KRT 4-BGP RT Background 5-Resolve tree 1
AS path: I (Originator) Cluster list: 1.1.1.1
AS path: Originator ID: 192.168.5.4
Accepted
Localpref: 100
Router ID: 192.168.5.1
BGP Preference: 170/-101
Next hop type: Indirect
Address: 0x8f93838
Next-hop reference count: 4
Source: 192.168.5.5
Next hop type: Router, Next hop index: 131084
Next hop: 172.22.205.2 via em3.25, selected
Next hop: 172.22.201.1 via em4.12
Protocol next hop: 192.168.5.4
Indirect next hop: 8faea50 131080
State:
Inactive reason: Not Best in its group - Cluster list length
Local AS: 65000 Peer AS: 65000
Age: 6:50 Metric2: 2
Task: BGP_65000.192.168.5.5+179
AS path: I (Originator) Cluster list: 5.5.5.5 1.1.1.1
AS path: Originator ID: 192.168.5.4
Accepted
Localpref: 100
Router ID: 192.168.5.5
p5@vr-device:p5> show route protocol bgp 10.123.4.0/24 detail
inet.0: 25 destinations, 28 routes (25 active, 0 holddown, 0 hidden)
10.123.4.0/24 (2 entries, 1 announced)
*BGP Preference: 170/-101
Next hop type: Indirect
Address: 0x8f937a8
Next-hop reference count: 4
Source: 192.168.5.1
Next hop type: Router, Next hop index: 875
Next hop: 172.22.203.1 via em4.45, selected
Protocol next hop: 192.168.5.4
Indirect next hop: 8fafc30 131081
State:
Local AS: 65000 Peer AS: 65000
Age: 7:07 Metric2: 1
Task: BGP_65000.192.168.5.1+62193
Announcement bits (3): 0-KRT 4-BGP RT Background 5-Resolve tree 1
AS path: I (Originator) Cluster list: 1.1.1.1
AS path: Originator ID: 192.168.5.4
Accepted
Localpref: 100
Router ID: 192.168.5.1
BGP Preference: 170/-101
Next hop type: Indirect
Address: 0x8f937a8
Next-hop reference count: 4
Source: 192.168.5.2
Next hop type: Router, Next hop index: 875
Next hop: 172.22.203.1 via em4.45, selected
Protocol next hop: 192.168.5.4
Indirect next hop: 8fafc30 131081
State:
Inactive reason: Not Best in its group - Cluster list length
Local AS: 65000 Peer AS: 65000
Age: 7:07 Metric2: 1
Task: BGP_65000.192.168.5.2+64180
AS path: I (Originator) Cluster list: 2.2.2.2 1.1.1.1
AS path: Originator ID: 192.168.5.4
Accepted
Localpref: 100
Router ID: 192.168.5.2
no-client-reflect
How we can stop routes learnt from a cluster client reflected to RR clients?
This unlikely scenario can be achieved with “no-client-reflect” command. Please note, however, that routes learnt from clients continue to be reflected to other normal iBGP peers and vice versa.
group Cluster-2222 {
type internal;
local-address 192.168.5.2;
export static-to-bgp;
cluster 2.2.2.2;
no-client-reflect;
neighbor 192.168.5.1;
neighbor 192.168.5.3;
neighbor 192.168.5.4;
neighbor 192.168.5.6;
}
The result is that P1 no longer receives route update for 10.123.3, 4, 5 behind other clients, as they are no longer reflected by P2, nor P5. However RRs continue to reflect routes between their clients and their non-client iBGP peers, shown in the output below.
p1@vr-device:p1> show route protocol bgp
inet.0: 24 destinations, 26 routes (24 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.123.2.0/24 *[BGP/170] 01:21:48, localpref 100, from 192.168.5.2
AS path: I
> to 172.22.201.2 via em3.12
[BGP/170] 00:00:10, localpref 100, from 192.168.5.5
AS path: I
> to 172.22.201.2 via em3.12
10.123.5.0/24 *[BGP/170] 00:30:54, localpref 100, from 192.168.5.5
AS path: I
> to 172.22.201.2 via em3.12
to 172.22.202.2 via em3.14
[BGP/170] 00:00:10, localpref 100, from 192.168.5.2
AS path: I
> to 172.22.201.2 via em3.12
to 172.22.202.2 via em3.14
p2@vr-device:p2> show route protocol bgp 10.123.1.0
inet.0: 27 destinations, 31 routes (27 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.123.1.0/24 *[BGP/170] 01:27:06, localpref 100, from 192.168.5.1
AS path: I
> to 172.22.201.1 via em4.12
[BGP/170] 00:05:28, localpref 100, from 192.168.5.5
AS path: I
> to 172.22.201.1 via em4.12
Packet Corner 