Blog

BGP communities part 3: Customer BGP Traffic Engineering communities

BGP communities part 3: Customer BGP Traffic Engineering communities

by | Jun 12, 2022


If you’ve ever been asked to prioritize one internet connection over another for any variety of reasons, cost, latency, SLA, etc… this is for you.

Often I hear the same tactics to solve this problem:

  • AS-PATH prepending
  • conditional advertisements
  • scripting
  • some other manual process

However, most carriers offer customer BGP TE communities that you can use to influence traffic within their AS, with one notable exception Hurricane Electric. If you’re not sure what a BGP community is take a quick look at this post on them first.

Lets explore how to utilize these, where to find them, and how they might give more deterministic path selection than the options laid out above.

BGP Topology

This image has an empty alt attribute; its file name is BGP-TE-communities-BGP-Topology-1-1024x667.jpeg

Default behavior with no modification

First to get familiar with the topology and show reachability we’ll leave all settings as “defaults” with no modifications.

ISP-1-RTR-1#traceroute 203.0.113.1 source 192.0.2.1
Type escape sequence to abort.
Tracing the route to 203.0.113.1
VRF info: (vrf in name/id, vrf out name/id)
  1 100.123.0.1 1 msec 1 msec 1 msec
  2 100.124.0.2 1 msec 0 msec 0 msec
  3 100.126.0.10 2 msec *  1 msec
ISP-1-RTR-1#ping 203.0.113.1 source 192.0.2.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 203.0.113.1, timeout is 2 seconds:
Packet sent with a source address of 192.0.2.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/3 ms

ISP-4-RTR-1#traceroute 203.0.113.1 source 198.51.100.1
Type escape sequence to abort.
Tracing the route to 203.0.113.1
VRF info: (vrf in name/id, vrf out name/id)
  1 100.120.0.1 1 msec 0 msec 1 msec
  2 100.124.0.2 2 msec 1 msec 1 msec
  3 100.126.0.10 2 msec *  1 msec

ISP-4-RTR-1#ping 203.0.113.1 source 198.51.100.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 203.0.113.1, timeout is 2 seconds:
Packet sent with a source address of 198.51.100.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms
CUSTOMER-1-RTR-1#traceroute 192.0.2.1 source 203.0.113.1
Type escape sequence to abort.
Tracing the route to 192.0.2.1
VRF info: (vrf in name/id, vrf out name/id)
  1 100.126.0.1 1 msec 1 msec 1 msec
  2 100.125.0.1 1 msec 1 msec 0 msec
  3 100.122.0.2 1 msec *  1 msec
CUSTOMER-1-RTR-1#ping 192.0.2.1 source 203.0.113.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.0.2.1, timeout is 2 seconds:
Packet sent with a source address of 203.0.113.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms

CUSTOMER-1-RTR-1#traceroute 198.51.100.1 source 203.0.113.1
Type escape sequence to abort.
Tracing the route to 198.51.100.1
VRF info: (vrf in name/id, vrf out name/id)
  1 100.126.0.9 1 msec 1 msec 1 msec
  2 100.124.0.1 1 msec 2 msec 3 msec
  3 100.120.0.2 2 msec *  2 msec
CUSTOMER-1-RTR-1#ping 198.51.100.1 source 203.0.113.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 198.51.100.1, timeout is 2 seconds:
Packet sent with a source address of 203.0.113.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms

We’re setting the source as only the public prefixes are advertised into BGP. The private CG-NAT prefixes seen in the traceroute are the transit links responding along the path.

You’ll also notice that the return path, the upload direction, utilizes a different path to 192.0.2.1. We’ll come back to this further down.

Path with AS-PATH prepending

This image has an empty alt attribute; its file name is BGP-TE-communities-Path-with-prepend-1-1024x667.jpeg

Lets look at what almost always comes as the first recommendation: AS-PATH prepending. In our use case we’ll take this approach and prepend 5 times on CUSTOMER-1-RTR-3.

CUSTOMER-1-RTR-3#show run | sec route-map
 neighbor 100.124.0.1 route-map PREPEND out
route-map PREPEND permit 10
 set as-path prepend 65000 65000 65000 65000 65000

This results in ISP-3-RTR-1 receiving the prefix with 65000 in the AS-PATH 6 times. As all of the other route attributes are default the BGP best path algorithm makes it to comparing AS-PATH where shorter is better. Well be using cisco’s best path algorithm as the reference:

  1. highest weight
  2. highest local-preference
  3. locally originated
  4. shortest AS-PATH
  5. prefer path with lowest origin type
  6. prefer path with lowest MED
  7. prefer eBGP over iBGP
  8. prefer path with lowest IGP metric to the next-hop
  9. determine if multipath needs installation
  10. oldest route
  11. lowest router-id
  12. minimum cluster list length
  13. prefer lowest neighbor address

This means that the path via ISP-2-RTR-1 is now better for 203.0.113.0/24 as you can see in the output below.

ISP-3-RTR-1#show ip bgp
BGP table version is 8, local router ID is 100.124.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path

<<clipped>>

 *   203.0.113.0      100.123.0.2                            0 65100 65200 65000 i
 *>                   100.121.0.1                            0 65200 65000 i
 *                    100.124.0.2                            0 65000 65000 65000 65000 65000 65000 i

Now running a traceroute from ISP-4 it appears as everything has been achieved.

ISP-4-RTR-1#traceroute 203.0.113.1 source 198.51.100.1
Type escape sequence to abort.
Tracing the route to 203.0.113.1
VRF info: (vrf in name/id, vrf out name/id)
  1 100.120.0.1 1 msec 1 msec 1 msec
  2 100.121.0.1 1 msec 1 msec 1 msec
  3 100.125.0.2 1 msec 1 msec 2 msec
  4 100.126.0.2 1 msec *  2 msec
ISP-4-RTR-1#

However, our outbound traffic hasn’t changed.

CUSTOMER-1-RTR-1#traceroute 198.51.100.1 source 203.0.113.1
Type escape sequence to abort.
Tracing the route to 198.51.100.1
VRF info: (vrf in name/id, vrf out name/id)
  1 100.126.0.9 2 msec 0 msec 1 msec
  2 100.124.0.1 1 msec 1 msec 1 msec
  3 100.120.0.2 2 msec *  1 msec

Most times I see people modify the metric to the next hop to get this behavior to change. Take notice that this is pretty far down the best path selection process. So lets raise the cost on the link from CUSTOMER-1-RTR-1 to CUSTOMER-1-RTR-3.

CUSTOMER-1-RTR-1#traceroute 198.51.100.1 source 203.0.113.1
Type escape sequence to abort.
Tracing the route to 198.51.100.1
VRF info: (vrf in name/id, vrf out name/id)
  1 100.126.0.1 1 msec 0 msec 0 msec
  2 100.126.0.18 1 msec 0 msec 0 msec
  3 100.124.0.1 1 msec 1 msec 2 msec
  4 100.120.0.2 2 msec *  2 msec

CUSTOMER-1-RTR-1#show run int g0/1
Building configuration...

Current configuration : 155 bytes
!
interface GigabitEthernet0/1
 ip address 100.126.0.10 255.255.255.248
 ip ospf 1 area 0
 ip ospf cost 100

Perfect now we ingress and egress the same router.


This image has an empty alt attribute; its file name is IPA-Blog-ad-template-network.jpg
iparchitechs.com/contact

Provider changes local-preference for customers

This image has an empty alt attribute; its file name is BGP-TE-communities-Path-with-prepend-and-LP-1-1024x667.jpeg

Something a lot of providers due to raise the local-preference on routes received from customers. This makes these routes preferred in their AS over the paths received from transit and peers. As you saw local-preference is higher in the BGP best path selection process.

ISP-3 is one of those providers. They set LP to be 120 on their routes received from customers and leave it at a default of 100 for peers (ISP-2 and ISP-1). What happens now?

ISP-3-RTR-1#show ip bgp
BGP table version is 10, local router ID is 100.124.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path

<<clipped>>

 *   203.0.113.0      100.123.0.2                            0 65100 65200 65000 i
 *                    100.121.0.1                            0 65200 65000 i
 *>                   100.124.0.2                   120      0 65000 65000 65000 65000 65000 65000 i

The best path is now the path with all of our AS-PATH prepends! This is because LP is further up in the BGP best path selection so the router doesn’t need AS-PATH length to determine the best path. The LP was 120 on one path and 100, default, on the other so it selected higher as better. Now, we’re back to where we started with the question of how do we influence return traffic to our AS?

ISP-4-RTR-1#traceroute 203.0.113.1 source 198.51.100.1
Type escape sequence to abort.
Tracing the route to 203.0.113.1
VRF info: (vrf in name/id, vrf out name/id)
  1 100.120.0.1 1 msec 0 msec 0 msec
  2 100.124.0.2 1 msec 1 msec 1 msec
  3 100.126.0.10 2 msec *  1 msec

customer BGP TE communities

This image has an empty alt attribute; its file name is BGP-TE-communities-Path-with-customer-community-1-1024x667.jpeg

Typically, the providers that do something similar to above offer their customers TE communities. You can send them a community to influence how they treat your traffic.

You may have to ask them for these values or it might be published publicly. A large listing can be found here, but verify before usage it is not an inclusive list of all vendors and I can’t speak to how up to date it is.

ISP-3 supports these and if you send 65300:80 they’ll set the local-preference on the routes received with this community to 80.

ISP-3-RTR-1#show run

<<clipped>>

router bgp 65300
 bgp log-neighbor-changes
 network 100.127.3.1 mask 255.255.255.255
 neighbor 100.120.0.2 remote-as 65400
 neighbor 100.121.0.1 remote-as 65200
 neighbor 100.123.0.2 remote-as 65100
 neighbor 100.124.0.2 remote-as 65000
 neighbor 100.124.0.2 route-map FROM-CUSTOMER in
!
<<clipped>>

ip bgp-community new-format
ip community-list standard SET-LP-80 permit 65300:80
!
route-map FROM-CUSTOMER permit 10
 match community SET-LP-80
 set local-preference 80
!
CUSTOMER-1-RTR-3#show run

<<clipped>>

router bgp 65000
 bgp router-id 100.127.0.2
 bgp log-neighbor-changes
 neighbor 100.124.0.1 remote-as 65300
 neighbor 100.124.0.1 send-community
 neighbor 100.124.0.1 route-map TO-INET out
 neighbor 100.127.0.0 remote-as 65000
 neighbor 100.127.0.0 update-source Loopback0
 neighbor 100.127.0.0 next-hop-self
 neighbor 100.127.0.1 remote-as 65000
 neighbor 100.127.0.1 update-source Loopback0
 neighbor 100.127.0.1 next-hop-self
!

<<clipped>>

route-map TO-INET permit 10
 set community 65300:80
!

The result is now that ISP-3 offloads all traffic destined to the customer through ISP-2 or ISP-1 because the local-preference for the same route received from these peers is higher.

ISP-3-RTR-1#show ip bgp
BGP table version is 12, local router ID is 100.124.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *   100.127.2.1/32   100.124.0.2                    80      0 65000 65200 i
 *                    100.123.0.2                            0 65100 65200 i
 *>                   100.121.0.1              0             0 65200 i
 *>  100.127.3.1/32   0.0.0.0                  0         32768 i
 *   192.0.2.0        100.121.0.1                            0 65200 65100 i
 *>                   100.123.0.2              0             0 65100 i
 *>  198.51.100.0     100.120.0.2              0             0 65400 i
 *   203.0.113.0      100.124.0.2                    80      0 65000 i
 *                    100.123.0.2                            0 65100 65200 65000 i
 *>                   100.121.0.1                            0 65200 65000 i
ISP-4-RTR-1#traceroute 203.0.113.1 source 198.51.100.1
Type escape sequence to abort.
Tracing the route to 203.0.113.1
VRF info: (vrf in name/id, vrf out name/id)
  1 100.120.0.1 0 msec 1 msec 1 msec
  2 100.121.0.1 1 msec 1 msec 1 msec
  3 100.125.0.2 1 msec 1 msec 1 msec
  4 100.126.0.2 1 msec *  2 msec

Everything is back to how we expect.

However, to avoid having to make one off changes in the IGP metrics lets utilize local-preference on received routes as well. We’ll set the metric back to default on the IGP and move up the BGP best path selection algorithm by using LP. We will raise the LP to 120 on the routes from ISP-2.

CUSTOMER-1-RTR-2#show run | sec route-map
 neighbor 100.125.0.1 route-map FROM-INET in
route-map FROM-INET permit 10
 set local-preference 120
CUSTOMER-1-RTR-1#show ip bgp
BGP table version is 16, local router ID is 100.127.0.0
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i 100.127.2.1/32   100.127.0.1              0    120      0 65200 i
 *>i 100.127.3.1/32   100.127.0.1              0    120      0 65200 65300 i
 *>i 192.0.2.0        100.127.0.1              0    120      0 65200 65100 i
 *>i 198.51.100.0     100.127.0.1              0    120      0 65200 65300 65400 i
 *>  203.0.113.0      0.0.0.0                  0         32768 i

Now the best path is always in and out CUSTOMER-1-RTR-2, as desired, as long as the peering to ISP-2 is up.



If you’re trying to influence traffic or need help implementing a customer BGP TE community scheme reach out to us at iparchitechs.

Recent Comments

No comments to show.