Traffic Engineering: Finding the Right Route

Understanding Multihoming and BGP Basics

When a network is multihomed, it attaches to two or more upstream providers. Every packet that leaves the network can choose between these links, and the choice is largely up to Border Gateway Protocol (BGP). In its default state, BGP prefers the shortest Autonomous System (AS) path. This means that if a packet can reach its destination through a route that traverses fewer AS numbers, BGP will pick that path, even if the underlying physical link is slower or more congested.

That default behavior works fine for many simple scenarios, but in a multihomed environment it can create imbalances. If one provider offers a 1 Gbps link and the other only a 100 Mbps one, BGP may still route a large portion of traffic over the shorter path, saturating the weaker link while leaving the stronger one underused. The problem is compounded when the network also hosts latency‑sensitive applications. A short AS path does not guarantee low round‑trip time; it only tells us that the path contains fewer intermediate routers.

To make BGP decisions that reflect your real network performance, you need to manipulate the attributes that BGP considers. The four attributes BGP examines are weight (Cisco only), local preference, AS path length, and Multi‑Exit Discriminator (MED). By adjusting these values, you can steer traffic toward a link that best meets the requirements of the applications that run behind your router.

Weight is the first attribute to check. It is local to a router and has the highest priority. Setting a higher weight on routes learned from a specific provider ensures that those routes win in the comparison phase. If you want traffic to favor ISP A over ISP B, you would set a higher weight on the routes from ISP A. The next attribute, local preference, is exchanged within an AS and is used to influence outbound traffic from your network. By assigning a higher local preference to routes from the faster or less congested provider, you direct the traffic accordingly. The AS path attribute is static and typically reflects the path length; you can prepend additional AS numbers to make a path appear longer, discouraging BGP from selecting it. MED is exchanged between neighbors and is a lower‑level signal that can indicate the preferred entry point to a destination when multiple paths exist.

Balancing the load is not just a matter of choosing the best link for a single destination. Real networks carry multiple flows, each with different bandwidth and latency requirements. The goal of traffic engineering is to allocate each flow to the route that offers the best combination of capacity, speed, and reliability. You can do this by assigning higher weights or local preferences to certain prefixes, by using MED to signal path quality to downstream providers, or by manipulating the AS path to influence how neighboring ASes route your traffic. The result is a more efficient use of the multihomed connection, fewer packet losses, and a smoother experience for end users.

In practice, configuring BGP for traffic engineering involves a careful trade‑off between the simplicity of default routing and the precision of custom attributes. The next section shows how to adjust those attributes to balance outgoing traffic, while the following sections explain how to discover which routes are actually performing well and how to enforce those decisions through route maps.

Balancing Outgoing Traffic with BGP Attributes

Once you understand the attributes BGP uses, the next step is to apply them in a way that distributes traffic across your providers. The simplest technique is to set a higher weight on the routes from the link you want to prioritize. Because weight is evaluated before local preference, it takes precedence over any other decisions. If you have a high‑bandwidth link to ISP B and a lower‑bandwidth link to ISP A, you can set a weight of 200 on routes learned from ISP B and a weight of 100 on those from ISP A. This ensures that, for every destination that receives updates from both providers, the router will favor ISP B.

Weight manipulation is quick and reversible, but it is local. If you need the preference to be seen by other routers inside your AS, you must use local preference. Setting local preference to a high value (e.g., 200) on routes from ISP B will cause all routers that receive the updates to treat those routes as better, thereby redirecting more traffic through the faster link. Conversely, if you want to offload traffic from a congested provider, you can lower the local preference on its routes to a minimal value, encouraging the network to route traffic elsewhere.

Sometimes the best solution involves a combination of weight and local preference. For instance, you may want all routers inside your AS to prefer ISP B, but only on specific destination ranges. In such cases, you can use route maps to match the prefixes and then set the local preference accordingly. The flexibility of BGP attributes allows you to craft policies that consider bandwidth, latency, or even cost, giving you granular control over traffic flow.

It is essential to monitor the effect of these changes. Use show ip bgp to inspect the attributes of each route and verify that the correct weight or local preference is applied. Check the output of show ip route bgp to confirm that the best path aligns with your expectations. Keep an eye on the link utilization on both providers; if one link starts to saturate, you can quickly adjust the weight or local preference to redistribute the load.

While outgoing traffic can be engineered with relative ease, incoming traffic remains a challenge. BGP only controls the path your network advertises to others; it cannot dictate how other networks send traffic to you. To achieve a balanced inbound flow, you often rely on external policies from your providers or on techniques such as reverse path filtering, which can affect how packets are accepted on each interface. The remainder of this guide focuses on how to assess route quality and how to enforce your policies using route maps.

Using Traceroute to Uncover Route Quality

Traceroute is a simple yet powerful tool for gauging the health of a network path. By incrementally increasing the Time‑To‑Live (TTL) value of each probe packet, traceroute forces intermediate routers to generate an ICMP “TTL exceeded” message, revealing the hop-by-hop sequence that packets traverse. When you run traceroute from your network to a destination, you receive a list of routers along with the round‑trip times for each probe. This information can expose congestion, latency, and packet loss.

When interpreting traceroute results, look for three main clues. First, consistent delays that rise sharply at a particular hop often indicate a congested link or a router that is heavily loaded. For example, a sequence of hop times that stay around 1 ms for the first ten hops but jump to 20 ms on hop 11 suggests that the router at hop 11 is the bottleneck.

Second, the presence of asterisks (*) in place of a round‑trip time signals that the router either dropped the probe or refused to reply. This could be due to firewall filtering, a deliberate rate limit, or a temporary failure. While occasional asterisks are normal, a high rate of missing replies points to a serious problem along the path.

Third, the names of the routers can offer geographic hints. Many service providers use meaningful hostnames, such as atl01 or nyc2, indicating a city or data‑center location. By mapping these names to real-world geography, you can estimate the physical distance the packets travel. Even if you don't have a precise mapping, a noticeable jump in delay often coincides with a long-haul fiber link or a cross‑continent detour.

Consider a traceroute to g.root-servers.net that shows a stable 1–2 ms delay for the first nine hops, then a sudden spike to 30 ms on hop 10. The following hop then drops back to 5 ms. This pattern indicates that the router at hop 10 is temporarily overloaded, causing packets to queue and experience high latency. If your traffic often routes through that hop, you might want to adjust BGP attributes to divert some of the load elsewhere.

It is important to remember that traceroute provides a snapshot, not a comprehensive measurement. The ICMP replies that trigger traceroute may follow a slightly different path than the actual data packets, especially in asymmetric networks. Therefore, use traceroute as a diagnostic aid rather than a definitive metric. Complement it with continuous monitoring tools like NetFlow or sFlow to get a fuller picture of traffic behavior over time.

Once you have identified a congested or suboptimal path, you can feed that information into BGP policy. By matching the destination prefixes associated with the problematic hop and assigning a lower weight or local preference, you steer traffic onto an alternative route. This feedback loop - measure, analyze, configure - forms the backbone of effective traffic engineering.

Applying Route Maps for Precise Traffic Engineering

Route maps are the primary mechanism for customizing BGP routing decisions on Cisco IOS devices. Think of a route map as a conditional statement: a match clause tests the route against a set of criteria, and if the test passes, a series of set clauses modifies the route’s attributes before it is entered into the BGP table or advertised to a neighbor. This flexibility lets you fine‑tune how traffic is routed based on prefix, community, or other attributes.

The first step in creating a route map is to decide whether you want to apply it to incoming or outgoing updates. Incoming route maps influence which routes the router accepts from a neighbor, allowing you to filter out undesirable paths. Outgoing route maps, on the other hand, modify routes before they are advertised to other neighbors, giving you control over the paths you publish.

For example, suppose you want to ensure that all routes to 10.0.0.0/8 received from ISP A are advertised with a high weight of 200, while the same prefixes from ISP B receive a lower weight of 100. You would create two route maps:

Apply the first map to the BGP process for incoming updates from ISP A:

And the second map to the updates from ISP B:

These configurations ensure that any route entering the BGP table from ISP A carries a weight of 200, making it the preferred path, while routes from ISP B carry a weight of 100. If the network also needs to signal preference to downstream providers, you can add a set local-preference clause instead of or in addition to the weight.

Route maps also support more complex matching conditions. For instance, you can match on the AS path to prepend your own AS number, effectively lengthening the path and making it less attractive:

Here, any route with a match on AS-PATH-ISPA will receive two extra AS numbers, making the path longer and less likely to be chosen by neighboring routers. Combining these techniques lets you implement a comprehensive traffic engineering strategy that considers bandwidth, latency, cost, and business agreements.

Once route maps are in place, monitor their effect with show route-map and show ip bgp to verify that the expected attributes are present. Adjust the sequence numbers if you need to reorder the evaluation order. Remember that the first route map with a matching condition is applied; any subsequent maps for the same route will be ignored unless the first map explicitly permits or denies the route.

By aligning route maps with the insights you gain from traceroute and BGP attribute manipulation, you create a resilient, high‑performance multihomed network that adapts to changing conditions and meets the demands of both latency‑sensitive and bandwidth‑hungry applications.