Impact of Load Balancing and Multiple Paths on Traceroutes

We have written a series of articles about traceroutes, the most popular tool that network engineers use to troubleshoot network performance.

In the previous articles, we described how a traceroute works and how to analyze it. In this article and in the following one, we will cover two advanced but very important topics about traceroutes and modern networks.

In order to increase capacity and resiliency between routers, it is common to have more than one connection between them. If at any point a router does not support higher speed interfaces, the only solution to support a higher capacity would be to aggregate two or more ports together.

There are usually two possible configurations that allow you to set up multiple connections between routers: the Link Aggregation and the Equal Cost Multi Path (ECMP).

The first configuration is Link Aggregation at the Ethernet layer. It is also often referred to as Port Channel by Cisco users, but terms like Bonding, Bundling, Teaming and Trunking are also used by equipment vendors. Whatever the number of interfaces (ports) in the LAG (Link Aggregation Group), there will be a single IP address associated with the LAG.

The second configuration is named Equal Cost Multi Path (ECMP). Compared to Link Aggregation, it operates at the IP Layer. There is one IP address for each interface and it’s the router’s routing engine that will load balance the traffic between the different paths.

Before we go back to traceroutes, we need to understand how the load balancing or load sharing is done by the routers when there are multiple interfaces (i.e. links or ports). There are two algorithms that can be used by routers: per-packet and per-flow. This applies to both Link Aggregation and ECMP.

Per-packet load balancing selects a different interface for each packet. This is an excellent method to achieve equal load balancing on the interface and get the best bandwidth utilization. However, it can lead to packet reordering, which is terrible for real-time traffic such as VoIP. It is never recommended to use per-packet load balancing unless you really know what you are doing.

Per-flow load balancing uses a hashing algorithm on the packet headers to select which interface to use. The goal is to always have all the packets from a single flow going through the same interface. Generally, two packets are considered in the same flow if they match the same five attributes:

• Source IP Address
• Destination IP Address
• IP Protocol (ex: ICMP, TCP, UDP)
• Source Port (if TCP or UDP)
• Destination Port (if TCP or UDP)

The 5 attributes can be changed on a per-router basis by the network engineers but generally, these are the 5 attributes that are used on the Internet. With this technique, a TCP session (ex: HTTP request) between a source and a destination (ex: desktop and web server) will always use the same path. The same goes for a VoIP call using UDP packets.

Let’s get back to traceroutes. So we learned that there are two technologies to increase the number of links between routers: Link Aggregation & ECMP.

With Link Aggregation, there is a single IP address configured on the whole LAG. With this configuration, it would be impossible for traceroutes to know if there are multiple links, unless the DNS hostname of the IP is clear about it. For example, ISP A can choose to use the hostname lag1.router1.ispA.com.

With ECMP, since there are multiple IPs and since the router source IP for the ICMP TTL Exceeded is the IP of the interface that received the packet, you will be able to know exactly which interface received the packet.

Figure A shows a traceroute from a business Internet connection from Fibrenoire to the obkio.com website hosted inside AWS. Take a look at hop #3. We see two hostnames because there are at least two paths between the router at hop #2 and the router at hop #3.

At hop #4, it's even more interesting where we see that the two hostnames are not from the same router. Since the two paths inside the network are equal and the router is using ECMP, it will route the packets on either path based on the packet header information and the hashing algorithm.

+----+---------------------------------------+-------+------+------+------+------+-------+
| #  | Hostname                              | Loss% |  Snt | Last |  Avg | Best |  Wrst |
+----+---------------------------------------+-------+------+------+------+------+-------+
| 1  | 192.168.1.1                           |  72.8 | 2220 |  3.2 |  2.0 |  0.6 |  34.4 |
| 2  | CUST01-C1.asr02.mtl1080.fibrenoire.ca |   0.0 | 2220 |  2.0 |  3.6 |  1.6 |  51.0 |
| 3  | et-0-0-2.mpr01.mtlcx03.fibrenoire.ca  |   0.0 | 2220 |  1.4 |  4.5 |  1.0 |  66.7 |
|    | et-0-0-3.mpr01.mtlcx03.fibrenoire.ca  |       |      |      |      |      |       |
| 4  | et-0-0-4.cre01.mtl1981.fibrenoire.ca  |   0.0 | 2220 |  1.2 |  5.6 |  1.1 |  84.9 |
|    | et-0-0-4.cre01.mtlsunl.fibrenoire.ca  |       |      |      |      |      |       |
| 5  | et-0-0-2.mpr02.mtlcx03.fibrenoire.ca  |   0.0 | 2220 |  1.4 |  5.6 |  1.2 | 135.0 |
|    | et-0-0-3.mpr02.mtlcx03.fibrenoire.ca  |       |      |      |      |      |       |
| 6  | 52.95.219.62                          |   0.0 | 2220 |  3.3 |  4.3 |  1.1 |  78.2 |
| 7  | 52.94.82.32                           |   0.0 | 2220 |  2.7 |  4.4 |  1.8 |  67.4 |
|    | 52.94.81.134                          |       |      |      |      |      |       |
| 8  | 52.94.81.193                          |   0.0 | 2220 |  7.8 |  3.3 |  1.4 |  56.3 |
|    | 52.94.81.191                          |       |      |      |      |      |       |
| 9  | 52.94.82.69                           |   0.0 | 2220 | 28.2 | 26.4 | 22.4 |  60.5 |
|    | 54.239.44.18                          |       |      |      |      |      |       |
| 10 | 150.222.242.150                       |  48.6 | 2220 | 28.9 | 26.4 | 22.3 |  50.5 |
|    | 150.222.242.118                       |       |      |      |      |      |       |
| 11 | 150.222.242.150                       |  89.1 | 2220 | 28.9 | 26.1 | 22.4 |  37.7 |
|    | 150.222.242.152                       |       |      |      |      |      |       |
| 12 | ???                                   | 100.0 | 2220 |    - |    - |    - |     - |
| 13 | ???                                   | 100.0 | 2220 |    - |    - |    - |     - |
| 14 | ???                                   | 100.0 | 2220 |    - |    - |    - |     - |
| 15 | 150.222.243.215                       |  23.6 | 2220 | 24.0 | 26.1 | 22.3 |  54.7 |
|    | 150.222.241.183                       |       |      |      |      |      |       |
| 16 | 150.222.241.193                       |  74.9 | 2220 | 25.5 | 25.8 | 22.3 |  41.9 |
|    | 150.222.243.223                       |       |      |      |      |      |       |
| 17 | ???                                   | 100.0 | 2220 |    - |    - |    - |     - |
| 18 | ???                                   | 100.0 | 2220 |    - |    - |    - |     - |
| 19 | ???                                   | 100.0 | 2220 |    - |    - |    - |     - |
| 20 | ???                                   | 100.0 | 2220 |    - |    - |    - |     - |
| 21 | ???                                   | 100.0 | 2220 |    - |    - |    - |     - |
| 22 | ???                                   | 100.0 | 2220 |    - |    - |    - |     - |
| 23 | ???                                   | 100.0 | 2220 |    - |    - |    - |     - |
| 24 | ???                                   | 100.0 | 2208 |    - |    - |    - |     - |
| 25 | ???                                   | 100.0 | 2146 |    - |    - |    - |     - |
| 26 | 52.93.28.204                          |  23.3 | 1977 | 22.7 | 26.3 | 22.3 |  76.3 |
|    | 52.93.28.202                          |       |      |      |      |      |       |
| 27 | 52.93.28.204                          |  73.7 | 1274 | 26.9 | 26.2 | 22.4 |  48.7 |
|    | 52.93.28.202                          |       |      |      |      |      |       |
| 28 | ???                                   | 100.0 | 1226 |    - |    - |    - |     - |
+----+---------------------------------------+-------+------+------+------+------+-------+
Figure A - Traceroute to obkio.com

As explained earlier, the load balancing is done using a hashing algorithm on the packet headers. Depending on the software used to do the traceroute, the packet sent could be ICMP, UDP or TCP SYN packets.

On Unix, the default traceroute command uses UDP with random destination ports between 33434 to 33534. Since the port is random, the packet header will not always be the same, so the load balancing algorithms will use multiple paths.

On Windows, the tracert command uses ICMP. This way, the traceroute will display a single path, making it easier to understand.

With the Obkio Live Traceroute feature, it's possible to select the protocol and the ports to use. Therefore, you can use ICMP to have an easy to read traceroute or use TCP (or UDP) with random ports to see the full paths between the source and the destination. You can also use fixed source and destination ports in the TCP and UDP traceroutes.

Obkio Vision: Visual Traceroute Tool

Leverage Obkio Vision to monitor, detect and troubleshoot network problems with visual traceroutes, IP route historic and graphical network maps.

Try for Free

This is the end of this our article on load balancing with traceroutes. The next articles will cover how to analyze traceroutes and which information is the most important.

We hope you enjoyed this article in the traceroute series.

• What is a Traceroute and How Do Traceroutes Work?
• How To Identify Network Issues with Traceroutes?
• Why Do Some Routers Drop Packets or Have High Latencies?
• Decode the Hidden Information from Traceroute DNS
• Internet Traffic is Asymmetrical - How to Catch Reverse Path Issues?
• How to Share a Traceroute With an ISP NOC?
• Impact of Load Balancing or Multiple Paths on Traceroutes (this article)
• MPLS Networks, TTL Propagation and ICMP Tunneling