Internet Traffic is Asymmetrical: Catch Reverse Path Issues

We have written a series of articles about traceroutes, the most popular tool that network engineers use to troubleshoot network performance.

When looking at a traceroute, people often forget that traffic on the Internet is asymmetrical most of the time. It is called the Hot Potato Routing. As soon as an ISP has a packet with a destination address outside its own network, it will try to pass the packet to the next ISP ASAP.

Figure A above is a good example of the Hot Potato Routing. In the figure, there are 2 ISPs (A and B) and they both have 3 routers located in New York City (NYC), Dallas (DAL) and San Francisco (SFO). In the 3 cities, the ISPs have interconnections to exchange traffic from one network to another.

When the source SRC sends a packet to the destination DST, ISP A receives the packet on router A-NYC. As soon as it receives the packet, it will search for a way to send it to ISP B. In this case, the interconnection between the two routers in NYC is the fastest path to reach ISP B. The packet then continues its way inside ISP B up to the destination DST. The traceroute from SRC to DST will look like this:

+---+----------+-------+-----+------+------+------+------+
| # | Hostname | Loss% | Snt | Last |  Avg | Best | Wrst |
+---+----------+-------+-----+------+------+------+------+
| 1 | A-NYC    |   0.0 |   1 |  1.0 |  1.0 |  1.0 |  1.0 |
| 2 | B-NYC    |   0.0 |   1 |  2.0 |  2.0 |  2.0 |  2.0 |
| 3 | B-DAL    |   0.0 |   1 | 40.0 | 40.0 | 40.0 | 40.0 |
| 4 | B-SFO    |   0.0 |   1 | 80.0 | 80.0 | 80.0 | 80.0 |
| 5 | DST      |   0.0 |   1 | 81.0 | 81.0 | 81.0 | 81.0 |
+---+----------+-------+-----+------+------+------+------+
Figure B - SRC to DST

On the other side, when DST replies back to SRC, the packet goes from ISP B to A in SFO because this is the fastest route between the two networks. The forward traffic from SRC to DST uses a different path than the reverse traffic from DST to SRC. The traceroute from DST to SRC will look like this:

+---+----------+-------+-----+------+------+------+------+
| # | Hostname | Loss% | Snt | Last |  Avg | Best | Wrst |
+---+----------+-------+-----+------+------+------+------+
| 1 | B-SFP    |   0.0 |   1 |  1.0 |  1.0 |  1.0 |  1.0 |
| 2 | A-SFO    |   0.0 |   1 |  2.0 |  2.0 |  2.0 |  2.0 |
| 3 | A-DAL    |   0.0 |   1 | 40.0 | 40.0 | 40.0 | 40.0 |
| 4 | A-NYC    |   0.0 |   1 | 80.0 | 80.0 | 80.0 | 80.0 |
| 5 | SRC      |   0.0 |   1 | 81.0 | 81.0 | 81.0 | 81.0 |
+---+----------+-------+-----+------+------+------+------+
Figure C - DST to SRC

Let’s take a look at the traceroute if there is network congestion (50% packet loss) on the reverse path between A-DAL and A-SFO exactly at the red circle on Figure D below.

The forward path traceroute will look like Figure E.

+---+----------+-------+-----+------+------+------+------+
| # | Hostname | Loss% | Snt | Last |  Avg | Best | Wrst |
+---+----------+-------+-----+------+------+------+------+
| 1 | A-NYC    |   0.0 |  10 |  1.0 |  1.0 |  1.0 |  1.0 |
| 2 | B-NYC    |   0.0 |  10 |  2.0 |  2.0 |  2.0 |  2.0 |
| 3 | B-DAL    |   0.0 |  10 | 40.0 | 40.0 | 40.0 | 40.0 |
| 4 | B-SFO    |  50.0 |  10 | 80.0 | 80.0 | 80.0 | 80.0 |
| 5 | DST      |  50.0 |  10 | 81.0 | 81.0 | 81.0 | 81.0 |
+---+----------+-------+-----+------+------+------+------+
Figure E - SRC to DST during congestion

The traceroute clearly indicates a network issue because the packet loss continues below hop 4. Learn more on how to analyze a traceroute with the article How To Identify Network Issues with Traceroutes?.

If the forward and the reverse paths were the same, we could say that there is congestion inside ISP B between Dallas and San Francisco. But we know it’s not that. The congestion is in ISP A’s network. The reverse path traceroute will show us the other side of the medal:

+---+----------+-------+-----+------+------+------+------+
| # | Hostname | Loss% | Snt | Last |  Avg | Best | Wrst |
+---+----------+-------+-----+------+------+------+------+
| 1 | B-SFP    |   0.0 |  10 |  1.0 |  1.0 |  1.0 |  1.0 |
| 2 | A-SFO    |   0.0 |  10 |  2.0 |  2.0 |  2.0 |  2.0 |
| 3 | A-DAL    |  50.0 |  10 | 40.0 | 40.0 | 40.0 | 40.0 |
| 4 | A-NYC    |  50.0 |  10 | 80.0 | 80.0 | 80.0 | 80.0 |
| 5 | SRC      |  50.0 |  10 | 81.0 | 81.0 | 81.0 | 81.0 |
+---+----------+-------+-----+------+------+------+------+
Figure F - DST to SRC during congestion

So in that example, Figure F, the reverse traceroute from DST to SRC gave us the good answer about where the problem was, but unfortunately, there is no way for us to know which traceroute (forward or reverse) is exact. However, with that information in hand, the network engineer at ISP A and B can help troubleshoot the network issue that is affecting the traffic between SRC and DST.

To help troubleshoot the issue further, traceroutes from sources and destinations that are in the same ISP can help locate the exact issue.

The Complete Guide to Traceroutes

Download Obkio's free Complete Guide to Traceroutes to learn to identify network problems with the most popular network troubleshooting tool for IT Pros.

Download Now

Networks are complex. There are millions of connections on the Internet where a network issue can happen. Let’s see what happens if there is 50% packet loss on the Dallas interconnection between ISP A and B.

The two traceroutes look like this:

+---+----------+-------+-----+------+------+------+------+
| # | Hostname | Loss% | Snt | Last |  Avg | Best | Wrst |
+---+----------+-------+-----+------+------+------+------+
| 1 | A-NYC    |   0.0 |  10 |  1.0 |  1.0 |  1.0 |  1.0 |
| 2 | B-NYC    |   0.0 |  10 |  2.0 |  2.0 |  2.0 |  2.0 |
| 3 | B-DAL    |  50.0 |  10 | 40.0 | 40.0 | 40.0 | 40.0 |
| 4 | B-SFO    |   0.0 |  10 | 80.0 | 80.0 | 80.0 | 80.0 |
| 5 | DST      |   0.0 |  10 | 81.0 | 81.0 | 81.0 | 81.0 |
+---+----------+-------+-----+------+------+------+------+
Figure H - SRC to DST during congestion in Dallas

+---+----------+-------+-----+------+------+------+------+
| # | Hostname | Loss% | Snt | Last |  Avg | Best | Wrst |
+---+----------+-------+-----+------+------+------+------+
| 1 | B-SFP    |   0.0 |  10 |  1.0 |  1.0 |  1.0 |  1.0 |
| 2 | A-SFO    |   0.0 |  10 |  2.0 |  2.0 |  2.0 |  2.0 |
| 3 | A-DAL    |  50.0 |  10 | 40.0 | 40.0 | 40.0 | 40.0 |
| 4 | A-NYC    |   0.0 |  10 | 80.0 | 80.0 | 80.0 | 80.0 |
| 5 | SRC      |   0.0 |  10 | 81.0 | 81.0 | 81.0 | 81.0 |
+---+----------+-------+-----+------+------+------+------+
Figure I - DST to SRC during congestion In Dallas

As we learned in the article How To Identify Network Issues with Traceroutes?, if the packet loss doesn’t continue, don’t panic, there is no network issue. Well, it's correct to assume that there is no network issue affecting the traffic from SRC to DST, but in that special case, all the ICMP TTL Exceeded responses from B-DAL to SRC are dropped at the interconnection issue because B-DAL is using the shortest path (i.e. the interconnection) to send back the packet. On the other side, the responses from A-DAL to DST are also dropped.

Now you understand why it’s important to have a reverse traceroute to compare the data. It’s also clear that a single traceroute can be misleading, so we must be careful when we think that we’ve pinpointed a network issue.

As one can imagine, a network performance monitoring solution like Obkio offers reverse traceroute to help troubleshoot network issues.

Obkio Vision: Visual Traceroute Tool

Leverage Obkio Vision to monitor, detect and troubleshoot network problems with visual traceroutes, IP route historic and graphical network maps.

Try for Free

This is the end of this article on monitoring internet traffic with traceroutes. The next articles will cover how to analyze traceroutes and which information is the most important.

We hope you enjoyed this article in the traceroute series.

• What is a Traceroute and How Do Traceroutes Work?
• How To Identify Network Issues with Traceroutes?
• Why Do Some Routers Drop Packets or Have High Latencies?
• Decode the Hidden Information from Traceroute DNS
• Internet Traffic is Asymmetrical - How to Catch Reverse Path Issues? (this article)
• How to Share a Traceroute With an ISP NOC?
• Impact of Load Balancing or Multiple Paths on Traceroutes
• MPLS Networks, TTL Propagation and ICMP Tunneling