Internet Traffic is Asymmetrical: How to Catch Reverse Path Issues

Jean-François Lévesque
Jean-François Lévesque Last updated on Sep. 1, 2020

Internet Traffic is Asymmetrical - How to Catch Reverse Path Issues?

We have written a series of articles about traceroutes, the most popular tool that network engineers use to troubleshoot network performance.

Internet Traffic is Asymmetrical - How to Catch Reverse Path Issues?
Internet Traffic is Asymmetrical - How to Catch Reverse Path Issues?

When looking at a traceroute, people often forget that traffic on the Internet is asymmetrical most of the time. It is called the Hot Potato Routing. As soon as an ISP has a packet with a destination address outside its own network, it will try to pass the packet to the next ISP ASAP.

Traceroute Two ISPs - Internet Traffic is Asymmetrical

Figure A - Normal Traffic Flow

Figure A above is a good example of the Hot Potato Routing. In the figure, there are 2 ISPs (A and B) and they both have 3 routers located in New York City (NYC), Dallas (DAL) and San Francisco (SFO). In the 3 cities, the ISPs have interconnections to exchange traffic from one network to another.

When the source SRC sends a packet to the destination DST, ISP A receives the packet on router A-NYC. As soon as it receives the packet, it will search for a way to send it to ISP B. In this case, the interconnection between the two routers in NYC is the fastest path to reach ISP B. The packet then continues its way inside ISP B up to the destination DST. The traceroute from SRC to DST will look like this:

+---+----------+-------+-----+------+------+------+------+
| # | Hostname | Loss% | Snt | Last |  Avg | Best | Wrst |
+---+----------+-------+-----+------+------+------+------+
| 1 | A-NYC    |   0.0 |   1 |  1.0 |  1.0 |  1.0 |  1.0 |
| 2 | B-NYC    |   0.0 |   1 |  2.0 |  2.0 |  2.0 |  2.0 |
| 3 | B-DAL    |   0.0 |   1 | 40.0 | 40.0 | 40.0 | 40.0 |
| 4 | B-SFO    |   0.0 |   1 | 80.0 | 80.0 | 80.0 | 80.0 |
| 5 | DST      |   0.0 |   1 | 81.0 | 81.0 | 81.0 | 81.0 |
+---+----------+-------+-----+------+------+------+------+
Figure B - SRC to DST

On the other side, when DST replies back to SRC, the packet goes from ISP B to A in SFO because this is the fastest route between the two networks. The forward traffic from SRC to DST uses a different path than the reverse traffic from DST to SRC. The traceroute from DST to SRC will look like this:

+---+----------+-------+-----+------+------+------+------+
| # | Hostname | Loss% | Snt | Last |  Avg | Best | Wrst |
+---+----------+-------+-----+------+------+------+------+
| 1 | B-SFP    |   0.0 |   1 |  1.0 |  1.0 |  1.0 |  1.0 |
| 2 | A-SFO    |   0.0 |   1 |  2.0 |  2.0 |  2.0 |  2.0 |
| 3 | A-DAL    |   0.0 |   1 | 40.0 | 40.0 | 40.0 | 40.0 |
| 4 | A-NYC    |   0.0 |   1 | 80.0 | 80.0 | 80.0 | 80.0 |
| 5 | SRC      |   0.0 |   1 | 81.0 | 81.0 | 81.0 | 81.0 |
+---+----------+-------+-----+------+------+------+------+
Figure C - DST to SRC

Congestion in the Reverse Path
Congestion in the Reverse Path

Let’s take a look at the traceroute if there is congestion (50% packet loss) on the reverse path between A-DAL and A-SFO exactly at the red circle on Figure D below.

Traceroute ISP A has congestion - Internet Traffic is Asymmetrical

Figure D - Congestion in the reverse path

The forward path traceroute will look like Figure E.

+---+----------+-------+-----+------+------+------+------+
| # | Hostname | Loss% | Snt | Last |  Avg | Best | Wrst |
+---+----------+-------+-----+------+------+------+------+
| 1 | A-NYC    |   0.0 |  10 |  1.0 |  1.0 |  1.0 |  1.0 |
| 2 | B-NYC    |   0.0 |  10 |  2.0 |  2.0 |  2.0 |  2.0 |
| 3 | B-DAL    |   0.0 |  10 | 40.0 | 40.0 | 40.0 | 40.0 |
| 4 | B-SFO    |  50.0 |  10 | 80.0 | 80.0 | 80.0 | 80.0 |
| 5 | DST      |  50.0 |  10 | 81.0 | 81.0 | 81.0 | 81.0 |
+---+----------+-------+-----+------+------+------+------+
Figure E - SRC to DST during congestion

The traceroute clearly indicates a network issue because the packet loss continues below hop 4. Learn more on how to analyze a traceroute with the article How To Identify Network Issues with Traceroutes?.

If the forward and the reverse paths were the same, we could say that there is congestion inside ISP B between Dallas and San Francisco. But we know it’s not that. The congestion is in ISP A’s network. The reverse path traceroute will show us the other side of the medal:

+---+----------+-------+-----+------+------+------+------+
| # | Hostname | Loss% | Snt | Last |  Avg | Best | Wrst |
+---+----------+-------+-----+------+------+------+------+
| 1 | B-SFP    |   0.0 |  10 |  1.0 |  1.0 |  1.0 |  1.0 |
| 2 | A-SFO    |   0.0 |  10 |  2.0 |  2.0 |  2.0 |  2.0 |
| 3 | A-DAL    |  50.0 |  10 | 40.0 | 40.0 | 40.0 | 40.0 |
| 4 | A-NYC    |  50.0 |  10 | 80.0 | 80.0 | 80.0 | 80.0 |
| 5 | SRC      |  50.0 |  10 | 81.0 | 81.0 | 81.0 | 81.0 |
+---+----------+-------+-----+------+------+------+------+
Figure F - DST to SRC during congestion

So in that example, Figure F, the reverse traceroute from DST to SRC gave us the good answer about where the problem was, but unfortunately, there is no way for us to know which traceroute (forward or reverse) is exact. However, with that information in hand, the network engineer at ISP A and B can help troubleshoot the network issue that is affecting the traffic between SRC and DST.

To help troubleshoot the issue further, traceroutes from sources and destinations that are in the same ISP can help locate the exact issue.

Congestion at the Dallas Interconnection
Congestion at the Dallas Interconnection

Networks are complex. There are millions of connections on the Internet where a network issue can happen. Let’s see what happens if there is 50% packet loss on the Dallas interconnection between ISP A and B.

Congestion as the Dallas interconnection - Internet Traffic is Asymmetrical

Figure G - Congestion at the Dallas interconnection

The two traceroutes look like this:

+---+----------+-------+-----+------+------+------+------+
| # | Hostname | Loss% | Snt | Last |  Avg | Best | Wrst |
+---+----------+-------+-----+------+------+------+------+
| 1 | A-NYC    |   0.0 |  10 |  1.0 |  1.0 |  1.0 |  1.0 |
| 2 | B-NYC    |   0.0 |  10 |  2.0 |  2.0 |  2.0 |  2.0 |
| 3 | B-DAL    |  50.0 |  10 | 40.0 | 40.0 | 40.0 | 40.0 |
| 4 | B-SFO    |   0.0 |  10 | 80.0 | 80.0 | 80.0 | 80.0 |
| 5 | DST      |   0.0 |  10 | 81.0 | 81.0 | 81.0 | 81.0 |
+---+----------+-------+-----+------+------+------+------+
Figure H - SRC to DST during congestion in Dallas
+---+----------+-------+-----+------+------+------+------+
| # | Hostname | Loss% | Snt | Last |  Avg | Best | Wrst |
+---+----------+-------+-----+------+------+------+------+
| 1 | B-SFP    |   0.0 |  10 |  1.0 |  1.0 |  1.0 |  1.0 |
| 2 | A-SFO    |   0.0 |  10 |  2.0 |  2.0 |  2.0 |  2.0 |
| 3 | A-DAL    |  50.0 |  10 | 40.0 | 40.0 | 40.0 | 40.0 |
| 4 | A-NYC    |   0.0 |  10 | 80.0 | 80.0 | 80.0 | 80.0 |
| 5 | SRC      |   0.0 |  10 | 81.0 | 81.0 | 81.0 | 81.0 |
+---+----------+-------+-----+------+------+------+------+
Figure I - DST to SRC during congestion In Dallas

As we learned in the article How To Identify Network Issues with Traceroutes?, if the packet loss doesn’t continue, don’t panic, there is no network issue. Well, it's correct to assume that there is no network issue affecting the traffic from SRC to DST, but in that special case, all the ICMP TTL Exceeded responses from B-DAL to SRC are dropped at the interconnection issue because B-DAL is using the shortest path (i.e. the interconnection) to send back the packet. On the other side, the responses from A-DAL to DST are also dropped.

Now you understand why it’s important to have a reverse traceroute to compare the data. It’s also clear that a single traceroute can be misleading, so we must be careful when we think that we’ve pinpointed a network issue.

As one can imagine, a network performance monitoring solution like Obkio offers reverse traceroute to help troubleshoot network issues.

Next Traceroute Articles
Next Traceroute Articles

This is the end of this first article on traceroutes. The next articles will cover how to analyze traceroutes and which information is the most important.

We hope you enjoyed this article in the traceroute series.

Related Blog Categories:
Traceroutes