Traceroutes in MPLS Networks: TTL Propagation & ICMP Tunneling

We have written a series of articles about traceroutes, the most popular tool that network engineers use to troubleshoot network performance.

Service providers (SP) and large enterprises use MPLS (Multiprotocol Label Switching) networks to better segment and manage their networks. Initially designed to allow faster switching than IP networks, the main usage of MPLS networks is to offer multiple services within the same network such as MPLS VPNs or MPLS Ethernet Pseudowires.

This article is not about MPLS networks. Rather, we will cover two aspects of MPLS networks that affect traditional IP traceroutes. It applies to all IP traffic going through an MPLS Network, whether it is for a private network (i.e. MPLS VPN service) or standard Internet connectivity.

MPLS is a technology that encapsulates every packet with one or many MPLS Labels that are then switched inside the MPLS network. To keep it simple, let’s say that the MPLS Label contains the information about the destination of the packet. Inside an MPLS network, only the first and the last router of an LSP (Label Switched Path) have the routing table for a specific service.

Note: This is always true for MPLS VPN services (what SP calls MPLS private networks). For the Internet, it depends on the configuration of the routing table.

When an MPLS router in the middle of the LSP needs to send the ICMP TTL Exceeded packet back to the source, it doesn’t know where to send the packet. When this happens, the router that is sending the ICMP TTL Exceeded packet will add the same MPLS Label as the original packet and forward that packet to the MPLS Destination router (based on the MPLS Label information). The destination router will remove the MPLS Label and then forward the packet back to the source. This is called ICMP Tunneling.

In Figure A, routers R2, R3 and R4 are in the middle of the LSP and they don’t know how to reach the source. The only thing they know is that the destination for the MPLS packet is R5. When they need to send the ICMP TTL Exceeded packet, they add the MPLS label to reach R5 and then R5 routes the packet back to R1 (with another MPLS Label that tells the other routers to forward to R1).

In the two following examples, there is exactly 10ms between each router and there is 50% packet loss between R4 and R5.

This is the traceroute without the MPLS networks (i.e. a traditional IP network):

+---+----------+-------+-----+------+------+------+------+
| # | Hostname | Loss% | Snt | Last |  Avg | Best | Wrst |
+---+----------+-------+-----+------+------+------+------+
| 1 | R1       |   0.0 |  10 |  1.0 |  1.0 |  1.0 |  1.0 |
| 2 | R2       |   0.0 |  10 | 11.0 | 11.0 | 11.0 | 11.0 |
| 3 | R3       |   0.0 |  10 | 21.0 | 21.0 | 21.0 | 21.0 |
| 4 | R4       |   0.0 |  10 | 31.0 | 31.0 | 31.0 | 31.0 |
| 5 | R5       |  50.0 |  10 | 41.0 | 41.0 | 41.0 | 41.0 |
| 6 | DST      |  50.0 |  10 | 42.0 | 42.0 | 42.0 | 42.0 |
+---+----------+-------+-----+------+------+------+------+
Figure B - Traceroute without MPLS Network

However, when it’s going through an MPLS network with ICMP Tunneling, the traceroute will look like this:

+---+----------+-------+-----+------+------+------+------+
| # | Hostname | Loss% | Snt | Last |  Avg | Best | Wrst |
+---+----------+-------+-----+------+------+------+------+
| 1 | R1       |   0.0 |  10 |  1.0 |  1.0 |  1.0 |  1.0 |
| 2 | R2       |  50.0 |  10 | 41.0 | 41.0 | 41.0 | 41.0 |
| 3 | R3       |  50.0 |  10 | 41.0 | 41.0 | 41.0 | 41.0 |
| 4 | R4       |  50.0 |  10 | 41.0 | 41.0 | 41.0 | 41.0 |
| 5 | R5       |  50.0 |  10 | 41.0 | 41.0 | 41.0 | 41.0 |
| 6 | DST      |  50.0 |  10 | 42.0 | 42.0 | 42.0 | 42.0 |
+---+----------+-------+-----+------+------+------+------+
Figure C - Traceroute with MPLS Network

Both the latency and the packet loss are different even if the network path is the same. So if the latency spikes and then stays the same for hops that are far away from each other, keep in mind that ICMP Tunneling might be the cause. To have an approximate idea of where a router is located, take a look at the article Decode the hidden information from traceroute DNS.

Earlier we briefly discussed the MPLS label that is added to the packet when it comes in an MPLS network. In the label, similar to the Ethernet or IP packet header, there are multiple fields: destination label, traffic class (QoS) and time-to-live (TTL).

If you are using QoS on your network, you should take a look at the QoS Monitoring with Obkio DSCP Features blog post.

The TTL field inside the MPLS label is used exactly as the IP TTL field. Each time it reaches a router, it is decremented by one.

But the question is, what is the initial TTL value? There are two choices:

1. It is copied from the IP TTL field. This is called TTL propagation. This is exactly what happened in the previous section of this article.

2. A new value of 255 is used so TTL Propagation is disabled.

When TTL propagation is disabled, some routers are not visible in the traceroute. Let’s get back to the same previous example but with TTL propagation disabled.

If there is exactly 10ms between each router and there is 50% packet loss between R4 and R5, the traceroute without TTL propagation will look like this:

+---+----------+-------+-----+------+------+------+------+
| # | Hostname | Loss% | Snt | Last |  Avg | Best | Wrst |
+---+----------+-------+-----+------+------+------+------+
| 1 | R1       |   0.0 |  10 |  1.0 |  1.0 |  1.0 |  1.0 |
| 2 | R5       |  50.0 |  10 | 41.0 | 41.0 | 41.0 | 41.0 |
| 3 | DST      |  50.0 |  10 | 42.0 | 42.0 | 42.0 | 42.0 |
+---+----------+-------+-----+------+------+------+------+
Figure E - Traceroute without TTL Propagation

As you can see, MPLS networks change the way we look at traceroutes without giving us the exact picture on what is going on. That being said, what we learned from the article How To Identify Network Issues with Traceroutes? is still valid: if the packet loss doesn’t continue, it’s not a problem.

Obkio Vision: Visual Traceroute Tool

Leverage Obkio Vision to monitor, detect and troubleshoot network problems with visual traceroutes, IP route historic and graphical network maps.

Try for Free

This is the end of this final article on traceroutes. Easily refer back to all this information by downloading our free complete guide to traceroutes!

The Complete Guide to Traceroutes

Download Obkio's free Complete Guide to Traceroutes to learn to identify network problems with the most popular network troubleshooting tool for IT Pros.

Download Now

Refer back to previous articles which analyze traceroutes and which information is the most important.

We hope you enjoyed this article in the traceroute series.

• What is a Traceroute and How Do Traceroutes Work?
• How To Identify Network Issues with Traceroutes?
• Why Do Some Routers Drop Packets or Have High Latencies?
• Decode the Hidden Information from Traceroute DNS
• Internet Traffic is Asymmetrical - How to Catch Reverse Path Issues?
• How to Share a Traceroute With an ISP NOC?
• Impact of Load Balancing or Multiple Paths on Traceroutes
• MPLS Networks, TTL Propagation and ICMP Tunneling (this article)