The Great Packet Hunt: A Comprehensive Guide to Troubleshooting Packet Loss

• What is Packet Loss?
• Step 1. Monitor Packet Loss In Your Network
• Step 2. Measure Packet Loss Rates
• Step 3. Where Packet Loss Happens In A Network
• Step 4. Compare the Network Sessions
• Step 5. Troubleshooting with Device Monitoring
• Step 6. Troubleshooting with Visual Traceroute
• Step 7. Understand the Cause of Packet Loss
• Step 8. Implement A Fix
• Start the Great Packet Hunt

Why did the network administrator have trouble troubleshooting packet loss? Because the packets were playing a game of hide and seek, and they were winning!

Are you tired of playing a game of hide-and-seek with your data packets? Does the thought of lost information keep you up at night? Fear not, brave network adventurer! You are about to embark on a journey that will take you through the wild and mysterious world of packet loss troubleshooting. We call it "The Great Packet Hunt," and it's a journey that will require your keen sense of observation, your analytical mind, and your unyielding determination to catch those pesky packets once and for all.

If you're experiencing network issues, such as slow Internet speeds, poor audio or video quality, or dropped connections, packet loss could be the culprit. Packet loss is a common network issue that affects networks of all sizes, and its impact on users and businesses can be very frustrating if it’s not dealt with quickly. So, grab your tools and gear up for an adventure like no other as we explore how to identify and troubleshoot packet loss, and the causes of packet loss in your network!

Packet loss is a network issue that occurs when data packets being transmitted over a network but get lost or dropped during transit and fail to reach their destination. Kind of like sending a letter in the mail that gets damaged in transit. Maybe the letter is ripped in half and only part of the message is delivered to the sender.

The impact of packet loss can degrade network and application performance and frustrated users and IT pros alike. And the longer it persists, the more it can affect your business. Which is why it’s important for IT specialists and network admins to know how to troubleshoot packet loss as soon as it appears.

Ready to start your packet hunting adventure? The first step is to verify that packet loss is indeed the issue you're facing. To do this, you'll need to accurately replicate the problem and closely monitor your network's performance to measure the extent of the issue. With this crucial information in hand, you'll be well on your way to tracking down those missing packets and resolving any network issues.

Once you've determined that packet loss is the cause of your network problems, you can begin to take steps to address the issue.

To first identify packet loss issues, you need to continuously monitor your network performance using an agent-based tool, like Obkio Network Performance Monitoring, to measure packet loss between two points in your network.

Obkio’s solution identifies even the earliest signs of packet loss anywhere in your network, and then collects the data you need to troubleshoot.

Get started with Obkio’s Free Trial!

To begin monitoring packet loss, you need to deploy Network Monitoring Agents which continuously measure network performance in key network locations like offices, data centers and clouds. The Agents exchange synthetic traffic to measure network metrics, like packet loss, between each other.

This will help you determine if packet loss is indeed the root cause of the problem and identify any network segments where packet loss is most pronounced. Essentially, to collect the data you need to translate, you’ll need:

1. Local Agents: Installed in the targeted office location experiencing connectivity issues. There are several Agent types available (all with the same features), and they can be installed on MacOS, Windows, Linux and more.
2. Public Monitoring Agent: Which are deployed over the Internet and managed by Obkio. They compare performance up to the Internet and quickly identify if the problem is global or specific to the destination. This will be great information for later in the troubleshooting process. You can use an AWS (monitor apps like Zoom) or Google Cloud Agent.

This way you can monitor, for example: packet loss between your head office and the Google Cloud, or even between Google Cloud and your data center.

Once your Monitoring Agents are deployed, they will continuously exchange synthetic UDP traffic between each other to monitor performance, this is called a Network Monitoring Session.

To identify packet loss issues, you’ll want to let the Agents run and collect data for some time. If you’re unlucky enough, you’ll see the performance issues after just a few minutes, but we recommend letting the Agents collect data for at least 3 hours.

Obkio's Network Response Time Graph and VoIP Quality Graph showing Latency, Jitter, and Packet Loss

When the Agents exchange synthetic traffic, they will monitor and collect performance data, which you can see on the Network Response Time Graph. At this point you can then measure your packet loss rate (the amount of packet loss in your network)

1. High Packet Loss: Typically above 10%, high packet loss can cause serious disruptions to your Internet connectivity. At this level, you may experience sluggish page loads, choppy video and audio playback, and delays in online gaming. In some cases, high packet loss can even lead to complete disconnection from the Internet.
2. Acceptable Packet Loss: Acceptable packet loss (or low packet loss) is typically below 1%. In other words, if 99% or more of packets are successfully transmitted and received without any loss, the network is generally considered to be performing well. Acceptable packet loss may not be noticeable In your internet usage. However, even small levels of packet loss can result in increased latency, which can make your Internet feel slower and less responsive.

When it comes to packet loss, no place is off-limits! Whether it's your local network, a WAN, your ISP, or the destination network, packet loss can strike at any point where data is transmitted or received. That's why it's essential to investigate and troubleshoot any issues thoroughly, in order to pinpoint the root cause of the problem and restore your network's optimal performance. Don't let those pesky packets get the best of you - start your packet hunting journey today!

Some common locations where packet loss can occur include:

1. Local network: Packet loss can occur on your local network, such as when data is transmitted between devices on the same network, due to issues with network congestion, outdated network equipment, or misconfigured network settings.
2. Wide Area Network (WAN): Packet loss can also occur on a WAN, such as when data is transmitted across different geographic locations, due to issues with network congestion, bandwidth limitations, or network infrastructure problems.
3. Internet Service Provider (ISP): Packet loss can occur due to issues with your Internet Service Provider, such as problems with their network infrastructure or congestion on their network.
4. Destination network: Packet loss can also occur on the destination network, such as when data is transmitted to a remote server, due to issues with their network infrastructure, congestion, or bandwidth limitations.

But how can you pinpoint where the packet loss is happening in your network? Well, you follow the next steps!

At this point, if you have a packet loss issue, you’ll be able to see it in the graph above. You can see the exact moments when packet loss occurred in your network and even the impact on MOS Score.

From here, you’re ready to troubleshoot the packet loss in just 3 more packet loss troubleshooting steps!

To begin troubleshooting your packet loss, you first need to catch it first.

To do this, you’ll need to compare the monitoring sessions between the AWS Public Monitoring Agent and the other Public Monitoring Agent you deployed.

In the screenshot below, you’ll see an example of a Network Session with no packet loss. As a reference, this is what it looks like when the packet loss isn’t the cause of your problem.

If it’s not a network issue, it might be a user issue. In this case, you can install a Monitoring Agent directly on a user’s workstation (the user who is experiencing the packet loss) to see the packet loss issue from their point-of-view.

If you still don’t find that it’s a network problem at this point, it may just be a problem that IT specialists can solve directly on the user's workstation or on the remote destination. The problem could be several other hardware-intensive videoconferencing systems, which use up a lot of resources (CPU, RAM).

Obkio also monitors these other resource metrics for you!

In the screenshot below is an example of a packet loss issue happening on both network sessions.

In this case, the packet loss is happening on a network segment that is common to both network sessions. This means that the problem is:

• Broader and not exclusive to a single network path or destination
• And affects all services and applications that depend on the network.
• This could be in the LAN, the firewall or the local loop Internet connection.

Skip to Step 5 for the next step.

In the screenshot below is an example of a packet loss issue happening on only one network session.

This means that the packet loss issue is happening specifically towards that specific location on the Internet and that the problem is further away.

In this case, skip right to Step 6.

If the packet loss is happening on both network sessions, compare that data using Obkio’s Device Monitoring feature with SNMP monitoring to further understand if the packet loss issue is happening on your end, or over the Internet, in your Service Provider’s network.

When you compare your previous data with Device Monitoring metrics, if you find CPU or Bandwidth issues, this likely means that the problem causing packet loss is on your end and you need to troubleshoot internally.

There are many different causes of packet loss, and device and resource issues, like high bandwidth and CPU usage causing network congestion, are some of the most common causes.

Here are a few suggestions:

1. What Traffic is Being Used: Look into if traffic going through your network is legitimate or not by looking at the firewall logs. Illegitimate traffic might be a security breach (data exfiltration), or mass data backup happening during business hours instead of during the night.
2. Manage Firewall Priorities: To prioritize important traffic in order to reduce the impact on those applications during periods of congestion, since network congestion is one of the major causes of packet loss. This means that all the traffic towards or from those applications will be prioritized first when passing through the network and will be less likely to cause congestion and packet loss.
3. Upgrade Internet Connection Bandwidth: With your ISP if you’re just running out of bandwidth.
4. Investigate Missing Resources: Look into your device to understand why it’s missing resources and causing high CPU usage that can lead to packet loss:
   • There may be a software problem
   • Your device may need a firmware update
   • The latest update may be bugged
   • Or the device may just not have enough resources available and needs to be upgraded to a bigger device

If you don’t see any resource issues from your devices, this is a sign that the problem causing packet loss is located on your Service Provider’s end.

In this case, you need to open a service ticket with all the information as you can collect to prove that the issue is on the ISP’s end, and get quickly escalated past 1st level support. Obkio will be your secret weapon for that.

The data you can provide is that from your dashboard in the previous steps, as well as your Traceroutes in the next step.

Now, you can now collect the last bit of data to help you pinpoint exactly where the network problem causing packet loss is located so you can share that information with your Service Provider.

To do this, we’ll be using Obkio Vision, Obkio’s free Visual Traceroute tool that runs continuously to interpret Traceroute results to identify network problems in your WAN and over the Internet.

Obkio Vision Quality Matrix

Note: If the packet loss is happening only on your end, you don’t need to do this step. It’ll just further confirm that conclusion.

By using Traceroutes, the Network Map, and the Quality Matrix you’ll be able to identify if:

If the packet loss is happening only towards a specific location over the Internet, it means that only one site is being affected.

Obkio Vision Network Map

If the packet loss issue is happening on your ISP’s side, and you need to open a service ticket with as much information as you can collect.

Obkio Vision Traceroutes

1. Contact your ISP to get technical support using the screenshots of Monitoring Sessions, Dashboards or Traceroutes in Vision.
2. Share results of Live Traceroutes with your ISP using a public link.
3. If your ISP wants to analyze your data further, you can create a temporary Read-Only User in your Obkio account for them.

How to Troubleshoot Networks with Obkio Vision Visual Traceroute

Learn how to use Obkio Vision’s Visual Traceroute tool to troubleshoot network problems with traceroutes both inside & outside your local network.

Learn more

Packet loss can be caused by a variety of different factors, and not all of them are easy to identify. For businesses with large enterprise networks, having a tool like Obkio to monitor packet loss is especially helpful to understand what causes packet loss in your network.

By following the previous steps we mentioned, Obkio's solution will help you identify what exactly is causing packet loss issues in your network. Here are some of the most common causes of packet losss that you'll encounter:

1. Network Congestion: occurs when there is high network traffic, data packets can be delayed or dropped due to limited bandwidth.
2. Faulty Network Hardware: such as routers, switches, or cabling, can cause packet loss. Software Bugs: bugs or glitches in network software can also cause packet loss.
3. Security Measures: some security measures, such as firewalls or intrusion detection systems, may block or filter certain types of data packets, resulting in packet loss.
4. Latency: long distances between network devices or multiple network hops can increase the likelihood of packet loss.
5. Quality of Service (QoS) Settings: QoS settings may prioritize certain types of network traffic over others, leading to packet loss for less important traffic.

Are you ready to delve deeper into the world of packet loss? If you've already collected information about what's causing packet loss in your network, then you're one step closer to conquering this issue! Now it's time to roll up your sleeves and get to work. With careful analysis and troubleshooting, you can identify the underlying cause of packet loss and take the necessary steps to address it.

So, grab your tools and let's explore some helpful tips for fixing packet loss once and for all!

1. Check network equipment: Check your network equipment, including switches, routers, and cables, to ensure they are functioning correctly. Replace or repair any faulty equipment.
2. Update firmware: Make sure all network devices are running the latest firmware, as outdated firmware can cause packet loss.
3. Optimize network settings: Review and optimize network settings, such as TCP/IP settings, buffer sizes, and window sizes. Ensure that your network is properly configured for your specific use case.
4. Reduce network congestion: Reduce network congestion by limiting the number of devices on your network, implementing Quality of Service (QoS), and optimizing your network topology.
5. Use error correction technologies: Use error correction technologies such as Forward Error Correction (FEC) and Automatic Repeat Request (ARQ) to detect and correct packet loss.
6. Contact your ISP: If the problem persists, contact your Internet Service Provider (ISP) and report the issue. They may be able to identify and resolve any problems with their network that are causing packet loss.
7. Increase bandwidth: Consider increasing your network's bandwidth to reduce the likelihood of packet loss during periods of high traffic.

Congratulations, fellow packet hunters! You have successfully completed "The Great Packet Hunt" and emerged victorious against the elusive packet loss beast. You've mastered the art of troubleshooting packet loss and armed yourself with the knowledge and tools to conquer this challenge.

Once you’ve collected the data you need to troubleshoot packet loss in your network, you don’t have to dread the next time you receive an emergency call from your boss.

So, what's next? Well, don't let your guard down just yet. Packet loss is one of the most common network problems and it can always come back. Remember to stay vigilant and monitor your network regularly to prevent future packet loss issues. And if you do encounter any more problems, fear not - you now have Obkio's Network Monitoring tool and the skills to track down those pesky packets and get your network back on track.

Install Monitoring Agents in all your network locations, and keep Obkio as your permanent packet loss monitoring and troubleshooting tool!

Get started with Obkio’s Free Trial - Happy hunting, and may the packets be forever in your favor!