Packet Timestamping

Depending on the type of agent and its configuration, the packet timestamping is different inside the agent.

For the agents running the Linux software (including hardware and virtual appliances) Kernel-based timestamping is used. This means the timestamp is calculated by the Linux Kernel when the packet leaves the network interface. With this method, the agent gets better accuracy on the packet timestamps (i.e. when the packets has been sent and received). The SO_TIMESTAMPNS socket option is used to get the timestamp. More details in the Linux Kernel Documentation.

Pros:

• Better accuracy
• No need for an external software

Cons:

• Docker container requires privilege mode
• Only supports the default network interface (VLAN untagged)
• If the Server Agent is on the same subnet (VLAN / LAN / Broadcast domain) than the Client Agent, when the Server Agent is completely down, it will not respond to ARP Requests. If there is no ARP entry on the Client Agent, the packet will not be sent by the OS and it will not be captured by the Client Agent as a sent packet. In this case, the Client Agent will not report packet loss. A workaround is to configure a static ARP entry.

Notes:

• The UDP source port never changes, which means a consistent flow per session. This is great for very restrictive firewall configurations but can cause issues with some firewalls.

When we ported the software agent to Windows, a new timestamping method was required because there is no equivalent to the Linux Kernel Timestamping on Windows. The agent software is built on Python, which is not a good option for near realtime timestamping because of the Python GIL. The solution was to build an external software in GO that will be in charge of sending, receiving and timestamping the packets.

The Obkio Agent Timestamper Software is available on all agent versions and is the default option (and the only one) for non-Linux operating systems. To use the Obkio Agent Timestamper on Linux, see the Linux Environment Variables Configuration.

Pros:

• A new dynamic source port is used at each connection
• A new connection is created if a new IP address is detected on the NICs
• A new connection is created if no packets are received after 5 seconds
• Docker container does not require privilege mode
• By using the OS network stack, it can have multiple interfaces (VLAN tagged or untagged)

Cons:

• Reduced accuracy on embedded devices
• Doesn't support DSCP marking
• Uses slightly more CPU and Memory (important for embedded devices only)

There are some cases where misconfigurations and bugs on the firewalls can block the monitoring traffic when using the Linux Kernel Timestamping. This is one reason why some users might find it useful to use the Obkio Agent Timestamper Software on Linux machines.

The issue happens because the source port of the UDP packet never changes. By default, it's the same as the server port: UDP/23999. This means that between a Client A and a Server B, the packets sent by Client A are always:

IP_CLIENT_A:23999 -> IP_SERVER_B:23999

This is not a problem. This is not a bug with Obkio. But in some rare cases, it can create stalled entries in the firewall's NAT or routing tables. A connection failover sometimes triggers that kind of bug. There are a few solutions when this happens:

• Clear the stalled entry
• Clear all the entries in the table if a single entry doesn't work
• Stop the agent traffic for a few minutes. This way, the entry is cleared with a timeout.
• Change the client source port to another port
• Reboot the firewall (Is this really a solution?)

This kind of issue does not affect only Obkio monitoring traffic. Some users reported the exact same issue with IPSEC tunnels between locations.

The Obkio Agent Timestamper Software is not affected by this kind of issue because, when no packets are received for 5 seconds, a new source port is used which makes a new entry in the firewall.