What would cause UDP packets to be dropped when being sent to localhost?

Question

I\'m sending very large (64000 bytes) datagrams. I realize that the MTU is much smaller than 64000 bytes (a typical value is around 1500 bytes, from my reading), but I would

Answer 1

UDP pkts scheduling may be handled by multiple threads on OS level. That would explain why you receive them out of order even on 127.0.0.1.

Answer 2

I don't know what makes you expect a percentage less then 1% of dropped packets for UDP.

That being said, based on RFC 1122 (see section 3.3.2), the maximum buffer size guaranteed not to be split into multiple IP datagrams is 576 bytes. Larger UDP datagrams may be transmitted but they will likely be split into multiple IP datagrams to be reassembled at the receiving end point.

I would imagine that a reason contributing to the high rate of dropped packets you're seeing is that if one IP packet that was part of a large UDP datagram is lost, the whole UDP datagram will be lost. And you're counting UDP datagrams - not IP packets.

Answer 3

Overview

What is causing the inability to send/receive data locally?

Mostly buffer space. Imagine sending a constant 10MB/second while only able to consume 5MB/second. The operating system and network stack can't keep up, so packets are dropped. (This differs from TCP, which provides flow control and re-transmission to handle such a situation.)

Even when data is consumed without overflowing buffers, there might be small time slices where data cannot be consumed, so the system will drop packets. (Such as during garbage collection, or when the OS task switches to a higher-priority process momentarily, and so forth.)

This applies to all devices in the network stack. A non-local network, an Ethernet switch, router, hub, and other hardware will also drop packets when queues are full. Sending a 10MB/s stream through a 100MB/s Ethernet switch while someone else tries to cram 100MB/s through the same physical line will cause dropped packets.

Increase both the socket buffers size and operating system's socket buffer size.

Linux

The default socket buffer size is typically 128k or less, which leaves very little room for pausing the data processing.

sysctl

Use sysctl to increase the transmit (write memory [wmem]) and receive (read memory [rmem]) buffers:

• net.core.wmem_max
• net.core.wmem_default
• net.core.rmem_max
• net.core.rmem_default

For example, to bump the value to 8 megabytes:

sysctl -w net.core.rmem_max=8388608

To make the setting persist, update /etc/sysctl.conf as well, such as:

net.core.rmem_max=8388608

An in-depth article on tuning the network stack dives into far more details, touching on multiple levels of how packets are received and processed in Linux from the kernel's network driver through ring buffers all the way to C's recv call. The article describes additional settings and files to monitor when diagnosing network issues. (See below.)

Before making any of the following tweaks, be sure to understand how they affect the network stack. There is a real possibility of rendering your network unusable. Choose numbers appropriate for your system, network configuration, and expected traffic load:

• net.core.rmem_max=8388608
• net.core.rmem_default=8388608
• net.core.wmem_max=8388608
• net.core.wmem_default=8388608
• net.ipv4.udp_mem='262144 327680 434274'
• net.ipv4.udp_rmem_min=16384
• net.ipv4.udp_wmem_min=16384
• net.core.netdev_budget=600
• net.ipv4.ip_early_demux=0
• net.core.netdev_max_backlog=3000

ethtool

Additionally, ethtool is useful to query or change network settings. For example, if ${DEVICE} is eth0 (use ip address or ipconfig to determine your network device name), then it may be possible to increase the RX and TX buffers using:

• ethtool -G ${DEVICE} rx 4096
• ethtool -G ${DEVICE} tx 4096

iptables

By default, iptables will log information about packets, which consumes CPU time, albeit minimal. For example, you can disable logging of UDP packets on port 6004 using:

iptables -t raw -I PREROUTING 1 -p udp --dport 6004 -j NOTRACK
iptables -I INPUT 1 -p udp --dport 6004 -j ACCEPT

Your particular port and protocol will vary.

Monitoring

Several files contain information about what is happening to network packets at various stages of sending and receiving. In the following list ${IRQ} is the interrupt request number and ${DEVICE} is the network device:

• /proc/cpuinfo - shows number of CPUs available (helpful for IRQ-balancing)
• /proc/irq/${IRQ}/smp-affinity - shows IRQ affinity
• /proc/net/dev - contains general packet statistics
• /sys/class/net/${DEVICE}/queues/QUEUE/rps_cpus - relates to Receive Packet Steering (RPS)
• /proc/softirqs - used for ntuple filtering
• /proc/net/softnet_stat - for packet statistics, such as drops, time squeezes, CPU collisions, etc.
• /proc/sys/net/core/flow_limit_cpu_bitmap - shows packet flow (can help diagnose drops between large and small flows)
• /proc/net/snmp
• /proc/net/udp

Summary

Buffer space is the most likely culprit for dropped packets. There are numerous buffers strewn throughout the network stack, each having its own impact on sending and receiving packets. Network drivers, operating systems, kernel settings, and other factors can affect packet drops. There is no silver bullet.

Further Reading

• https://github.com/leandromoreira/linux-network-performance-parameters
• http://man7.org/linux/man-pages/man7/udp.7.html
• http://www.ethernetresearch.com/geekzone/linux-networking-commands-to-debug-ipudptcp-packet-loss/

Answer 4

Your expectations, as expressed in your question and in numerous comments to other answers, are wrong. All the following can happen even in the absence of routers and cables.

1. If you send a packet to any receiver and there is no room in his socket receive buffer it will get dropped.

2. If you send a UDP datagram larger than the path MTU it will get fragmented into smaller packets, which are subject to (1).

3. If all the packets of a datagram don't arrive, the datagram will never get delivered.

4. The TCP/IP stack has no obligation to deliver packets or UDP datagrams in order.

Answer 5

UDP packets are not guaranteed to reach their destination whereas TCP is!