Valid HTML 4.01 Transitional
Prev: NTP Won't Sync to Slow Clocksource Next: Install Linux on Virtualbox, Infinite Loop
Jim Carter's Bugfixes

Hogging Internet Connection

James F. Carter

On a net, typically at home, with multiple users, one person is using ssh to connect to work, one is playing World of Warcraft, one is reading web mail or news. Someone starts downloading a big ISO image or other file. Everyone is unhappy with their network performance, particularly with latency, i.e. the time from pressing return on the keyboard until seeing a response from the remote site.

What's happening:

At each stage of the process of sending packets, there are queues, which are generally bigger than they need to be for optimal performance. When the link is saturated, first, the big download is taking a proportionate share and not leaving enough for the others, and when a packet arrives from one of the smaller connections it has to go through the whole queue, which could take as much as 2 seconds.

How to fix:

Use Linux network traffic control. For the script detailed below the goals are:

Web links:

Results: Formerly when I started a big download the net would go gelid, with latency of 2 secs or more. Here is what I found using the script below, on a Verizon ADSL line with 768 kbit/sec download and 128 kbit/sec upload. Sunset is the outside host that I'm using to test.

The script: This is from section 15.8 of the above traffic control HOWTO, edited into a boot script suitable for use on SuSE or other LSB-type distros. Minor editing may be needed on Debian or Ubuntu, e.g. they won't have the rc_status function, which shows success or failure in color. Save this page to a file and then edit to remove the HTML. It would go in /etc/init.d and I call it netpolice.J. Make it executable. Remember to do insserv /etc/init.d/netpolice.J or make the appropriate startup links by hand, so it will be executed at boot time.

#     Network Traffic Control for Home Network
#     Plagarized by Jim Carter <> from HOWTO by 
#     Bert Hubert <> (editor):
# (dated approx. 2004).  
#     Skeleton Copyright (C) 1995--2005  Kurt Garloff, SUSE / Novell Inc.
#     Skeleton licensed under LGPL.

# Contains bash-isms (an arithmetic expression).  Debian policy requires
# all scripts to run under any POSIX shell.  

# See section 15.8.3 in the above HOWTO, and for docs on HTB (Hierarchical
# Token Bucket) queueing discipline, see sect. 9.5.5 and Devik's writeup:

# Provides:          netpolice
# Required-Start:    $syslog $network
# Should-Start:      
# Required-Stop:     $syslog $network
# Should-Stop:       
# Default-Start:     3 5
# Default-Stop:      0 1 2 6
# Short-Description: Limits data rate on individual network connections.
# Description:       Allocates network bandwidth fairly among
#	ad-hoc network connections, both thru traffic and
#	connections terminating on this box.

# Check for missing binaries (stale symlinks should not happen)
# Note: Special treatment of stop for LSB conformance
test -x $TC || { echo "$TC (package iproute2) not installed"; 
	if [ "$1" = "stop" ]; then exit 0;
	else exit 5; fi; }

# Configuration parms should be in sysconfig but instead are hardwired here.
#   Parameter format: decimal number followed by optional unit, e.g. 0.4mbps
#	k means multiply by 2^10 (1024), m means 2^20.  
#   Rates: mbps, kbps, bps, or bare number = bytes/sec (times k or m).
#	mbit, kbit = bits/sec
#   Amounts: mb, m, kb, k, b, or bare number = bytes (times k or m). 
#	mbit, kbit = bits.
#   Durations (seconds): [s, sec, secs], [ms, msec, msecs] (milliseconds),
#	[us, usec, usecs, or bare number] (microseconds).
# Set the following values to somewhat less than your actual download
# and uplink speed.  For us, SLA is 768kbit down, 128kbit up.  Discount 5%.  
# Supposedly, max latency will be around BURST/SPEED.  Larger IBURST is OK.
# BURST must be greater than the MTU and greater than SPEED/HZ (clock tick 
# rate).  Unit k means 1024 bytes.  

# Shell functions for SuSE sourced from /etc/rc.status: rc_status etc.
. /etc/rc.status

# Reset status of this service

# Get rid of pre-existing setup (silently).
function do_stop () {
    $TC qdisc del dev $DEV root    > /dev/null 2>&1
    $TC qdisc del dev $DEV ingress > /dev/null 2>&1
    return 0

# Set up the queueing discipline.  
function do_start () {
		# Set up the uplink (outgoing traffic on $DEV)
		# Creates root HTB; unmatched traffic goes to 1:20
    $TC qdisc add dev $DEV root handle 1:0 htb default 20
		# Choke outgoing traffic eliminating queues in upstream 
		# routers.  Our traffic shaper manages latency better -- 
		# reduces latency from over 2 secs to under 100 msec.
    $TC class add dev $DEV parent 1:0 classid 1:1 htb \
	rate $UPLINK burst $BURST
		# This subclass for interactive QOS.  
    $TC class add dev $DEV parent 1:1 classid 1:10 htb \
	rate $UPLINK burst $BURST prio 1
		# SFQ (Stochastic Fair Queuing) within the class makes each
		# connection (conntrack?) share equally, if overbooked.
    $TC qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
		# Filters to attract particular traffic to this class.
		# QOS "Minimum Delay", ICMP, and TCP ACK packets are selected.
    $TC filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
	match ip tos 0x10 0xff  flowid 1:10
    $TC filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
	match ip protocol 1 0xff flowid 1:10
    $TC filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
	match ip protocol 6 0xff \
	match u8 0x05 0x0f at 0 \
	match u16 0x0000 0xffc0 at 2 \
	match u8 0x10 0xff at 33 \
	flowid 1:10

                # This subclass for everything else (bulk mail), hence less
		# favorable priority.  The original script gives it 90% of the
		# rate of 1:10 but let's keep it simple.
    $TC class add dev $DEV parent 1:1 classid 1:20 htb \
	rate $UPLINK burst $BURST prio 2
		# SFQ (Stochastic Fair Queuing) within the class makes each
		# connection (conntrack?) share equally, if overbooked.
    $TC qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10

		# Set up downlink (incoming traffic on $DEV) ingress policer.
		# ** Handle must be ffff: (or ffff:0).  Arbitrary handle not OK.
    $TC qdisc add dev $DEV handle ffff:0 ingress
		# Choke the flow (toss excess packets) to avoid queues in
		# upstream routers that wreck your latency.  This filter 
		# applies to all traffic on the interface.
    $TC filter add dev $DEV parent ffff:0 protocol ip prio 50 u32 \
	match ip src \
	police rate $DOWNLINK burst $IBURST drop flowid :1

case "$1" in
    start | restart | try-restart | reload | force-reload )
	echo -n "${1}ing netpolice "
	if do_start ; then : ; else rc_failed ; fi
	rc_status -v
	echo -n "Stopping netpolice "
	rc_status -v
	echo -n "Checking for service netpolice "
		# Check for a few keywords in the "show" output.  This would
		# detect absence of the mechanism but many incomplete setups
		# would not be detected.  The "match" is for TOS = interactive.
	if \
	    $TC qdisc show dev $DEV | grep -w htb > /dev/null && \
	    $TC class show dev $DEV | grep -w htb > /dev/null && \
	    $TC filter show dev $DEV | grep 'match 00100000/00ff0000 at 0' > /dev/null ; then : ; else
		rc_failed 3
	rc_status -v
	echo "Usage: $0 {start|stop|status|try-restart|restart|force-reload|reload}"
	exit 1

Possibilities for improvement: I wish I could do something more elaborate on the ingress (download) datastream, particularly HTB and SFQ to split the capacity between multiple big download streams, similar to what's done on egress. But the ingress queueing discipline is like TBF and isn't able to have subclasses.

Possibly I could do traffic control on the LAN interface. I would need to restrict it to traffic coming from off-site. My box does a lot of service functions beyond routing, and I want internal hosts to be able to use the higher LAN bandwidth when talking to it. Also, big downloads typically would go to this box and an egress router would not be able to manage them.

Prev: NTP Won't Sync to Slow Clocksource Next: Install Linux on Virtualbox, Infinite Loop