| Prev: NTP Won't Sync to Slow Clocksource | Next: Install Linux on Virtualbox, Infinite Loop |
| (Index) |
On a net, typically at home, with multiple users, one person is using ssh to connect to work, one is playing World of Warcraft, one is reading web mail or news. Someone starts downloading a big ISO image or other file. Everyone is unhappy with their network performance, particularly with latency, i.e. the time from pressing return on the keyboard until seeing a response from the remote site.
At each stage of the process of sending packets, there are queues, which are generally bigger than they need to be for optimal performance. When the link is saturated, first, the big download is taking a proportionate share and not leaving enough for the others, and when a packet arrives from one of the smaller connections it has to go through the whole queue, which could take as much as 2 seconds.
Use Linux network traffic control. For the script detailed below the goals are:
Respond to the quality of service bits in the packets (outgoing). This puts interactive and streaming media ahead of bulk data (uploading).
Give priority to outgoing ACK packets, so incoming datastreams are not choked because the ACK packets have to go through the outgoing queue(s).
Limit the incoming data rate by dropping packets, so it's not possible for a queue to build up in the upstream routers. This way, returning packets for interactive connections aren't delayed by going through a big queue, even if the upstream router is not too smart about honoring quality of service bits.
Web links:
Linux Advanced Routing & Traffic Control HOWTO", Bert Hubert (editor), no date, but mod time is 2004-03-30. Describes traffic control in kernel 2.2 and 2.4 using the iproute2 package.
Documentation on parameters for Hierarchical Token Bucket (HTB) queueing discipline by Devik.
Results: Formerly when I started a big download the net would go gelid, with
latency of 2 secs or more. Here is what I found using the script below, on a
Verizon ADSL line with 768 kbit/sec download and 128 kbit/sec upload.
Sunset
is the outside host that I'm using to test.
With no load, ping reports a latency (time from sending the ICMP ping packet until getting a reply from Sunset) of 39 to 42 msec.
Downloading an ISO image from Sunset using scp. It reports 85 kbyte/sec, fairly steady. The theoretical maximum is 96 kbyte/sec and without traffic control speeds over 90 are often seen.
Ping latency goes up to 500 msec for a few seconds, then drops back to 48 to 77 msec.
Response on a ssh session is immediate -- you can't tell that a download is going on, where without traffic control it would be almost unuseable.
Web use (Google Maps) was its usual speed. When I moved the viewport the ping latency went up to around 110 msec until the map was downloaded.
Stopped the download, started uploading. The rate reported by scp
varies a lot between 8 and 16 kyte/sec, and it says stalled
from
time to time due to dropped packets. The theoretical maximum is 16
kbyte/sec. Possibly I need to work on tuning the parameters.
Ping latency varies erratically up to 115 msec. Google Maps response time was its usual speed. Response on ssh is immediate.
The script: This is from section 15.8 of the above traffic control HOWTO,
edited into a boot script suitable for use on SuSE or other LSB-type distros.
Minor editing may be needed on Debian or Ubuntu, e.g. they won't have the
rc_status function, which shows success or failure in color.
Save this page to a file and then edit to remove the HTML. It would go in
/etc/init.d and I call it netpolice.J. Make it executable.
Remember to do insserv /etc/init.d/netpolice.J
or make the appropriate
startup links by hand, so it will be executed at boot time.
#!/bin/bash
# Network Traffic Control for Home Network
# Plagarized by Jim Carter <jimc@math.ucla.edu> from HOWTO by
# Bert Hubert <bert.hubert@netherlabs.nl> (editor):
# http://lartc.org/howto/ (dated approx. 2004).
# Skeleton Copyright (C) 1995--2005 Kurt Garloff, SUSE / Novell Inc.
# Skeleton licensed under LGPL.
# Contains bash-isms (an arithmetic expression). Debian policy requires
# all scripts to run under any POSIX shell.
# See section 15.8.3 in the above HOWTO, and for docs on HTB (Hierarchical
# Token Bucket) queueing discipline, see sect. 9.5.5 and Devik's writeup:
# http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm
### BEGIN INIT INFO
# Provides: netpolice
# Required-Start: $syslog $network
# Should-Start:
# Required-Stop: $syslog $network
# Should-Stop:
# Default-Start: 3 5
# Default-Stop: 0 1 2 6
# Short-Description: Limits data rate on individual network connections.
# Description: Allocates network bandwidth fairly among
# ad-hoc network connections, both thru traffic and
# connections terminating on this box.
### END INIT INFO
# Check for missing binaries (stale symlinks should not happen)
# Note: Special treatment of stop for LSB conformance
TC=/usr/sbin/tc
test -x $TC || { echo "$TC (package iproute2) not installed";
if [ "$1" = "stop" ]; then exit 0;
else exit 5; fi; }
FOO_CONFIG=/etc/sysconfig/FOO
# Configuration parms should be in sysconfig but instead are hardwired here.
# Parameter format: decimal number followed by optional unit, e.g. 0.4mbps
# k means multiply by 2^10 (1024), m means 2^20.
# Rates: mbps, kbps, bps, or bare number = bytes/sec (times k or m).
# mbit, kbit = bits/sec
# Amounts: mb, m, kb, k, b, or bare number = bytes (times k or m).
# mbit, kbit = bits.
# Durations (seconds): [s, sec, secs], [ms, msec, msecs] (milliseconds),
# [us, usec, usecs, or bare number] (microseconds).
# Set the following values to somewhat less than your actual download
# and uplink speed. For us, SLA is 768kbit down, 128kbit up. Discount 5%.
DOWNLINK=730kbit
UPLINK=120kbit
# Supposedly, max latency will be around BURST/SPEED. Larger IBURST is OK.
# BURST must be greater than the MTU and greater than SPEED/HZ (clock tick
# rate). Unit k means 1024 bytes.
BURST=7.3k
IBURST=12k
DEV=eth1
# Shell functions for SuSE sourced from /etc/rc.status: rc_status etc.
. /etc/rc.status
# Reset status of this service
rc_reset
# Get rid of pre-existing setup (silently).
function do_stop () {
$TC qdisc del dev $DEV root > /dev/null 2>&1
$TC qdisc del dev $DEV ingress > /dev/null 2>&1
return 0
}
# Set up the queueing discipline.
function do_start () {
# Set up the uplink (outgoing traffic on $DEV)
# Creates root HTB; unmatched traffic goes to 1:20
$TC qdisc add dev $DEV root handle 1:0 htb default 20
# Choke outgoing traffic eliminating queues in upstream
# routers. Our traffic shaper manages latency better --
# reduces latency from over 2 secs to under 100 msec.
$TC class add dev $DEV parent 1:0 classid 1:1 htb \
rate $UPLINK burst $BURST
# This subclass for interactive QOS.
$TC class add dev $DEV parent 1:1 classid 1:10 htb \
rate $UPLINK burst $BURST prio 1
# SFQ (Stochastic Fair Queuing) within the class makes each
# connection (conntrack?) share equally, if overbooked.
$TC qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
# Filters to attract particular traffic to this class.
# QOS "Minimum Delay", ICMP, and TCP ACK packets are selected.
$TC filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
match ip tos 0x10 0xff flowid 1:10
$TC filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
match ip protocol 1 0xff flowid 1:10
$TC filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
match ip protocol 6 0xff \
match u8 0x05 0x0f at 0 \
match u16 0x0000 0xffc0 at 2 \
match u8 0x10 0xff at 33 \
flowid 1:10
# This subclass for everything else (bulk mail), hence less
# favorable priority. The original script gives it 90% of the
# rate of 1:10 but let's keep it simple.
$TC class add dev $DEV parent 1:1 classid 1:20 htb \
rate $UPLINK burst $BURST prio 2
# SFQ (Stochastic Fair Queuing) within the class makes each
# connection (conntrack?) share equally, if overbooked.
$TC qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
# Set up downlink (incoming traffic on $DEV) ingress policer.
# ** Handle must be ffff: (or ffff:0). Arbitrary handle not OK.
$TC qdisc add dev $DEV handle ffff:0 ingress
# Choke the flow (toss excess packets) to avoid queues in
# upstream routers that wreck your latency. This filter
# applies to all traffic on the interface.
$TC filter add dev $DEV parent ffff:0 protocol ip prio 50 u32 \
match ip src 0.0.0.0/0 \
police rate $DOWNLINK burst $IBURST drop flowid :1
}
case "$1" in
start | restart | try-restart | reload | force-reload )
echo -n "${1}ing netpolice "
do_stop
if do_start ; then : ; else rc_failed ; fi
rc_status -v
;;
stop)
echo -n "Stopping netpolice "
do_stop
rc_status -v
;;
status)
echo -n "Checking for service netpolice "
# Check for a few keywords in the "show" output. This would
# detect absence of the mechanism but many incomplete setups
# would not be detected. The "match" is for TOS = interactive.
if \
$TC qdisc show dev $DEV | grep -w htb > /dev/null && \
$TC class show dev $DEV | grep -w htb > /dev/null && \
$TC filter show dev $DEV | grep 'match 00100000/00ff0000 at 0' > /dev/null ; then : ; else
rc_failed 3
fi
rc_status -v
;;
*)
echo "Usage: $0 {start|stop|status|try-restart|restart|force-reload|reload}"
exit 1
;;
esac
rc_exit
Possibilities for improvement: I wish I could do something more elaborate on the ingress (download) datastream, particularly HTB and SFQ to split the capacity between multiple big download streams, similar to what's done on egress. But the ingress queueing discipline is like TBF and isn't able to have subclasses.
Possibly I could do traffic control on the LAN interface. I would need to restrict it to traffic coming from off-site. My box does a lot of service functions beyond routing, and I want internal hosts to be able to use the higher LAN bandwidth when talking to it. Also, big downloads typically would go to this box and an egress router would not be able to manage them.
| Prev: NTP Won't Sync to Slow Clocksource | Next: Install Linux on Virtualbox, Infinite Loop |
| (Index) |