[Discuss] [BBLISA] November meeting: Jim Gettys on Internet buffer bloat

[tmetro+blu at gmail.com (Tom Metro)]

> Subject: [BBLISA] November meeting: Jim Gettys on Internet buffer bloat
> 
> Wedesday, November 14th
> 
>        Internet buffer bloat, by Jim Gettys (Bell Labs)
> 
>        VOIP and teleconferencing often perform much more poorly on today's
>        Internet than the Internet of a decade ago, despite great gains in
>        bandwidth. [...]
> 
>        The mistaken quest to never drop packets has destroyed interactivity
>        under load, and often results in actual higher packet loss, as TCP's
>        congestion avoidance algorithms have been defeated by these buffers.

I "live blogged" this on BLU's IRC channel:
http://blu.wikispaces.com/IRC

Here's a slightly edited transcript:

BBLISA talk on buffer bloat by Jim Gettys

Almost all modern devices have too much buffering.
Few use queue management.

Jim's investigation came about because his home network didn't work for
teleconferencing.
His research turned up that it was a widespread problem, and that there
were mistakes being made at multiple levels.

Netalyzer - a java tool to measure and report buffer bloat.
Data aggregated from Netalyzer shows that almost all broadband
connections have too much latency for telephony.

Buffers exist at every level - hardware, device drivers, VM, and OS.
Then switches, wireless access points, routers, and cable modems.

Cellular carriers also have buffering problems.

Web browsers have eliminated limits on the number of outbound TCP
connections.
Web sites have intentionally split up, or 'sharded', their content so it
can be retrieved in parallel.
These two things combine to destroy TCP congestion avoidance.
The batch of reply packets loads up the receive buffers of your net
connection.
A cnn.com connection will induce 60 connections.

Fixed sized buffers don't work, because sizing depends on bandwidth, and
that varies dynamically, sometimes over orders of magnitude.

Buffers often optimized for maximum bandwidth at 1 Gbps, with no
intelligent adjustments made if actually running at 100 Mbps or less.

Active Queue Management (AQM) is the answer.

All queues need management, including those in end-point devices
(laptops, phones, etc.)

Red (or weighted red (WRED)) is best known queue management algorithm.
RED is flawed. It requires tuning.

CoDel - constant delay - algorithm is the new alternative to RED.
Supported in Linux 3.5
Two flavors.

CoDel isn't yet proven to be a universal answer. Known to have some
bugs, and expected to not be optimal for high speed networks inside data
centers.
It doesn't play well with BitTorrent's bandwidth management, which looks
at delays. CoDel reduces delays so BT boosts utilization, even though it
is supposed to be a background task.

Lots of progress in Linux in the last 6 months.
Linux has made the most progress in this area of any OS. Byte queue
limits, CoDel, etc.

Software on consumer routers can be 4 or 5 years old. Rarely updated
beyond the first year after release.
Little money is spent on router software engineering.
Author of Dnsmasq, which is included with almost all routers, has never
been contacted by a router vendor seeking improvements.

Atheros wireless chip has best radio, and Linux driver has been fix to
address buffering.
Use OpenWRT instead of stock firmware.
Jim recommends a WNDR3800 router which has an Atheros chip.
(That appears to be a Netgear model:
http://www.netgear.com/home/products/wirelessrouters/high-performance/wndr3800.aspx
)

You can start addressing the buffer problem with your existing router,
if it supports QoS. Set your upstream bandwidth to 70% of your actual
measured bandwidth to create an artificial choke point. This way the
router and modem buffers never get filled.

Jim will be collaborating with smallnetbuilder.com in the future (when
things settle down) so their reviews evaluate how routers handle buffering.

Standard in works to add buffer controls to cable modems. (Amendment to
DOCSIS3 and will be part of DOCSIS4.) Some modems available today have
it. Comcast leading the way on the ISP side in rolling out the new
technology. Expected within the year.

Jim's blog: http://gettys.wordpress.com
http://www.bufferbloat.net/projects/bloat

End of talk.

 -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: http://tmetro.venturelogic.com/