Re: Why BitTorrent causes so much latency and how to fix it

Interesting article George. However, I'm still not sure if that's all there is to it or not. I'm not a networking expert (I'm a neophyte at best), but here are some things I've observed:

1. If I'm uploading for a long time, port forwarding seems to stop.
Before memorial day, my open ports always came up as blocked on sites that check such things.

I've sometimes fixed this by changing the mac address of my router (don't ask me why, but for some reason, the mac address of the router seems to determine your IP on cable networks), but not always.

Resetting the Cable Modem/Router/Computer has no effect......anyway, none of that worked, bu after a long memorial day weekend at my parents, I came home and it was working.

Maybe it's a coincidence, but it seems very odd that a weekend of no torrenting and suddenly it's unblocked. I'm not sure if Comcast can do something to turn off port forwarding (presumably in the cable modem itself) or not.

Finally, while I think some sort of net neutrality bill would be good, I have to agree that the idea that QOS might be banned disturbs me.

What disturbs me even more is the supposedly tech informed people on several tech sites actually think that QOS discriminates against them :rolleyes:.

One thing I think that all ISPs need to do is list a minimum level of service, in addition to the max.

Regardless, i don't think that they need to allow charging for priority. ISPs should be motivated to implement such things, because it will improve their customers service and likely mean they can get more out of their bandwidth.

Verizon embraces P4P, a more efficient peer-to-peer tech

FYI:

http://arstechnica.com/news.ars/post/20080314-verizon-embraces-p4p-a-more-efficient-peer-to-peer-tech.html

An attempt to improve upon P2P.

Thanks George for illuminating on the subject. Very interesting.

Again, P4P has nothing to do with this dietrich

A lot of people are overstating the capability of this solution and I had to correct them on it.

http://blogs.zdnet.com/BTL/?p=8229
As I explained on the bottom

"P4P technology, or network-aware and network-optimized P2P (Peer to Peer) technology, is a huge step forward for P2P. It not only improves performance of P2P technology, but it also alleviates congestion on the distribution and core layer of the Internet because peers will more intelligently select the shortest paths instead of the more random approach that P2P takes now.

What P4P can’t do however is alleviate last-mile congestion on shared medium networks. That means cable broadband and wireless broadband technology will not benefit from the innovations of the P4P working group and therefore must not be confused as a possible solution for Comcast. On a DOCSIS 1.1 cable broadband network, there is only 10 Mbps of shared upstream capacity amongst an average of 200 to 400 customers. Even DOCSIS 3.0 cable broadband can only support 120 Mbps of upstream capacity shared between 200 and 400 customers."

Also note that "Ono" is an open source and open system alternative of P4P that deduces shortest path from CDNs like Akamai caching servers. If two clients get data from the same CDN, chances are they're close to each other.

My web seeding solution (http://www.formortals.com/Default.aspx?tabid=36&EntryID=22) which I've pitched to Comcast as something that ought to be pushing can complement P4P to help alleviate last-mile upstream congestion.

However, you're always going to need network management regardless of how much capacity you throw at it. Just look at Japan with their 100 Mbps fiber to the home. Their networks are heavily congested due to P2P traffic. P4P takes some of that stress away but it is the same as adding more capacity, people simply use the network more if you give them more capacity. They'll simply go to higher bit-rate higher quality videos if you give them more capacity and more efficient delivery.

Then there is this issue of latency which happens independent of whether you have bandwidth available or not. This is what this article addresses.


George Ou

You need to check your own router notgonnatellya

Sounds like you might be relying on dynamic port opening which doesn't always work. If you want that port to stay open, go to your router interface and manually create a permanent and static port forward. You would need to set your computer to a static internal IP. This has nothing to do with your ISP and it has everything to do with your own equipment. Your ISP isn't blocking those ports; your own router is.

"One thing I think that all ISPs need to do is list a minimum level of service, in addition to the max."

I've always said that more transparency is better. It would certainly take a lot of heat off of the ISPs if they were more transparent about what they did and didn't do.

"Regardless, i don't think that they need to allow charging for priority. ISPs should be motivated to implement such things, because it will improve their customers service and likely mean they can get more out of their bandwidth. "

But are you sure you want the Government to impose a no-charge rule? That would pretty much force the bundling of the service so that people who don't need or want it would have to subsidize it for others. It could also mean that the ISP can't justify offering it to everyone so they don't offer it at all. Do you want the Government to force an all-or-nothing situation?

Re: Why BitTorrent causes so much latency and how to fix it

There is already a solution to this problem.

Go get CfosSpeed, a traffic shaper that maximize the download/upload while
keeping your latency to a minimum. And you can also prioritize diffrent protocols and applications.

http://www.cfos.de/index2_e.htm

Can you contact me Stefan?

I have some questions for you and I'd like to check out the software. I would need to understand more about the solution and I would need to validate the results with some hard data before I can give any kind of assessment.

Re: Why BitTorrent causes so much latency and how to fix it

"Again, P4P has nothing to do with this dietrich"

Not true. Go here and see the 'players' involved in P4P Working Group. It is a 'work in progress':

http://www.pandonetworks.com/p4p

Thanks

No Dietrich, even Pando won't try and tell you it solves the last-mile congestion and latency problem

No Dietrich, even Pando won't try and tell you it solves the last-mile congestion and latency problem. If they did, they're selling you a load of Kool-Aid and you shouldn't be so eager to drink it and repeat it.

You should understand that P4P doesn't magically make the last-mile problem and the latency problem go away because short of having the peers on your Local Area Network, you're going to stress that first-hop. Webseeding on the other hand does eliminate the upload congestion problem but it doesn’t fix the download problem.

Re: Why BitTorrent causes so much latency and how to fix it

If torrents give you high latency, that's not an issue with BT itself, but with your network setup. What you need is at least a shape to make sure outgoing data is limited to a rate and burst size that doesn't clog your modem's outgoing buffer. Modems tend to have huge buffers, and when a P2P client starts uploading, it tends to use up all of it, which means that other data will get stuck. The solution is simple: limit and prioritize data before it hits the modem. This is rather easy on Linux and BSD (see e.g. the Linux advanced Routing & Traffic Control HOWTO, or LARTC) if your torrent client uses the proper traffic classes, or if you're able to properly mark the packets in a firewall rule.

It also helps to just throttle the upstream in your P2P client of choice.

With a rather easy setup (token bucket shaper for high-bandwidth data), I'm able to play stuff like BF2 in parallel with a torrent quite fine. So instead of trying to "fix" the protocol, fix your network.

Of course it is an issue with BitTorrent

Of course it is an issue with BitTorrent. You can mitigate the problem with good traffic engineering, but that's not something that's available to the vast majority of users. What I'm saying is that if it's possible to engineer a P2P application to play nice with EXISTING home networks, then that P2P application will become very popular because most of the existing home networks have zero traffic engineering capability.

"It also helps to just throttle the upstream in your P2P client of choice."

As you can see in my results, I am throttling and it reduces the frequency of the ping spikes but not the magnitude of the spikes. Prioritizing data before it hits the modem is easier said than done. My DD-WRT Buffalo router didn't do anything even though I configured it. I just tried CfosSpeed in Vista which Stefan above recommended and that seems to help mitigate the latency in gaming but I doubt it can help in the situation where the BitTorrent user is on a different computer on the same LAN. I still need to do more detailed testing on it before I draw any conclusions about it.

Re: Why BitTorrent causes so much latency and how to fix it

"It also helps to just throttle the upstream in your P2P client of choice."

I use openSUSE 10.3 and Deluge wherein my upload limit is set to 30KiB/s over a linksys WRT54GL running DD-WRT and maximum ports set to 4096.

Deluge is written in Python and is very fast compared to Azureus. It is configured to use randomized high-end ports with full inbound/outbound encryption with up to 512 maximum connections, just 15 upload slots and Mainline Distributed Hash Tables.

I believe there is a Windows version George you can try for comparative purposes.

For VoIP, I've got a dedicated openSUSE Apache Web/Asterisk PBX v1.4 server proxying to sipphone.com. I have a linksys ATA 2102-NA and a Nokia N95 serving as SIP phone extensions. (The Nokia cellphone being WiFi- and SIP-capable supports inbound/outbound calls 'anywhere' over my AT&T wireless Data bearer plan.)

I have no QoS settings on the WRT54GL, no overclocking, and the ATA router sits behind it.

I've not experienced latency to speak of and on an HP Pavilion AMD Turion64x2 with 2GB RAM running openSUSE can watch movies in MPlayer, BitTorrent, and place VoIP calls concurrently without issues.

Comcast is now taking a conciliatory approach and working with BitTorrent to also work toward improving the P2P protocol indepent of the P4P Work Goup.

Verizon is practicing Network Neutrality and has no plans for traffic shaping to deal with BitTorrent.

Verizon also says that network management is a necessity

"Verizon is practicing Network Neutrality and has no plans for traffic shaping to deal with BitTorrent."

Verizon also says that network management is a necessity. Read their FCC filing.

Also, Verizon and Comcast's video service comes over the same wire but runs on a different optical wavelength or Radio Frequency respectively. FTTH (Fiber to the home such as Verizon FiOS) and CATV operators are doing a hard separation where their TV service cannot be affected negatively by network congestion even when no QoS mechanism is in place. CATV operators also allocate a separate frequency for their phone service. This prevents your Internet connection from using TV or phone bandwidth even when your TV and phone isn’t in use so it essentially prevents you from pooling your resources. Now that isn’t a huge problem for FTTH because of their abundance of bandwidth but it is a huge problem for Cable Broadband. FTTH is also very expensive to deploy and unfortunately it has very limited coverage.

FTTN (Fiber to the Node services like AT&T U-Verse) VDSL providers on the other hand use a single IP network to deliver data and IPTV service and they absolutely must provide assurances to their IPTV service. IPTV service - unlike Video on Demand services - requires low buffering so that it can support "channel surfing" and match CATV and Satellite TV offerings. That means IPTV service must have flow rate assurances that guarantee sufficient bandwidth and guarantee low latency using network prioritization. IPTV consumers signed on for their IPTV service and they expect it to work like any other TV.

IPTV gives consumers an alternative to Cable TV and you wouldn’t have that alternative if you cripple FTTN operator’s ability to deliver a reliable TV service. Because consumers spend way more money on TV services than Internet, one could reasonably argue that TV competition is even more important than Internet competition. IPTV services are also the thing that gives DSL operators the financial incentive to build those expensive VDSL FTTN networks.

Why should Cable operators be allowed to guarantee TV service just because they carved out a fixed amount of resources using frequency division multiplexing? Why shouldn’t DSL operators be permitted to guarantee their IPTV service just because they’re using a dynamic form of Time Division Multiplexing? After all, IPTV allows you to get that extra bandwidth back when you’re not watching TV whereas you can’t get access to the TV frequency spectrum when you’re not watching Cable TV. Besides, it isn’t as if the DSL operator is hogging the entire pipe. They’re just dynamically carving out 8 Mbps of that 20 Mbps pipe to make sure that episode of the Sopranos comes through without a glitch and that you can flip channels on the fly. Now would you rather have the Government force DSL operators to do a hard allocation of resources and say you can't get access to that bandwidth whether you want the IPTV service or not like they do with Cable Broadband?

Furthermore, Google is spending big money to lobby the Government to cripple DSL operator's ability to get in to the TV business and they’re financing Net Stupidity legislation to ban the sale of QoS. Heck you had Vint Cerf in a debate with David Farber bluntly telling AT&T to get out of the TV distribution business. Google is trying to kill off a competitor by using the Government rather than earning it.

These are the real questions that need to be debated Dietrich, not a bunch of quotations from Tim Wu who won’t even debate the legislation when he’s facing someone who understands the technical aspects of the issues. So drop this “all packets are created equal” attitude because you should be smart enough to understand that network engineering is far more complex than that.

Re: Why BitTorrent causes so much latency and how to fix it

I am not worthy. ;)

Re: Why BitTorrent causes so much latency and how to fix it

Thanks Gorge for your BT analysis article, awesome reading and really provides a good insight into the root causes of why I would lag out on CoD4 when I or my brother would be BT'ing.

I've recently picked up a "gaming" router from Dlink with Ubicom's latest StreamEngine QoS (v2.0) chip. I haven't had any lag issues since but the tradeoff is (obviously) slower BT when gaming or running VoIP.

My question is, how does consumer grade QoS such as Streamengine handle BT packet bursting you described in your article? Specifically does it split these big packet chunks in hardware?

Network solutions re-order the packets going out

Network based solutions like Ubicom StreamEngine typically re-order the packets going out to the Internet such that your gaming packets get moved to the front of the line and get transmitted first. I'm not sure exactly how and if the're dealing with the download problem.

Of course, you want to make sure that your brother's BitTorrent application is set to cap bandwidth utilization. While the QoS network hardware may handle this, it's better not to do the damage in the first place.

Re: Why BitTorrent causes so much latency and how to fix it

Sounds like you might be relying on dynamic port opening which doesn't always work. If you want that port to stay open, go to your router interface and manually create a permanent and static port forward. You would need to set your computer to a static internal IP. This has nothing to do with your ISP and it has everything to do with your own equipment. Your ISP isn't blocking those ports; your own router is.
-----------------

No, my router has the ports set. It's not UPnP or any other dynamic setting. In fact, it's limited to a specific IP (and yes, occasionally my internal IP changes, but that's easy enough to fix). As for my router blocking the ports, I'll take your word for it, but I don't see why it'd bock all ports (changing it to a different one had no effect) or why reseting it does nothing to fix it, but ceasing all torrents over a weekend did.


---------------------------
n> "Regardless, i don't think that they need to allow charging for priority. ISPs should be motivated to implement such things, because it will improve their customers service and likely mean they can get more out of their bandwidth. "<<

go>But are you sure you want the Government to impose a no-charge rule? That would pretty much force the bundling of the service so that people who don't need or want it would have to subsidize it for others. It could also mean that the ISP can't justify offering it to everyone so they don't offer it at all. Do you want the Government to force an all-or-nothing situation?<<
---------------------------

No I'm not sure. But isn't other bandwidth/LD generally handled in a way where ISPs trade off? As a result, if google sends tons of data to me, my ISP get's credits from googles ISP? Utimately, I kinda think the ISPs are already charging Google to send/receive data. It seems a bit ridiculous to charge extra to receive the data.

It seems like QOS should be used based on classes of service. IOW, IP Telephony should have a higher priority than Torrent traffic. In fact, Torrents should probably get the lowest priority, while Phone, Video and Gaming should be near the top. Web Browsing should be somewhere in the middle.

But maybe you're right. Maybe we should wait and see what problems occur before we try to solve it. I think it really depends on what happens with a small site operator who wants to host a video blog. How will selling QOS affect that person? My fear is that ISPs will ultimately just tack on more fees and generally raise costs of entry.

Call me skeptical, but as a rule, i don't trust phone companies or cable companies to have MY best interest (and I work for a phone company).

Re: Why BitTorrent causes so much latency and how to fix it

I see what you're saying; BT corp can prevent the problem by doing a little more due-diligence, improve efficiency, and responsible programming to minimize problems at the source. QoS at this point can look very similar to a "band-aid".

Re: Why BitTorrent causes so much latency and how to fix it

Many home routers can perform the QoS function you are looking for on the upstream interface queue, even when the physical connection is 100Mb/s to the modem.

Particular mention must be made for OpenWRT (Kamikaze build) which is Linux-based and uses the Linux kernel traffic shaper (tc) and various queuing algorithms. To apply bandwidth-aware QoS it uses the Hierarchical Fair Service Curve (H-FSC) algorithm which is capable of not just prioritising packets in the queue but also performing the smoothing you are looking for, by controlling both bandwidth and delay. The queues can be told how much upload and download bandwidth there is available, and tweak the transmit queues on your Internet-side and LAN-side interfaces.

Don't get me wrong: QoS in Linux is a minefield of brain-melting material posted all over the Internet and the QoS tools themselves in OpenWRT have little to no documentation, which necessitates that you read some of the aforementioned brain-melting material. However, once a good understanding of QoS capabilities is reached, the possibilities are almost limitless. :-)

Re: Why BitTorrent causes so much latency and how to fix it

pork chop sandwiches?

Re: Why BitTorrent causes so much latency and how to fix it

Thanks George,
I've noticed that the gaming router fixes my network issues, but not my Network Card issues..
That is, when using bit torrent, everyone else can still use the network but my card/stack seems to be congested.
Prevents me from remotely connecting to my pc or browsing while bt is on.

Re: Already a fix

It costs money from that one company. This is showing people what they need to change themselves.

Furthermore, that guy talking about playing movies in mplayer and stuff needs to realize that doesn't matter unless your streaming video over your network. We are talking about unintentional side effects of P2p and torrenting, not throttling.

I'm gonna get much use of this article. Thanks for doing good research for us dummies.

Anti-neutrality propaganda??

There are fundamental technical flaws with article, the data collected, and the analysis of it. The author ignores that it is the number of open connections rather than the application opening them, which us the culprit. Open ten HTTP or ftp downloads and your latency similarly jumps to 6000 ms. The author never tried this, and jumps to ridiculous conclusions from his limited dataset.

But the worst and most dangerous part of the article are the political opinions he offers on the sensitive issue of net neutrality, which might unfortunately get swallowed at face value by the overwhelming majority of folks who lack the technical knowledge to see the argument's flaws.


Giving the big ISP's carte blanche to do anything they want to your traffic seems a high price to pay for the ability of certain people to run certain apps simultaneously on their single home circuits.

Re: Why BitTorrent causes so much latency and how to fix it

If you want to test your proposal for VoIP-friendly bittorrent uploads, why not set the MTU of your Internet-facing interface to around 770 and see if that helps.

It might not benefit you without the upstream packet prioritisation that you would gain from OpenWRT or similar (already mentioned here)

I'd be interested to see the results of that.

Ed, I did try using DD-WRT with QoS and it didn't work

Ed, I did try using DD-WRT with QoS and it didn't work. As I mentioned in the article, it's not that easy to do when you're doing it on a device that doesn't have the mismatch in speed because packets leave on the 100 Mbps LAN interface as soon as they enter. So when you have nothing in the queue, it's kind of hard to prioritize that traffic. It's not that this is impossible to do because there are devices that can help, but my point is that it is easier said than done.

Furthermore, the home router QoS solutions aren't a complete solution even when they are in place. What I mean by is that is that they don't completely eliminate the extra latency caused by BitTorrent or P2P usage. So that is why I'm suggesting that modifying the P2P client to be more latency friendly is always a useful thing.

Paul, it's not just an MTU size, it's the spacing of those packets

Paul, it's not just an MTU size, it's the spacing of those packets. In my final illustration, I show the spacing with the assumption that either the application intelligently spaces its transmissions or that some network-level or TCP stack level mechanism artificially inserts the spacing. So having a bunch of small packets doesn't help if BitTorrent is still releasing large chunks of data on to the network at once because the TCP stack will simply carve that chunk of data in to more packets and you still have a bunch of packets in the queue.

OpenBSD PF and ALTQ

Prioritizing empty TCP ACKs is a really helpful solution.

For an explanation and a how-to go here.
http://www.benzedrine.cx/ackpri.html

With a fast connection and a beefy network, BitTorrent should cause minimal latency

I think your problem may more be related to a slow connection and your router/modem not being able to keep up with the flood of BitTorrent packets or your ISP's network and routing just being poor in general.

I personally don't have any latency problems from BitTorrent on my 20Mbit/20Mbit FTTH connection which is shared via my Linksys RV0041 router (I'm not running any sort of QoS and neither does my ISP).

http://img110.imageshack.us/img110/4586/pingsie2.png

As the above image shows my ping stayed at 1ms (with one brief 2ms ping) while uploading at 3MiB/s (125% of my rated capacity) via BitTorrent. Outside the network my pings stay stable as well with no more than 1% fluctuation while maxing out my upload on BitTorrent.

Alex, this is only true for now in a very narrow case.

Alex, slightly more than 1 million people have access to Verizon FiOS. Most of us don't have access to it.

However, BitTorrent releases relatively small chunks of data on to the network which is insignificant to a 20 Mbps upstream queue. At 4 packets deep at 20 Mbps, that's only 2.3 ms deep in the transmit queue of your FTTH ONT. 4 packets deep at 416 Kbps per second is automatically 112 ms deep in the transmit queue of the DSL modem.

So what this says is that BitTorrent is well tuned for about 1% of the population that has access to Verizon FiOS and it needs to be tuned for the rest of us.

Now does this mean that with FiOS you'll never need QoS? No, it means you don't need it now. But as desktop computers adopt gigabit Ethernet and 20+ Mbps FTTH becomes more popular like in Japan, people will start torrenting multi-gigabyte movies and you can expect the queue depths to reach hundreds of packets or even thousands of packets deep. Then at that point in time, latency becomes an issue again and you still need intelligent traffic engineering techniques. Right now, you as a FiOS user are in a very unique and fortunate point in time that you don't need it.

Re: Why BitTorrent causes so much latency and how to fix it

You rock. I bow to you. I am a huge uTorrent user, so this would certainly help me out.

Re: Why BitTorrent causes so much latency and how to fix it

http://community.smoothwall.org/forum/viewtopic.php?p=164920

Re: Why BitTorrent causes so much latency and how to fix it

Why bother with solving hardware problems that have an easy solution in hardware through software? If you want to run high demand services such as games or VOIP, just get a second internet connection for that. I've done that, works perfectly.

Re: Why BitTorrent causes so much latency and how to fix it

You might look at trickle/trickled. User-level net app shaping. I see it's available in Ubuntu.

Troberg, getting a second Internet connection isn't practical nor always possible

Troberg, getting a second Internet connection isn't practical nor always possible. If anything, people tend to share a single Internet connection with multiple users to save money. This is why we have roommates in the first place; to save money. Most of us have limited money and we deserve a solution.

Re: Why BitTorrent causes so much latency and how to fix it

The reason most ppl can't get OpenWRT working is they don't set upload/download parameters correct. Another problem they use sfq instead of esfq. By default, sfq treats all connections equally. Since BT opens so many connections, it wins. W/ esfq, you can give different connections differnet run time.

Another thing is, you need to control the download very aggressively. W/ my 199KB down link, I had to cap to to 175KB in order to have decent latency control. Basically, I had to sacrafice download throughput for latency control.

I use Linux HSFC for uplink throttle and IMQ+HTB for downlink control and it's almost flawless. Even w/ seeding, my ssh sessions and WOW sessions (almost) run w/o a hitch.

mike

I did set upload/download parameters correctly and I tried both modes.

I set upload bandwidth to 416 Kbps and download bandwidth to 2.4 Mbps which is my sustainable peak throughput and not the advertised signaling rate on my DSL service.

DD-WRT HTB didn't seem to have much effect and HSFC jacked my latency sky high. I prioritized ICMP traffic so that I can use the ping command to test it.

Re: Why BitTorrent causes so much latency and how to fix it

There are a few problems with smaller packets. First of all, the overhead goes way up. Secondly, while many routers can handle a high bit rate in megabits per second, they're much more constrained in the number of PACKETS per second they can handle. BitTorrent with small packets would be likely to max them out.

I did set upload/download parameters correctly and I tried both modes.

I set upload bandwidth to 416 Kbps and download bandwidth to 2.4 Mbps which is my sustainable peak throughput and not the advertised signaling rate on my DSL service.

DD-WRT HTB didn't seem to have much effect and HSFC jacked my latency sky high. I prioritized ICMP traffic so that I can use the ping command to test it.

Re: Why BitTorrent causes so much latency and how to fix it

Just to clarify, the P4P Working Group has 75+ members (ISPs, P2Ps and researchers) and is pursuing a wide range of strategies for ISPs and P2Ps to work better together. One approach, based on Yale's P4P work, is making the p2p network aware of the physical network, so that the p2p network can choose nearby peers instead of distant peers. Initial trials show that network awareness dramatically reduces the delivery cost of the data, and improves delivery speeds, because data that transits within a local loop is much faster and cheaper than data from across the Atlantic Ocean.

But shortening the distance that data transits doesn't (overall) change the volume of data uploaded or downloaded by peers, because in p2p networks the data volume uploaded always equals the data volume downloaded, by definition. If the ISP's edge network is capacity limited, then solutions like caching are also very interesting, because they can reduce the demand on users' uplink. And certainly anything that can help with uplink congestion is very interesting.

We're doing some testing with Comcast, and a number of other carriers, both US and international, and I'm looking forward to learning a lot more about how well localizing network traffic works for them.

Laird Popkin, Co-Chair of P4P Working Group (and CTO, Pando Networks).

Re: Why BitTorrent causes so much latency and how to fix it

Great article.

I saw similar issues with applications that saturate links (heavy file transfers, ftp, bittorrent, streaming video, etc.). I setup snort to detect when I was on a VoIP call (you can substitute this for an online game or other latency-sensitive application), a logfile analysis tool to kick-off a script that limited all TCP to 1/2 the available bandwidth (upstream or downstream) while the latency-sensitive application was running. After the application was done, the bulk file transfer applications could then use 100% of the bandwidth again.

I call this Application Aware Triggered Quality of Service.

More information, detailed how-to, graphs, etc. are available here: http://www.xmission.com/~hidden/aatqos

Re: Why BitTorrent causes so much latency and how to fix it

Here's my qos script I used. It first setup the upload queues, then use iptables to classify the upload pkts. After that, it use setup the download queue using imq + htb and uses iptables to classify them.

Note: your stock linux kernel/iptables probably don't have esfq, layer7 and imq built in. This script also works w/ openwrt kamikaze w/ little modification.

http://www.megaupload.com/?d=AW0UFWRV

mike

notgonnatellya, I didn't suggest that the ISP should be able to charge for offramp access

"No I'm not sure. But isn't other bandwidth/LD generally handled in a way where ISPs trade off? As a result, if google sends tons of data to me, my ISP get's credits from googles ISP? Utimately, I kinda think the ISPs are already charging Google to send/receive data. It seems a bit ridiculous to charge extra to receive the data. "

notgonnatellya, I didn't suggest that the ISP should be able to charge for off-ramp access. In fact, I'm vehemently opposed to that and I slammed AT&T's Ed Whitacre for implying that they might do this http://blogs.zdnet.com/Ou/?p=512&page=2.

Google and every other Internet user pays for on-ramp access to the Internet and the inter-carrier traffic is handled by existing peering arrangements. At no time should Google or any other Internet user, large or small, ever be charged for off-ramp access or face the threat of blockage or degradation. Of course the ISP has a right and duty to throttle bandwidth hogs so that every paying customer gets their fair share of the resources they paid for and your share of the broadband pipe should be proportional (not necessarily linear) to what you pay for.

What I am talking about is that YOU the user should have the right to tell your ISP that you want a particular source or traffic type to be prioritized over other traffic for throughput and/or low-latency. This could either be done by you going to a website and setting these settings yourself or telling your ISP to just do this for you over the phone and prioritize common stuff like VoIP or gaming.

This also means if you voluntarily sign up for IPTV service which requires QoS to function, you are implicitly asking the ISP to prioritize and dynamically carve out 8 Mbps of your 20 Mbps VDSL pipe when you are watching TV. When your IPTV box is in sleep mode or it's turned off, you get that 8 Mbps back whereas your Cable Broadband provider can't give you the bandwidth back when you're not using their Cable TV service even if you don't order their Cable TV service.

What the net neutrality laws try to do is forbid anyone from buying prioritization service PERIOD and I’m saying that this ban on the sale of QoS is wrong and that I oppose it on technical and moral grounds.

George Ou

Mike, I already approved 1 of your 4 repeat comments, thanks!

Users who are not logged in require moderator approval to post comments. Users who are logged in can immediately post comments without moderator approval. Thanks for your comments!

Mike, I already approved 1 of your 4 repeat comments, thanks!

Users who are not logged in require moderator approval to post comments. Users who are logged in can immediately post comments without moderator approval. Thanks for your comments!

Re: Why BitTorrent causes so much latency and how to fix it

Is the well-known Azureus plugin Auto Speed a good fit for a solution?
It adjusts upload and download bandwidth limits according to ping times to a configurable server (usually, your default gateway).
Did you use and discard it?

Re: Why BitTorrent causes so much latency and how to fix it

Mr. Ou,

Your article clearly shows that you know nothing about TCP/IP and your blog entries, in general, show that you are not a technical person.

cFos _is_ the answer

I've been using cFos literally for years now, and this problem doesn't exist for me anymore (first on a PPPoA, then on a PPPoE connection). This software is a must-have for people who share connections or use their computer / connection for many things at a time.

Give it a shot George :)

Re: Why BitTorrent causes so much latency and how to fix it

Did you ever tried edonkey ? Please do and you'll see that Bittorrent is much more Home networking friendly. I'd be curious to see a comparison based on the same test and measurement protocol. Not to say things can't or shouldn't be improved ...

I did try cFos and it seems to help

I did try cFos and it seems to help, but it won't help the external Lingo VoIP ATA box since you can't run cFos on it. Furthermore, you have to purchase/install a bunch of unsigned drivers and you need to install it on every computer on the LAN to make them cooperate with each other. So I'm not sure it's something I want to install and I'm not sure if it's practical to ask your roommate to install it.

Having said that, it did seem to help with the games even when BitTorrent was running. The utilization and latency indicators with the transparent background was cool. So I'm not entirely opposed to the solution.

The free Smoothwall plugin solution Sam mentioned above seems like something worth trying.
http://community.smoothwall.org/forum/viewtopic.php?p=164920

Re: Why BitTorrent causes so much latency and how to fix it

Hint: All bittorrent clients that I've used, starting from the initial one written in Python, DO HAVE an option to limit upload rate.

Next hint: The correct upload rate needs to be figured by the user. The only device in the mix that might have the data to allocate bandwidth automatically is the outbound gateway.

Now, I guess the author should either get himself a brain transplant, or change the industry.

yacc143

Re: Why BitTorrent causes so much latency and how to fix it

Using an aftermarket firmware/etc you can observe the number of active connections. Even though the data only comes to and from a few points, your network equipment will leave old connections open numbering from the 100's to the low 1000's depending on how many peers youve contacted. One possible solution ive seen is to greatly reduce your connection timeouts, but this seems to be impossible with most home router firmwares. (DD-WRT and openwrt firmwares are also good for people looking to learn more about the networking equipment they use)

Here's a hint yacc143, try reading the article next time. I limited the upload to just 10 KB/sec and it STILL caused latency.

Here's a hint yacc143, try reading the article next time. I limited the upload to just 10 KB/sec and it STILL caused latency. My sustainable upload is 52 KB/sec.

Re: Why BitTorrent causes so much latency and how to fix it

I've noticed that utorrent is not very strict with upload limits, it will happily send more KB in a second than the limit according to two bandwidth meters (Windows graph thingy, Everest home), and on two different computers. When there's a download going on as well, the limit somehow becomes a lower limit, not getting below it, instead of not going above it. So my guess bittorent uses the worst possible combination for a stable internet connection.

The proper way to fix it

It's NOT a problem with Bittorrent. The problem is at _your_ end. Really.

You don't have access to the modem, but you have access to the router.

Configure the router to assume the uplink is 400kbps or lower. Pick a value slightly less than your actual uplink speed - actual not what the ISP claims they give you. Basically you saturate the link, then reduce the router uplink speed till your QOS stuff starts working properly. If your router can't do that, get one that can.

The narrowest part of the straw controls the flow. So make your router the narrow point.

If a 20+ms (the time it takes to send one 1500 byte packet out your uplink) latency hit is too high, then you will have to get your router to assume the wan uplink can only support say a max packet size of 500 bytes or something small. Your router must also be able to insert online game packets between the 500 byte fragments of the large packets. With this your max latency hit will be reduced to a third. Naturally this will hurt P2P and HTTP download throughput, and cause other problems (paranoid/clueless sites/people might drop IP fragments etc). So I don't recommend this last bit - it's only for people who are desperate and know what they are doing.

Re: Why BitTorrent causes so much latency and how to fix it

I stand by the former comment that routers around the world (not just home routers) would get CPU load problems if all the packets in bandwidth-heavy transmission suddenly were cut to half size. This would mean twice as many routing decisions per time unit, and could clog up networks around the world worse than the initial problem you're describing.

Already did that on DD-WRT link

"Configure the router to assume the uplink is 400kbps or lower. Pick a value slightly less than your actual uplink speed - actual not what the ISP claims they give you"

Already did that on DD-WRT link. Didn't seem to work.

"It's NOT a problem with Bittorrent. The problem is at _your_ end. Really."

Why assume that? Why assume that only a network based solution can fix this? I can certain agree that a good network based QoS solution can fix this, but why must that be the only way? I'm saying that it's possible to fix this on the application side and that could serve a marge larger market share of people who don't have network based solutions in place.

MortenJ, smaller packets isn't the only solution.

P2P applications could just space out their 1472 or 1464 byte packets evenly and that would help a lot. As for CPU loads, I believe the existing router infrastructure should be able to handle it. With the growth of BGP tables, routers are constantly being upgraded.

Re: Why BitTorrent causes so much latency and how to fix it

If you have 1024kbs download and 512ksb upload bandwith,

doing 50kb/s download causes the the same latency than doing 25kb/s upload. When combined (uplink and downlink) the latency is more.

math. :D

from my expirience, uploading while gaming causes pl and latency *

Re: Why BitTorrent causes so much latency and how to fix it

What you're chasing is H-FSC. Build a Net or Free BSD box and insert it between your switch and router.

http://www.cs.cmu.edu/~hzhang/HFSC/

Already tried HFSC in DD-WRT, it did something but actually made things worse

Already tried HFSC in DD-WRT, it did something but actually made things worse

Re: Why BitTorrent causes so much latency and how to fix it

Not mentioned here as of yet is the excellent pfsense (http://pfsense.org) which utilizes openbsd's pf. Their QoS may have something to offer in this situation.

Re: Why BitTorrent causes so much latency and how to fix it

We all know that once P2P gets going on one computer, seems that on a second computer - the processor feels like it even takes a hit.

I am a network guy and need my home network for work testing. I mean sometimes I need to log into my home computers and hit my work network and test OWA, web sites, and ftp sites. I have to log in and shutdown Azureus in-order to work.

Thanks George, for pointing this out to the community as we all exist to make technology better and to make it work with as little resources as possible. Otherwise, multi-tasking is pretty much obsolete.

Re: Why BitTorrent causes so much latency and how to fix it

The issue here can be explained simply by a well-known queueing theory idea, captured as "Little's Theorem" or "Little's Lemma".

If you have a queue that can handle requests (packets) at a fixed "service rate", the queue length as a function of "request rate" grows asymptotically to infinity at the point where the request rate = service rate. (this is strictly true for poisson arrivals of requests, but many random arrival processes cause the same behavior).

That is, trying to use up the full capacity of a router, link, etc. causes a queue to build to infinity. Since buffers are only finite, this will cause dropped packets, and long latencies.

The solution is to avoid behavior that fills the queues. The KEY IDEA is that the ONLY reason that queues should ever be a "good thing" is if there are random gaps on the outgoing link that could be filled if there were elements in the queue. This gives better throughput, but longer latency. Thus there is a REAL tradeoff - throughput gets better only when latency gets long - a decreasing returns situation. In most cases, having an average queue length of *one* is the best state for a bottleneck router to be in. The observation you see is that the modem/router does not complain until a HUGE queue backs up, and then it drops packets.

Two proposals have been made in IETF congestion control groups to fix this: RED (random early drops) and ECN (early congestion notification). Sally Floyd and Van Jacobsen showed how they work with thorough analysis.

TCP stacks that support ECN are available, but they are not turned on in most cases by default. RED does not require any modification of endpoints, but does require router support. You can fix this by implementing RED in your router today.

There is some interesting background as to why router vendors are not eager to do either ECN or RED. In general, router vendors get measured on *throughput* not on latency. Thus, in "bake offs" such as those sponsored by folks like the computer press in reviews, a system that implemented ECN or RED would *lose* by a few percent in comparative benchmarks.

The media (George - this means you) could make a difference by rating routers (home routers) on latency in overload cases. A linux PC based router implementing ECN and RED could be used
as a benchmark. This would move the industry to helping with latency, rather than meaningless hot-rod-engine-revving comparisons.

Re: Why BitTorrent causes so much latency and how to fix it

Queueing is not something you can just 'turn on' and have it work :( I'm not usre how much configuration you gave queueing before, but you could try as below. The beneift of HFSC is that it decouples latency and bandwidth. Normally a low bandwidth queue (such as voice) gets starved by a higher bandwidth queue. THe HFSC latency priority 'realtime' essentially allows your voice packets to jump the queue up to their configured bandwidth limit during contention. I note the packet size issue you mentioned above as well - you could try changing your voice packetization from 20ms to 30ms which will make it slightly less sensitive to jitter at the expense of greater susceptability to packet loss.

HSFC gives you 5 latency priorities to go with your bandwidth definitions. If you can give it another go, ensure that your voice traffic is marked EF (DSCP 46) from the device generating the voice traffic (ATA, IP handset etc - most will allow you to set this). Set your link bandwidth according to your upstream sync speed (eg 420kbps). Define EF traffic to get realtime priority on your HFSC queue and define the minimum guarantee according to your codec (eg 10% or 42kbps for G.729) and typical concurrent call limit. This should help your outbound stream (People hearing you). There's not a heck of a lot you can do for the incoming stream except to really try to limit the torrent download to keep bandwidth available for voice. Throttle your torrent client download bandwidth and limit the number of connections per torrent and / globally. This should help to smooth out your individual connections and soften any peaks.

It's not the job of the application to queue the packets - thats normally the job of the edge device as its only the edge device that has knowledge of all traffic from all applications and clients on the LAN sending out to the WAN. The router is ideally placed to queue and decide in what order the packets should go out to the WAN based on some means of classification of traffic (in this case DSCP values).

And how many home users have good QoS routers or know how to configure QoS?

"It's not the job of the application to queue the packets - thats normally the job of the edge device as its only the edge device that has knowledge of all traffic from all applications and clients on the LAN sending out to the WAN."

And how many home users have good QoS routers or know how to configure QoS? I would submit that it's lower than a few percent. That being the case, there is a real market out there where people are so fed up with bad house mates that they want to take a baseball bat to their roommate's computer. What do we do about this mass market? Tell them to buy new and expensive router gear and/or learn about poorly documented and unfriendly Linux projects and learn how to be a network engineer?

BitTorrent or P2P applications are a very competitive field. The question is whether there is a demand for a network latency friendly P2P client or not and I would suggest there is a huge demand.

Agreed David

David Reed says: "There is some interesting background as to why router vendors are not eager to do either ECN or RED. In general, router vendors get measured on *throughput* not on latency. Thus, in "bake offs" such as those sponsored by folks like the computer press in reviews, a system that implemented ECN or RED would *lose* by a few percent in comparative benchmarks.

The media (George - this means you) could make a difference by rating routers (home routers) on latency in overload cases. A linux PC based router implementing ECN and RED could be used
as a benchmark. This would move the industry to helping with latency, rather than meaningless hot-rod-engine-revving comparisons."

I agree with you David. This would be a much better way to benchmark.

Re: Why BitTorrent causes so much latency and how to fix it

I stopped reading when I got to "Unfortunately, these types of QoS services may never see the light of day if poorly conceived Net Neutrality legislation gets passed that ban the sale of packet prioritization." By that time I had already come to the conclusion that you do not understand the technical details behind your arguments, and that statement convinced me that you don't understand what Network Neutrality is about either. This quote from the Net Neutrality bill currently in the House of Commons here in Canada does a very good job of summarizing what it is.

"Network operators shall not engage in network management practices that favour, degrade or prioritize any content, application or service transmitted over a broadband network based on their source, ownership or destination."

Notice that it does NOT attempt to unilaterally ban all network management - in fact it has exceptions specifically to allow prioritization of packets for things like VOIP, IPTV, etc. (it does not list every exception, so that future latency-sensitive protocols may also be prioritized).

With your anti-net-neutrality point of view, I can't help but think you're in bed with the big telecom companies somehow. Shame that you're fattening your own wallet by giving out bad advice to your readers.

Re: Why BitTorrent causes so much latency and how to fix it

I see there's a lot of discussion on this already, but i thought i'd add my 2c anyway.

Using OpenWRT w/ X-Wrt admin interface did wonders for HTTP and Skype latency for me, and was very easy to set up. Here's a screenshot: http://wiki.x-wrt.org/index.php/Image:Wrp_network-qos.png

Just make sure you limit the bandwidth to a value below the actual speed so that the queue on the DSLAM never fills up.

Re: Why BitTorrent causes so much latency and how to fix it

Rob says: "I stopped reading when I got to "Unfortunately, these types of QoS services may never see the light of day if poorly conceived Net Neutrality legislation gets passed that ban the sale of packet prioritization." By that time I had already come to the conclusion that you do not understand the technical details behind your arguments, and that statement convinced me that you don't understand what Network Neutrality is about either."

Rob, I think you stopped reading much sooner than this. Sounds like you stopped reading before you actually read the Net Neutrality legislation. So let me make it easy for you, I'll give you the actual text right here.


Markey proposal (2006):
SECTION 201. NETWORK NEUTRALITY.
(b) IN GENERAL.—Each broadband network provider has the duty—

(3) if the provider prioritizes or offers enhanced quality of service to data of a particular type, to prioritize or offer enhanced quality of service to all data of that type (regardless of the origin of such data) without imposing a surcharge or other consideration for such prioritization or enhanced quality of service;


Snowe-Dorgan proposal (2006):
(5) only prioritize content, applications, or services accessed by a user that is made available via the Internet within the network of such broadband service provider based on the type of content, applications, or services and the level of service purchased by the user, without charge for such prioritization;


When I said "ban the sale of packet prioritization", I said it based on what the facts and what the actual legislation says. So Rob, try and read a little more than you flame out of sheer ignorance.

Rob says: ""Network operators shall not engage in network management practices that favour, degrade or prioritize any content, application or service transmitted over a broadband network based on their source, ownership or destination."

Notice that it does NOT attempt to unilaterally ban all network management - in fact it has exceptions specifically to allow prioritization of packets for things like VOIP, IPTV, etc. (it does not list every exception, so that future latency-sensitive protocols may also be prioritized)."

Um Rob, did you notice that YOUR OWN cited text says that you can't prioritize applications? It's also interesting that you claim their are exceptions for "VoIP or IPTV, etc" when there is no such language and you fail to cite any actual text. You are straight up misleading the public on the legislation when you claim these exceptions.

The only exception in some of those Net Neutrality bills are for 911 emergency services. But how does the ISP determine that you're calling 911? Even "DPI" Deep Packet Inspection doesn't go as deep as the application layer to figure out what phone number you're dialing. Now you want the Government to mandate that the ISP dig deeper in to your packets to determine what phone number you're dialing before you get to prioritize the VoIP traffic?

No Rob, you failed miserably.

Re: Why BitTorrent causes so much latency and how to fix it

I have a Linux based router. Check out http://ipcop.org/, It has an easy traffic shaper tool to help prioritize traffic.

You don't need much speed to route and prioritize . I'm running my router on a 233Mhz and 64mb of RAM.

I've used IPCop, but haven't tried the QoS feature in it yet

I've used IPCop, but haven't tried the QoS feature in it yet. I may have to dig that box out again to test it. The problem is that when most people talk about a particular QoS solution, have they actually done a detailed quantitative analysis on the before and after?

Re: Why BitTorrent causes so much latency and how to fix it

George, great post! But I have a question. I've observed an effect on my cable modem that when I do a bandwidth unrestricted upload (like your ftp upload, but usually I'm using ssh), ping times can go through the roof, much worse than in the tests you ran. This seems to be because of an issue you have not discussed: many cable modems have large (I would say vastly overprovisioned) uplink queues, and most tcp congestion control (Reno) only backs off when packets are dropped. As a result, ping times can get enormous because the queues have much much more than the ideal one packet mentioned by David Reed, and so the ICMP packets wait a long time behind queued tcp packets. However, your test did not exhibit this phenomenon. Do you know why? I would speculate that either your DSL modem is not overprovisioned with buffers, or you're using a more modern tcp congestion control algorithm.

Re: Why BitTorrent causes so much latency and how to fix it

BT can be a pain in the rear end but its all we got so we just gotta deal with it. Either that or go back to actually buying CDs and games and nobody wants that now do they? LOL

JT
http://www.Ultimate-Anonymity.com

Re: Why BitTorrent causes so much latency and how to fix it

I had high latency while bit torrent was running. So much so that I basically couldn't even browse the web. It helped to throttle my upload to almost nothing, but it seemed pretty ridiculous for not even coming close to my potential bandwidth. I tried changing settings on my home router and my torrent client, upping the buffer sizes in my os and the max connection limit. Nothing really had an impact, until I swapped out my home router for an old box running linux setup as a router with iptables. I can now just about max my connection speed with torrents, and have an increase of about ~10ms latency pinging various sites. I can browse the web and even play games with torrents going with no noticeable latency problems. Sorry, I don't use VOIP, so I can't speak to that.

Re: Why BitTorrent causes so much latency and how to fix it

QoS.... duh

Re: Why BitTorrent causes so much latency and how to fix it

What about TCP vs UDP issues? This might simply be a matter of changing your torrent client over to TCP so that it isn't clogging up the channel with spastic packets. This may reduce your roommates download rates a little, but it should decrease the flooding that's probably causing your latency.

Re: Why BitTorrent causes so much latency and how to fix it

I looked into it, IPCOP uses something called WonderShaper. I couldn't find any analysis/benchmarks using it. I remember doing a half-ass test when I first enabled it and notice a huge improvement in my uploading, but I didn't test latency issues.

link: http://lartc.org/wondershaper/

Re: Why BitTorrent causes so much latency and how to fix it

Not capping your upload speed is the biggest reason others on the internal network experience problems.

[Moderator note from George Ou] Try and at least look at the pretty little graphs next time.

Re: Why BitTorrent causes so much latency and how to fix it

DLSAM connectivity is a Layer 1 OSI reference model technology. You cannot apply Layer 3 OSI QoS markings to a Layer 1 technology. Layer 1 (physical) is terminated at the DSLAM, Layer 2 is carried to the aggregation router, and out to the internet on a provider router.

The DSL connection, depending on the type of connection, may already be doing some of the proposed packet jumbling. If the connection is PPP, the network is perhaps already chopping up all the packets into neat ATM cells instead of packets for physical transmission in 53 byte cells, (ideal for QoS already!)

Article had great intent, but some of the information is wrong.

[Moderator response from George Ou]
First of all, the DSLAM is a Layer 2 device
Second, DSLAMs do support QoS technologies like DiffServ and priority queues
Third, ATM cell sizes used in DSL doesn't change the order of the packets at layer 3

Re: Why BitTorrent causes so much latency and how to fix it

I didn't read this anywhere in replies or in your article yet, but you do know that P2P traffic is not the same at HTTPD traffic? You do realize that P2P connection number in the hundreds, that stay open, where httpd traffic is "socket open/ socket send / socket close"..

It's not the traffic that is killing you alone, it's the fact that your "home" router cannot handle hunderds of open and concurrent connections at a time..

Lession of the day... get a Cisco PIX or better hardware for routing and you wont have this issue.

Re: Why BitTorrent causes so much latency and how to fix it

The problem with P4P is that it still allows content providers to set up servers on ISPs' networks without permission and without compensation. This is simply a nonstarter for any ISP. As for the comment that this article is "anti-neutrality propaganda:" that's just silly. What George has ably demonstrated is that it is BitTorrent that is non-neutral, and that throttling or blocking it is an effort to RESTORE neutrality, not to harm it.

Finally, as for David Reed's comment above: I think that it's very interesting that he suggests ECN as a solution, because ECN violates the "end-to-endian" principle. (It requires intelligence in the middle.) But I'm for anything that intelligently deals with congestion. The harder problem is the economics: Content providers wanting to take ISPs' bandwidth without paying, and users wanting every last bit they can hog.

Re: Why BitTorrent causes so much latency and how to fix it

I looked into it, IPCOP uses something called WonderShaper. I couldn't find any analysis/benchmarks using it. I remember doing a half-ass test when I first enabled it and notice a huge improvement in my uploading, but I didn't test latency issues.

link: http://lartc.org/wondershaper/

Agreeing with David Reed

I more or less agree with Prof. Reed regarding RED and ECN. Microsoft has finally implemented ECN in Vista, but has it turned off by default because they fear old-timey Internet routers will choke on packets with ECN code-points set to non-zero values. ECN uses the two bits in the old TOS field that aren't used by DSCP. Home router companies can reverse Microsoft's recalcitrance by doing ECN in their NATs, fortunately. This illustrates the trouble with End-to-End architecture, of course: it means, in effect, that Microsoft controls the Internet (because it clearly controls the end points.) But middleware comes to the rescue, and allows consumers to have some choice.

Regarding RED, it's my understanding that routers do tend to implement it, since it avoids the cycling problems that are the inevitable result of Jacobson's Kludge. They also look inside packets in order to avoid discarding UDP and such things as TCP ACKs, because the first don't care about Jacobson and discarding the second increases congestion.

The Intelligent Network is looking better all the time.

Re: Why BitTorrent causes so much latency and how to fix it

Props to the author for trying to figure out a way to fix this. People may not want to hear this, but if you find a solution for this problem, market it. Just remember to sell it cheap or people will reject it. Thanks for your work, I appreciate the time you have put into this!

Re: Why BitTorrent causes so much latency and how to fix it

So you're against Net Neutrality? You want others to price you out of home access? You don't need the provider to prioritize the traffic you are sending it. How about you don't step on your own traffic via one hop QoS at your egress point? This and TCP are 99% of the puzzle. If your VoIP sounded great before and you added P2P, guess what? You just need to make sure the VoIP is prioritized at your node, the whole internet does not need to be divided up. That's what leased services are for, and you can contract out what traffic you want prioritized and exactly how.

Informative overall, but please don't knock net neutrality with this simplistic an argument. It won't even help you, so please take that off your page.

Also, while you're editing this, you spelled their incorrectly. It should be "there" -> "So despite the fact that *their* is plenty of remaining bandwidth"

Re: Why BitTorrent causes so much latency and how to fix it

Easy answer. One connection vs multiple connections. While the download is the same, the network card works harder and has more to manage for the same effect.

Re: Why BitTorrent causes so much latency and how to fix it

I agree with ruggy and noname -- I think the author's understanding of the issue is weak. I also agree with leongmz -- most people who set up router-based QoS and don't see any difference don't have their settings right.

For most router-based QoS schemes, you want to set your bandwidth at 10% less than your actual (not advertised) speed. Author, if you can't get DD-WRT or some similar software to fix your problem, you're doing it wrong. I can have two people torrenting and a skype call will never skip a beat. Your admission that software could fix the problem but that you weren't satisfied with that solution betrays your ultimate purpose -- to turn people against network neutrality.

I agree with you that any bill that would block packet prioritization carte blanche is a bad one, but the core tenet of network neutrality -- that networks shouldn't discriminate against traffic because of its source or destination -- is a good one. Just because some members of congress can craft a bad bill out of a good idea doesn't change anything and shouldn't surprise anyone.

Back to the article... the reason one HTTP download doesn't have the same negative affect on latency as bittorrent is not because it's using smaller packets. A single TCP connection -- with the built-in flow-control of TCP -- will throttle itself to leave room for other traffic. Multiple connections constantly being negotiated, started, and stopped will lead to spikes in latency regardless of the application creating them. The solution is to implement multiple priority queues for outgoing traffic on the gateway router. No ISP QoS or revamping of the bittorrent protocol is required.

John, before you make such simplistic and kneejerk comments, read what I actually say about net neutrality.

John says "So you're against Net Neutrality? You want others to price you out of home access?"

John, before you make such simplistic and kneejerk comments, read what I actually say about net neutrality.

A rational debate on Net Neutrality | George Ou | ZDNet.com
http://blogs.zdnet.com/Ou/?p=512

Revived Net Neutrality bill cripples Internet for real-time applications
http://www.formortals.com/Home/tabid/36/EntryID/34/Default.aspx

John says: "You don't need the provider to prioritize the traffic you are sending it. How about you don't step on your own traffic via one hop QoS at your egress point?"

Oh you're a genius John. Let's stop BitTorrent usage altogether while we use VoIP or online gaming. Or lets just take a baseball bat to our roommate or brother's computer. Or let’s just buy a second DSL line. No John, we want the Internet to work for everyone simultaneously and we want all the applications to coexist in peace and every applications flow as quickly or problem free as possible. This is why we have traffic engineering.

I see a marketing opportunity

Maybe BitTorrent Inc. should start certifying routers with good QoS support. As we can see from this thread, some of them work better than others.

BTW, to all the people suggesting Linux/BSD: This site is called Technology FOR MORTALS. I'm a Linux hacker and I don't even use a Linux router.

Wes, I don't think that telling the customer to buy new hardware or software is ever a good solution

Wes, I don't think that telling the customer to buy new hardware or software is ever a good solution. VoIP vendors tried to get the network changed for years with little success. Skype came along and built an application with non-conventional approaches and made their VoIP software "just work".

The vast majority of customers won't buy the high-end gear and they don't understand traffic engineering. The real opportunity here is for a P2P client vendor to write their software in such a way to be friendlier to the network rather than asking the network to make changes for the software.

Again, this isn't to say that I'm against network-based solutions at all. I just think that there is room for an application-based solution.

Wes, I don't think that telling the customer to buy new hardware or software is ever a good solution

Wes, I don't think that telling the customer to buy new hardware or software is ever a good solution. VoIP vendors tried to get the network changed for years with little success. Skype came along and built an application with non-conventional approaches and made their VoIP software "just work".

The vast majority of customers won't buy the high-end gear and they don't understand traffic engineering. The real opportunity here is for a P2P client vendor to write their software in such a way to be friendlier to the network rather than asking the network to make changes for the software.

Again, this isn't to say that I'm against network-based solutions at all. I just think that there is room for an application-based solution.

Re: Why BitTorrent causes so much latency and how to fix it

Really now? Straight out of Cisco's CCNP learning exam material why does it say that DSLAMs are a Layer 1 technology? You can feel free to argue with all the CCIEs that wrote the book if you'd like.....but again you are completely wrong.

TOS/COS bits are the only thing you have to mark a packet with at Layer 2. You can also fragment and interleave frames at layer 2. If you cared to listen, I'd explain that serialization delay can cause your VoIP packets to be dropped before being sent if you have giant data packets plugging your lines...but I think I am wasting my time explaining reality.

A DSLAM is pretty much like a Switch operating at layer 2.

A DSLAM is pretty much like an Ethernet Switch operating at layer 2 when we're talking about it in the general context. Like the Ethernet Switch which supports multiple copper CAT-5 ports, the DSLAM connects and switches multiple copper phone-line ports. Ethernet switches support QoS and so do DSLAMs. Most newer switches have routing support but that's beyond the classical definition of a switch. 802.11 Access Points for example are also layer 2 devices in general.

Sure, there are layer 1 technologies in a Switch and DSLAM, but they at the very least go up to layer 2. A device that operates purely at layer 1 for example is an optical transceiver.

And as a response to those people questioning my technical knowledge, I built and designed very large network for Fortune 100 companies.

Re: The proper way to fix it

"Already did that on DD-WRT link. Didn't seem to work."

You should get it to prioritize your gaming and other latency sensitive packets over everything else (in addition to the uplink rate limit mentioned earlier). It will work, you will not have (as you say) 3 torrent packets in front of your game packet at the modem. The router will be assuming a slow uplink, so packets will be queued up at the _router_ not the modem. And if configured correctly when your game packet arrives while the 3 torrent packets are queued up at the _router_ the router will send your game packet out _first_.

As long as the router is the narrowest straw, there will not be a long queue of packets at the modem (unless there is a problem with the uplink).

"Why assume that? Why assume that only a network based solution can fix this? I can certain agree that a good network based QoS solution can fix this, but why must that be the only way?"

Because your stated problem scenario was multiple users. It is easier to fix it at a single device than to fix it on multiple different users using multiple and disparate applications. If your scenario is actually single user then the solution is the trivial one that the semi literate here keep suggesting to you ;).

Even if you manage to convince Bittorent et all to change the way they do things, and manage to get ALL the users (behind the modem) to use the same client and also configure them "properly" (rate limit as per the uplink and max number of users), the users' bittorent clients won't be aware of each other. Every so often they will end up sending their packets at about the same time, even though they are putting gaps in them, and those packets will end up queued at the modem in front of your gaming packets.

I doubt it will be easier to get all P2P clients behind the modem to cooperate and synchronize their packet sending than to fix it at the network device end. If you want efficiency you would even need the clients to rate limit depending on how many active clients there are - otherwise your torrent uploads will be really slow even if you are the only user online.

Better to fix it the proper way :).

As for network neutrality (or not). I think one should aim for fairness and truth. If people don't know what's fair and true, then they should not be in charge of maintaining it. Seems there's so much greed, deceit and deception involved, so good luck to everyone.

Re: Why BitTorrent causes so much latency and how to fix it

Our ISP uses GRED for most traffic management, because it interacts well with Van Jacobsen's awful hack (er, I mean his AIMD algorithm). However, any variant of RED drives UDP-based VoIP crazy as you reach the limit, so we bypass GRED for packets that are clearly VoIP.

Re: Why BitTorrent causes so much latency and how to fix it

Good research. This is old news to me, although more from a casual measurement perspective. I solved the latency issue by using TorrentFlux on a dedicated server in a colo facility and just downloading files as they complete with an rsync script.

Re: Why BitTorrent causes so much latency and how to fix it

--------------------------------------
What I am talking about is that YOU the user should have the right to tell your ISP that you want a particular source or traffic type to be prioritized over other traffic for throughput and/or low-latency. This could either be done by you going to a website and setting these settings yourself or telling your ISP to just do this for you over the phone and prioritize common stuff like VoIP or gaming.

This also means if you voluntarily sign up for IPTV service which requires QoS to function, you are implicitly asking the ISP to prioritize and dynamically carve out 8 Mbps of your 20 Mbps VDSL pipe when you are watching TV. When your IPTV box is in sleep mode or it's turned off, you get that 8 Mbps back whereas your Cable Broadband provider can't give you the bandwidth back when you're not using their Cable TV service even if you don't order their Cable TV service.

What the net neutrality laws try to do is forbid anyone from buying prioritization service PERIOD and I’m saying that this ban on the sale of QoS is wrong and that I oppose it on technical and moral grounds.
---------------------------------------------

George, I've argued with on you on this before, and this is the first time I've heard you make this claim....or perhaps I should say that it's the first time that it's clear to me that you and I likely agree on this issue. To me, net neutrality has always been about preventing them from charging the content providers

So with that said, why would Google oppose this them offering (for free or fee) QOS services to the the subscriber?

Google wants that prioritization service to be provided for everyone or no one, which effectively means no one

"So with that said, why would Google oppose this them offering (for free or fee) QOS services to the the subscriber?"

You'd have to ask Google for their exact motivations. When I ask them in public forums and debates, Google pretty much back away from it just like Tim Wu refuses to talk about the actual legislation. Then you have guys like Rob who simply lie about what Net Neutrality legislation is claiming that there are exceptions for IPTV and VoIP traffic when there is no such thing. The Net Neutrality advocates don't want to debate on the technical merits of their legislation because they know they can't defend it. So their strategy is to represent the issue as a free speech issue and never debate the actual technology.

Google wants that prioritization service to be provided for everyone or no one, which effectively means no one. When you prioritize everyone, you are in effect prioritizing no one. Even if certain types of application prioritization is allowed provided that it's given to every customer, that still means that those who don't need prioritization (don't game or VoIP) would subsidize the bundled service for those who do need the prioritization.

Google is getting in to the video delivery business and they want every advantage they can get. If that means they can kill a competitor like AT&T or any FTTN IPTV provider by lobbying, they will do it and that's precisely what they're doing. Google's Vint Cerf pretty much tells AT&T that they should get out of the video business. But the reality is that you can't compare Video on Demand services which can be buffered to IPTV services which have to be real-time. The FTTN provider's only incentive to spend billions of dollars on FTTN infrastructure is to be able to compete against Cable TV and Satellite TV.

Google can talk about how wonderful a big dumb pipe is but they won’t spend a dime building it. Google gets to monetize their own technology that they invested in and they data mine people’s email accounts making them one of the richest corporations in the world and they’ll fight legislation that regulates them. Their CEOs pay lip service to symbolic hybrid cars gladly while they fly around in the world’s biggest private jet with king sized beds and they talk about doing no evil while they cooperate with the Chinese government. Here's an example of what I'm talking about.

http://online.wsj.com/public/article/SB115222788536400097-i72SXBBTMX_EPvtfDIn9uNjtiss_20070707.html
“Mr. Jennings says Messrs. Brin and Page "had some strange requests," including hammocks hung from the ceiling of the plane. At one point he witnessed a dispute between them over whether Mr. Brin should have a "California king" size bed, he says. Mr. Jennings says Mr. Schmidt stepped in to resolve that by saying, "Sergey, you can have whatever bed you want in your room; Larry, you can have whatever kind of bed you want in your bedroom. Let's move on." Mr. Jennings says Mr. Schmidt at another point told him, "It's a party airplane."”

Re: Why BitTorrent causes so much latency and how to fix it

I'm no gray-bearded networking expert, but the whole idea of using small and evenly-spaced packets to reduce jitter sounds a lot like reinventing ATM's cells and adding time-division multiplexing. How well could this really work with many (maybe hundreds) of TCP connections? Suddenly you either need to perfectly sync all your peer activity, or put up with massively reduced bandwidth. And you can forget managing downstream bandwidth this way, there is no way latencies will stay consistent enough.

This also seems like a gnarly layering violation. Since when is it the job of the application to micro-manage the TCP stack? Microsoft did something similar by allowing its media layer to throttle the networking layer (to keep HD content playback predictable by way of reducing interrupt latency, IIRC), and I think everyone remembers the reaction when some Gigabit users realized they couldn't even use 30% of the link...

QoS could be much easier for the average user to setup, I agree. However, the state of the art is pretty well advanced and works well for many. Just ask any university who can cram thousands and thousands of individual users onto an undersized link, and keep latencies low enough for VoIP, gaming, etc. (Sustained download speeds are awful, of course, but the sky doesn't fall.)

Maybe I just don't get it...

Nick, ATM QoS works within the context of ATM; it does not work in the context of running IP over ATM.

Nick, ATM QoS works within the context of ATM; it does not work in the context of running IP over ATM. Multiple ATM cells make up a single IP packet and you can't just go reshuffling ATM packets to see an effect on IP flows. You need to intelligently shuffle the IP packets either at the network end or you try to engineer the application to spit out an even flow of periodic traffic.

Universities have hundreds of megabits per second and they do manage their flows with very expensive flow-management gear and/or they limit BitTorrent traffic. Their hundred megabit links are much better suited for the students hundred Mbps LAN clients and the queueing dynamics is much different.

Re: Why BitTorrent causes so much latency and how to fix it

Try setting up QoS on both WAN and LAN ports on your "router" device this has definetly fixed the issue for me. I am using the open source firmware for a linksys wrt54g called dd-wrt. Prioritizing http, xboxlive and counterstrike ports has fixed the issue for me, ive done hours of testing and the gameplay and web browsing usage has dramatically changed over non QoS prioritization. Great article, i agree this should be fixed at the application layer not at the networking layer.

I did use DD-WRT with a Buffalo router and the QoS latency gains were questionable

I did use DD-WRT with a Buffalo router and the QoS latency gains were questionable. I prioritized the ports for Team Fortress 2 (same as Counter Strike) and for ICMP (for ping testing) and it didn't do anything for the ping tests and the game play still sucked.

RE:

I think the best program is miu torrent :)

regards: http://ooyes.net

Re: Why BitTorrent causes so much latency and how to fix it

I must say that your analysis is flawed at best, politically motivated at worst.

First of all: TCP is stream-based from the application perspective, while you can control packet scheduling and sizing to a certain extent by disabling nagle's algorithm that's an ugly hack. TCP is a stream and it's the responsibility of the whole network stack and the routing equipment to take care of fair scheduling. Technically speaking it's a no-brainer to give priority to outgoing, small-sized packets, either at the OS or at the router level. You shouldn't blame bittorrent for it just because OS or router manufacturers don't implement it.

Decreasing packet sizes for a bulk transfer is also horrible from a network engineering perspective, you get the maximum efficiency out of a latency-insensitive bulk transfer if the packet size is maximized. That's the one point of TCP ... it auto-tunes itself to the MTU.

And your take on network neutrality is also wrong, most aims would not prevent QoS (which we're talking about here) but only discriminating equal services based on origin, destination or protocol being used. E.g. a comcast video store vs. a bittorrent video store. Or vonage VOIP vs. Skype.
QoS to prioritize vonage AND skype over the bittorrent/comcast video store would still be allowed.

You are wrong 8472

8472 says: "First of all: TCP is stream-based from the application perspective, while you can control packet scheduling and sizing to a certain extent by disabling nagle's algorithm that's an ugly hack."

So you admit that it is possible for the application to trickle data to the TCP stack so you are basically admitting that I am technically correct.

I'm not so much "blaming BitTorrent"; I'm saying there's an opportunity to tweak the application in such a way that it could address the vast majority of consumers who don't have a network-based solution in place and who aren't network engineers. For the longest time in the VoIP world, VoIP vendors expected the network to be re-engineered to support Voice. Skype came along and made the application support the network and made it "just work" and it took off like wildfire. So perhaps there is a lesson to be learned here.

The "too many packets" argument is overblown. Most people have the wrong MTU set for their DSL PPPoE broadband service; it's the default 1500 BYTE instead of the optimized 1492. That means the packets are being fragmented anyways so the 815 BYTE packets isn't so bad. Furthermore, the number 815-BYTE is only needed for slower uplink connections and you'll be able to use optimized 1464-BYTE packets at 50 PPS if your uplink is faster.

You're so busy criticizing me for my "political motivations" that you don't even bother to read the legislation. I've cut-pasted the exact language of the regulation in a comment above and I'll paste it here again for you. It absolutely does ban the SALE of QoS services. So if you offer it to one customer you have to offer it to every customer which is the forced bundling of services. That means those who don't need QoS would be forced to subsidize those who do. Either that or the ISP simply decides that not enough people want it and they give it to no one.

Markey proposal from 2006:
SECTION 201. NETWORK NEUTRALITY.
(b) IN GENERAL.—Each broadband network provider has the duty—
(3) if the provider prioritizes or offers enhanced quality of service to data of a particular type, to prioritize or offer enhanced quality of service to all data of that type (regardless of the origin of such data) without imposing a surcharge or other consideration for such prioritization or enhanced quality of service;

Snowe-Dorgan proposal from 2006:
(5) only prioritize content, applications, or services accessed by a user that is made available via the Internet within the network of such broadband service provider based on the type of content, applications, or services and the level of service purchased by the user, without charge for such prioritization;

Conyers-Lofgren proposal from 2008:
"If a broadband network provider prioritizes or offers enhanced quality of service to data of a particular type, it must prioritize or offer enhanced quality of service to all data of that type (regardless of the origin or ownership of such data) without imposing a surcharge or other consideration for such prioritization or enhanced quality of service."

So what you have shown is that I am technically correct and that at best, your own politics and ignorance blinds you to the truth about the legislation and at worst, you're trying to mislead the public on what Net Neutrality legislation is.

George Ou

Re: Why BitTorrent causes so much latency and how to fix it

> So you admit that it is possible for the application to trickle data to the TCP stack so you are basically admitting that I am technically correct.

yes, absolutely. To quote RFC 1925:

(3) With sufficient thrust, pigs fly just fine. However, this is
not necessarily a good idea. It is hard to be sure where they
are going to land, and it could be dangerous sitting under them
as they fly overhead.



> "If a broadband network provider prioritizes or offers enhanced quality of service to data of a particular type, it must prioritize or offer enhanced quality of service to all data of that type (regardless of the origin or ownership of such data) without imposing a surcharge or other consideration for such prioritization or enhanced quality of service."

exactly my point. The ISP could give "enhance quality of service" to ALL voip traffic, "regardless of origin or ownership". Which is exactly what everyone wants. Schedule small voip packets first, then let the bulk transfers through (such as VOD or bittorrent). This is exactly the kind of network neutrality that does not interfere with QoS.

So 8472, you believe that the sale of QoS should be banned

I'm not sure what your objection is, we have cheap $50 multi-core processors today that would hardly notice the extra load of trickling data periodically to the TCP stack. If an application can be made to be latency-friendly in the absence of other network-level or TCP stack-level fixes, which is the vast majority of customers, it only makes sense to cater to that market. Your thinking is too “in-the-box” and this kind of narrow minded thinking that prevented PC telephony from taking off until Skype came along and made their software just work. What consumers want is a P2P client that “just works” and doesn’t require them to buy QoS capable hardware or require them to be network engineers. I use to build large enterprise networks for a living and I can configure QoS all day long and even I don’t want to mess with this stuff if I don’t have to.

If these bans on QoS charges are turned in to law, then customers who never use VoIP services because they prefer traditional land line services will be forced to subsidize QoS services for those who do use VoIP. Then what if the ISP decides that it isn't worth doing if they can't charge for it? Then you don’t get QoS even if you’re willing to pay for it.

I would love to get free massages too, but the last thing I want is the Government to mandate free massages because that's a sure way to effectively ban massages. What you're saying is that because you want free QoS, you want the Government to mandate "free" QoS which is effectively bundled QoS or no QoS. What I'm saying is that this kind of thinking is short sighted, selfish, and ultimately foolish.

Then what about IPTV service? You realize that IPTV service has to dynamically carve out a portion of an FTTN broadband pipe with low latency properties right? Are you suggesting that if a DSL provider like AT&T prioritizes their own U-Verse video, then they must prioritize all other video streams because they happen to use the same application? Do you not understand the unintended consequences of this? I'll give you a hint. You go passing these short sighted laws and you'll see DSL providers switch to HARD allocations of bandwidth for their IPTV service. So even if you don't want IPTV service from your DSL provider, you're not getting that bandwidth back. Now what do you do smart guy?

Re: Why BitTorrent causes so much latency and how to fix it

This has nothing to do with subsidizing or forcing the use of QoS. ISPs just take the wrong approach right now. Instead of fairly throttling all bulk protocols to ease congestion (and thus make VoiP usable for other customers) they just throttle protocols that do not represent their interests. The point is that all this is anti-competitive as you can see with bell canada which rolls out its own video store but shapes bittorrent inc.'s and vuze's video store.

So, assuming that the overselling business model won't go away one form of shaping or another is necessary anyway and mandatory (as the user can't choose). Net neutrality is only about ensuring fairness of this shaping. To make it a proper QoS instead of Quality of Business.

They alraedy have QoS devices... what do you think are ellacoya applicances doing? So it's not a matter of investment but a matter of policy/configuration.

Regarding IPTV. IPTV does not require low latencies, it only requires bandwidth guarantees. And under net neutrality laws ISPs might be required to wholesale IPTV-allocations to other IPTV providers if those other providers wish to offer similar services over the internet. But unbundling is never a bad thing as you can see here in europe.


Anyway, we're digressing... this is about subpar packet scheduling. I have initially stated that it's "possible to a certain extent" and "a hack". You make that into a "possible" and turn that one into a "should be done".
No, it should not be done, because TCP applications are bandwidth-latency-product- and jitter-agnostic. Doing any "tweaks" on the application level would be detrimental to performance _at least_ under some edge cases, most likely under many more conditions.

If you have an issue then ask your OS manufacturer or router manufacturer why their network stacks don't do what a network stack should do.

The optimal and feasible solution lies in properly applied QoS-implementations. You demand an automagic solution from p2p-developers... why should this be any easier for them to implement - despite having less knowledge what's currently going on on the network than the OS stack or the router - than the OS/router manufacturer?

I suggest you aim your demands at them instead... i.e. demand a default-on QoS implementation that gives priority to latency-sensitive services... and hey, windows even claims to have a QoS-driver installed by default, maybe the VoiP application would just have to request higher-priority scheduling and doesn't... so it might even be the fault of the VoiP application.

Point is that you're looking for solution in the wrong place.

Re: I did use DD-WRT with a Buffalo router and the QoS latency gains were questionable

I did use DD-WRT with a Buffalo router and the QoS latency gains were questionable. I prioritized the ports for Team Fortress 2 (same as Counter Strike) and for ICMP (for ping testing) and it didn't do anything for the ping tests and the game play still sucked.


The one thing that is CRITIAL to do is enable QoS for both LAN and WAN and prioritize them to the "premium" option. Also make sure your DD-WRT version is up to date some of the older one had bugs ;) Try using this firmware ver. this is what I am running: DD-WRT v23 SP2 (09/15/06) std
Also best thing to test is http traffic latency. If bittorrent slows down your http traffic enable QoS for http and test. it dramatically changed my experience with websurfing w/ bittorrent running. (another thing i should mention is that I am using rtorrent as my bittorrent client)

Re: Why BitTorrent causes so much latency and how to fix it

@host
There's a difference between QoS and Tiered internet. If you're non-discriminitive, QoS will NOT be banned.
However, what the ISP's are doing now IS discriminitive. When they have competition like Skype or vonage, and a phone company blocks those services for "QoS" they're actually just cutting off competition (which is an Anti-trust situation) or if nothing else, it's a conflict of interest.
Now let's take your situation of Google "trying to cripple DSL from getting into the TV business." Try again. An open internet would not cripple DSL from offering TV, it'd force those services to compete in an open market where multiple services are available. Customers can choose (I know a lot of companies HATE that word) what service they want as opposed to being forced to choose the service that you just so happen to get with your ISP.
If you think this will never happen, it's already started. Some ISP's have started small by replacing ads on webpages (do a search for "isp replacing ads" without the quotes).
Again, non-discriminitive QoS is perfectly fine.

Re: Why BitTorrent causes so much latency and how to fix it

Internal network QoS: Tomato
http://www.polarcloud.com/tomato

Works amazingly well for my sad DSL. I primarily use Xbox Live while Bittorrent-ing.

Its not magic, but it works as advertised, and once a particular connection is established is prioritized, no worries.

You're dodging the questions 8472, and you're sadly misinformed about IPTV

You're dodging the question on why you would prohibit the sale of QoS. Now answer the question. Why do you want the Government to mandate free QoS or no QoS for Broadband Internet?

You're changing the subject to an irrelevant topic. Vuze isn't a "service" that delivers packets; Vuze offloads their bandwidth costs to the ISP using P2P and has absolutely no comparison to a CDN Video on Demand model. Throttling P2P application is strictly a fairness issue because P2P uses the multi-stream cheat to get a disproportionate amount of bandwidth. Jacobson’s algorithm is being thoroughly exploited by P2P. http://blogs.zdnet.com/Ou/?p=1078.

It's also quite ignorant of you to believe that IPTV doesn't require low latency. You're confusing it with Video on Demand which is buffered enough to not need low latency and only flow rate assurance. IPTV is a live streaming service with very low buffering primarily because consumers want to channel-surf. You cannot have a half-second delay any time you flip the channel because it wouldn't work the way people expect it to when they compare it to Cable or Satellite TV service. You also need the low latencies if you want the audio to sync up perfectly with the video.

So you must absolutely prioritize IPTV service with minimal bandwidth guarantees and low latency guarantees if you want a functional TV service competitive with Cable and Satellite. This is being prioritized at the expense of all other Internet traffic but you have to do this or else the system doesn’t work. If you give other Internet-based video streaming services (which are typically VoD services that don’t need low latency) equal priority as the Net Neutrality bills require, you kill the IPTV service. Because the DSL provider can’t afford to lose the IPTV business, they will simply be forced to use a fixed slice of bandwidth from their FTTN service and never make that available to you or any other content provider and this would be a horrible unintended consequence of these rules.

8472 says: "I suggest you aim your demands at them instead... i.e. demand a default-on QoS implementation that gives priority to latency-sensitive services... and hey, windows even claims to have a QoS-driver installed by default, maybe the VoiP application would just have to request higher-priority scheduling and doesn't... so it might even be the fault of the VoiP application."

Again, you want the Government to mandate "free" QoS which makes about as much sense as mandating "free" car wash service with gas. Sure you can demand it, but you're just mandating bundled service which ironically the advocates claim they're against.

As for the QoS-driver in Windows, I would suggest you do a little more research about that before you embarrass yourself further.


George Ou

Re: Why BitTorrent causes so much latency and how to fix it

@George Ou
What are you talking about? They're not "offloading" their bandwidth costs, they pay (exorbitantly) for their bandwidth to their ISP.
These Internet companies pay much more than the average web operator because their sites use greater bandwidth from host sites than other operators. Where do the companies that have to serve these pages to their customers get their income? From THEIR customers.

If the ISP's offers "Unlimited" internet at X speed they need to deliver that.
It's the ISP's fault if they cannot keep up with the data rates they promise. Congress authorized the big bells to charge extra fee's to cover the cost of installing Fiber Optics to the Curb. What did they do with that cash? They bought local phone companies and stifled competition even more.

In business, this is what's called a "trend." Objects in motion tend to stay in motion unless acted upon by an equal or greater opposing force. If you give the ISP's full reign to do whatever they want despite contracts with their customers to the otherwise, it will create a monopoly that would make DeBeers look like a local mom-and-pop store.

Craysh, do you not understand how P2P distribution models work?

Craysh says: "What are you talking about? They're not "offloading" their bandwidth costs, they pay (exorbitantly) for their bandwidth to their ISP."

Google (YouTube), Apple, Microsoft, Netflix etc pay a ton of money on bandwidth and CDN costs. They pay their own end of the pipe and the internconnect fees are already covered by the peering arrangements between the carriers. These companies for example are not offloading their costs and they really can't use P2P for their distribution model because they need in-order delivery so that the user can watch the movie as they are downloading it.

Vuze uses the P2P distribution model. They pay just enough to seed the content but the vast majority of the distribution costs are pushed out to the ISPs and their customers.

Craysh says: "If the ISP's offers "Unlimited" internet at X speed they need to deliver that."

ISPs don't offer unlimited. Comcast for example does not advertise unlimited. No ISP offers unlimited or guaranteed throughput. All broadband connections are shared.

Re: Craysh, do you not understand how P2P distribution models work?

So what happens when Comcast decides that Netflix new download services are eating into their on-demand income? Or if Hulu starts getting more popular and people drop Comcast cable and only keep their internet? With no limits on what the ISP can do, the connection can be degraded to encourage (read:force) people to keep the ISP's offering.
In an ideal free market situation, there would be competition so this wouldn't be an issue, since you could simply go with another ISP. Unfortunately, most cities only have one or two choices at the most.

And you're right, Vuze doesn't have as much bandwidth costs as some of the major sites, but they pay for that which they use. Any additional bandwidth that is used across the network distributing those files are covered by the paying customers on either end of that connection. If you believe that they should pay additional fees because other people are using the service, then what's to stop the ISP's from charging other companies because use of their programs consume large amounts of bandwidth too? Let's take FTP for instance: To use FTP you only have to download a client such as Filezilla. Should Filezilla be required to pay a surcharge every time someone uses their client to transfer large files via FTP? No, they shouldn't because the bandwidth that is being used is being paid for by the person who's using the client and the person paying the FTP server's bandwidth.
Or an even more common comparison: perhaps Mozilla should be charged because the price in bandwidth they pay for (5.7MB for each copy of FireFox) pales in comparison to the amount of data transferred by using it (viewing web pages daily etc).
It's the same situation with Vuze. Vuze pays for the initial distribution bandwidth costs (several times for each item I'm sure) because that's all they use. Should Vuze pay extra because the users of their product are swapping data between each other simply because they use Vuze's client? No, because the people on either side of that P2P network are paying for the bandwidth they are using. So you see, every end of bandwidth usage is paid for by the people who use it

As for ISP's not being unlimited, they do not list limits on their site, contracts, commercials, fliers, or anything else (unless you count Time Warner testing it's usage plan in Texas). If no bandwidth limits are disclosed, it's "un-limited."
You'd be pretty pissed if you bought a car and for some reason it has an undisclosed "30 miles a day" limit imposed on it.

Re: Craysh, do you not understand how P2P distribution models work?

So what happens when Comcast decides that Netflix new download services are eating into their on-demand income? Or if Hulu starts getting more popular and people drop Comcast cable and only keep their internet? With no limits on what the ISP can do, the connection can be degraded to encourage (read:force) people to keep the ISP's offering.
In an ideal free market situation, there would be competition so this wouldn't be an issue, since you could simply go with another ISP. Unfortunately, most cities only have one or two choices at the most.

And you're right, Vuze doesn't have as much bandwidth costs as some of the major sites, but they pay for that which they use. Any additional bandwidth that is used across the network distributing those files are covered by the paying customers on either end of that connection. If you believe that they should pay additional fees because other people are using the service, then what's to stop the ISP's from charging other companies because use of their programs consume large amounts of bandwidth too? Let's take FTP for instance: To use FTP you only have to download a client such as Filezilla. Should Filezilla be required to pay a surcharge every time someone uses their client to transfer large files via FTP? No, they shouldn't because the bandwidth that is being used is being paid for by the person who's using the client and the person paying the FTP server's bandwidth.
Or an even more common comparison: perhaps Mozilla should be charged because the price in bandwidth they pay for (5.7MB for each copy of FireFox) pales in comparison to the amount of data transferred by using it (viewing web pages daily etc).
It's the same situation with Vuze. Vuze pays for the initial distribution bandwidth costs (several times for each item I'm sure) because that's all they use. Should Vuze pay extra because the users of their product are swapping data between each other simply because they use Vuze's client? No, because the people on either side of that P2P network are paying for the bandwidth they are using. So you see, every end of bandwidth usage is paid for by the people who use it

As for ISP's not being unlimited, they do not list limits on their site, contracts, commercials, fliers, or anything else (unless you count Time Warner testing it's usage plan in Texas). If no bandwidth limits are disclosed, it's "un-limited."
You'd be pretty pissed if you bought a car and for some reason it has an undisclosed "30 miles a day" limit imposed on it.

Re: Why BitTorrent causes so much latency and how to fix it

> Because the DSL provider can’t afford to lose the IPTV business

err... are we talking about a DSL provider or an IPTV provider? Because if you're saying that the DSL provider and IPTV provider are not only the same entity but also are inseparable you obviously get a conflict of interest and an anti-competitive market as all IPTV competitors would have a disadvantage compared to the DSL/IPTV conglomerate.

So i suggest you make a clear distinction between access providers and content providers, otherwise it would like you're advocating lock-in schemes and mono-/duopolies.

> ISPs don't offer unlimited.

maybe not in their fineprint, but in their adverts. And most people expect what they've been promised and not what some weasels sneak into their contract.... most people don't even read it. So it's a marketing-failure among other things. People have been fed with big promises, now that they actually use what they have been promised the ISPs reap the fallout. Deceptive advertising is hardly something one has to defend.

> You're dodging the question on why you would prohibit the sale of QoS

i never said i would prohibit the sale of QoS. QoS can be sold to customers, as long as it is applied equally to all services of the same type. E.g. you could sell them a "speed up VoIP and games"-package. Which would apply to all VoIP services (as far as technically possible) and not just prefer a specific vendor.

We have something similar in germany (although not technically related). DSL can operated in interleaved or fastpath mode. The latter has (theoretically) higher error rates but better RTTs, hence it's sold as an additional option and frequently used by gamers. This option applies to all services equally, and yet it can be sold.

> You also need the low latencies if you want the audio to sync up perfectly with the video.

not really, as long as audio and video frames that belong together are in the same buffer window it does not matter which one arrives earlier, it's only important that you have both of them.

> You cannot have a half-second delay any time you flip the channel because it wouldn't work the way people expect it to when they compare it to Cable or Satellite TV service.

people actually prefer VoD over TV, so i'd say they can live with "slow" channel flipping if you replace it with a low-res preview feature for example. Not to mention that near-guarantees for latency of 250ms is feasible with good QoS solutions.

> So you must absolutely prioritize IPTV service [...] if you want a functional TV service competitive with Cable and Satellite.

oh really? is that the only mode of competition? With IPTV you have the advantage that only that what is currently watched must be delivered. With cable and satellite you have to deliver all at once through a limited resource (frequencies). Thus you can offer a much broader choise with IPTV, i'd say this is a quite neat mode of competition.



> Vuze offloads their bandwidth costs to the ISP using P2P and has absolutely no comparison to a CDN Video on Demand model.

The users serve as your CDN, that's the only difference. Regarding your claim that p2p applications cheat... this one is actually simple to solve with QoS and Diffserv. Simply stuff P2P traffic into a traffic category below bulk and let them compete for the remaining bandwidth, i.e. that's what left after allocating low-latency, best-effort and reliability traffic. The TOS/Diffserv value for that is 0x20 iirc.

And your claim that what ISPs doing right now is about fairness is bogus. If it would be about fairness ISPs could still allocate all remaining bandwidth (see above, just via DPI instead Diffserv) to P2P applications. But they don't they simply hard-cap the traffic to rediciously low amounts during certain hours (bell), completely sever TCP connections 24/7 instead of shaping them (comcast). See http://arstechnica.com/news.ars/post/20080602-new-filings-reveal-extent-damage-of-bell-canada-throttling.html
Does this look like adaptive shaping to ensure fairness to you? It certainly doesn't to me. It's just a "let's reduce the overall traffic, it'll save us coasts"-approach, regardless of necessity.

Another point is: Over here in mainland europe most ISPs do NOT shape p2p traffic. Even in sparely populated (=> more expensive infrastructure) countries like sweden where they have internet connections comparable to those in japan (100Mbit symmetric to the home) there is near to none traffic-shaping. Especially nothing comparable to Comcast. And that's despite the fact that sweden is one of the world leaders in filesharing.
The fact that such ISPs do exist alone proves that it IS possible to build the necessary infrastructure and stay competitive... thus it's a failure of ISPs to build out infrastructure or a lack of a larger scale infrastructure like open IXPs (the 3 largest IXPs are all in europe: AMS-IX, DE-CIX, LINX). You can hardly blame the endusers for market failure.

> Again, you want the Government to mandate "free" QoS

i said

----
[...] than the OS/router manufacturer?

I suggest you aim your demands at them instead...
----

where did i mention the govt in that paragraph? Making QoS-hardware/software available to the enduser who has "issues" with his gaming+p2ping does not involve politics. I only said it should be part of most network stacks, not that it must be.

The govt should only enforce fairness (no lock-in, no market distortion, non-anti-competetive etc. etc.) on QoS on the ISP-side, that's what net neutrality is about at its core, subpar law proposals do not define net neutrality, they are only attempts at its implementation.

8472, you started off questioning my knowledge and now you show that you don’t even grasp the basics

8472, you started off questioning my knowledge and now you show that you don’t even grasp the basics. You don't answer the questions and you chang the subject.

Do you not understand that DSL providers are the IPTV providers? The DSL provider's key incentive to building out FTTN infrastructure (AT&T U-Verse) is to be able to sell IPTV services, it's also important to the FTTH providers like Verizon FiOS. You remove those incentives, they don't build those bigger networks. These are private businesses that have to pay back their investors. They're not going to do it for charity just like Google won't spend a dime on building Broadband infrastructure or municipal Wi-Fi. When they build out that infrastructure, they get to carve out a piece of that infrastructure for their TV business. Verizon uses a separate optical wavelength separate from the broadband service but AT&T and other FTTN providers dynamically allocate resources from the broadband service and carve out a virtual circuit that assures sufficient throughput and low latency. It is understood that FTTN and FTTH providers get to have this IPTV service to justify their investment and it is understood that this creates better competition in the TV space.

As for "unlimited", Comcast for example does NOT advertise unlimited speeds. However, I have always said that I am for more transparency both from a public policy point of view and a good business stand point. In fact I believe that Government oversight on transparency is good for the free market. However, you are being unreasonable in your ignorant expectation that advertised peak speeds are supposed to be for minimum speeds.

"i never said i would prohibit the sale of QoS. QoS can be sold to customers, as long as it is applied equally to all services of the same type."

Yes but you're supporting Net Neutrality legislation which does prohibit the sale of QoS so you're a hypocrite. I'm saying by definition, the FTTN provider should be able to protect their own IPTV service and by definition they're not allowing other Internet content companies access to that bandwidth when the customer is using their IPTV. However, the customer is in effect saying they want their IPTV service and they have the choice to not purchase that IPTV service. This is what the consumer wants and the consumer should be able to demand this type of QoS that does NOT apply to all content providers.

"people actually prefer VoD over TV, so i'd say they can live with "slow" channel flipping if you replace it with a low-res preview feature for example. Not to mention that near-guarantees for latency of 250ms is feasible with good QoS solutions."

Again, don't confuse the context of "VoD" here. We're talking TV services for NORMAL people and what NORMAL people expect out of their TV service. If YOU don't like it, YOU DON'T BUY THE IPTV service and you get that entire pipe for the Internet. What is your problem?

"[...] than the OS/router manufacturer? I suggest you aim your demands at them instead... "

Again, what does your OS and your routers have to do with download latency? Hello? Is someone home up there? We're talking DOWNLOAD. We're talking about the consumer's right to tell their ISP what services and content providers they want prioritized going over their own broadband link. We're not talking about the ISP deciding what to de-prioritize and prioritize. Net Neutrality legislation, which you say you support, would ban consumer's choice.

George Ou

To Craysh

Craysh says: "So what happens when Comcast decides that Netflix new download services are eating into their on-demand income? Or if Hulu starts getting more popular and people drop Comcast cable and only keep their internet? With no limits on what the ISP can do, the connection can be degraded to encourage (read:force) people to keep the ISP's offering."

You're changing the subject here to Comcast, but I'll answer it. If Comcast or any ISP degrades a specific service beyond reasonable network management, that should be a anti-trust issue and it is illegal. But this is a totally separate issue from banning consumers from purchasing QoS and harming the IPTV business.

Reasonable network management has multiple forms. Primitive forms try to make things a little more fair by throttling the bandwidth hog applications but it's not the ideal solution since it isn't completely accurate. This is the cheap and less-than-ideal way of dealing with the problem but it shouldn't be illegal. Comcast is going to a more advanced form of network management that is independent of the protocol by the end of this year and they're already starting field trials. Comcast's new system simply decides that if a customer uses a disproportionate amount of bandwidth, say for example 2% uses 50% of resources, then they throttle that 2% down towards 2% of the resources. It turns out that the congestion problems are usually gone by the time they knock them down to 25% of the resources and they don't need to throttle any more. So the hogs still hog a huge amount of bandwidth but nowhere near as much. When there is no congestion, they go back to hogging 50% of the resources.

As to the rest of your post, I’ve already explained it before but I’ll keep it brief. At no point should Vuze or Mozilla or Google or Microsoft or any other company be charged for interconnect or off-ramp fees under the threat of blockage or unreasonable degradation. The Internet is a best-effort service and there are no guarantees on throughput or latency. As long as there is no specific targeting of a service to harm that service beyond reasonable network management as I defined in the previous paragraph, then there should be no problems.

Craysh says: “If no bandwidth limits are disclosed, it's "un-limited."

Now you’re just being irrational and unreasonable and you’d be laughed out of court if you tried to bring that complaint. I’m all for MORE transparency and I would be for more Government oversight on this matter but you’re just being flat out unreasonable with that kind of statement.

As for the ArsTechnica article on Bell Canada and the claims from CAIP, there are some serious problems with CAIP's allegations about VoIP blockage. They tried to shove 4.5 Mbps of traffic through a VoIP protocol which is clearly ludicrous since VoIP never exceeds 83 Kbps in the worst case. It seems that they are deliberately trying to scare the VoIP users when it's clear that their test was designed to trigger the system to identify a 4.5 Mbps VoIP UDP stream when this would never happen in reality and the fact that no Lingo or Vonage customers are complaining prove it. Because if you took a 32 Kbps or 82 Kbps VoIP stream and forced it down to 30 Kbps, you pretty much break the latter completely and make the former very problematic with that much packet loss.

George Ou

Re: Why BitTorrent causes so much latency and how to fix it

Great article George. I was wondering what tool you used for generating the graphs. I want to see if I get similar result in my home network.

Jesse, here's how I made the graphs

I set BitTorrent (official client) to use 10, 28, and 47 KB/sec upstream out of a peak sustainable upstream of 52 KB/sec.

Then I did a tracert to a common web site like yahoo.com just to figure out the first hop beyond my home router and determined it was 12 ms minimum latency under clear conditions.

Then I ran a ping to that first hop beyond my home router continuously as I downloaded or uploaded data. If you're in Windows, be sure to set the CMD console buffer to a minimum of 3000 lines which is something I do anyways and I keep this as a permanent setting and I set the window to display double the number or vertical lines. I then do a select all and pasted it in to Excel 2007 (any version will do). Then I did a global replace to erase everything except the ping number. Then I created a new column that detected 12 ms from that first column of data. This new column is the increase in ping over time that I get to plot out in Excel.

Re: Why BitTorrent causes so much latency and how to fix it

They never say that it would be illegal to buy/use QoS software. But if they use that as an excuse to throttle competitors, then yes it should be regulated.

Learn to read Craysh

"They never say that it would be illegal to buy/use QoS software. But if they use that as an excuse to throttle competitors, then yes it should be regulated."

Learn to read Craysh. I've written it many times and the proposed Net Neutrality laws are clearly preventing the ISP from offering QoS services. I've patiently explained why you need to permit this kind of service and you're not even taking the time to read or be genuine about your arguments. You're wasting my time here.

Re: Why BitTorrent causes so much latency and how to fix it

As i have stated before (which has been ignored): Looks like you (Ou) are confusing the concept of net neutrality with (potentially flawed) law proposals implementing said concept. Most QoS advocates are NOT against QoS, it's useful to use various services (p2p, voip, vod) simultanously after all. Net neutrality at its core is about preventing discriminating, unreasonable and anti-competetive traffic-shaping.

You repeatedly cite law proposals. So yes, these proposals might need some work (or maybe not, depending on how they are interpreted). But you mistake them for the concepts advocated by network neutrality proponents themselves.

> Again, what does your OS and your routers have to do with download latency? Hello? Is someone home up there? We're talking DOWNLOAD.

Oh, and why the hell are you talking about "spacing" packets if this is all about downloads? Spacing packets would only make (a small amount of) sense if we were talking about upload, at least there wouldn't be any intermediate steps. If you talk about download then this is a lost cause as packets get rescheduled en route.

But we are not talking about download, we ARE talking about upload, looking at your own article would tell you as much:

> If BitTorrent bursts 3 packets upstream over a 100 Mbps FastEthernet LAN connection, they will all sit in the upload queue of my DSL modem for 85 ms.

And what is comcast traffic-shaping? Right... bittorrent-uploads (seeding). So Net neutrality in this context is also about upload. Although net neutrality _in general_ is about fair QoS regarding download AND upload.

Re: Why BitTorrent causes so much latency and how to fix it

i couldn't say it any better: http://arstechnica.com/news.ars/post/20080604-arpanet-architect-bring-fairness-to-traffic-management.html

Re: Why BitTorrent causes so much latency and how to fix it

I don't have any problem with Bittorent. Last night while sleeping I downloaded 10 movies. Before I left for work I disabled Bittorent in the Network Properties. BT uses 2 ports and I simply uncheck them. Next time I want to download, I'll go in and turn the ports on.

Re: Why BitTorrent causes so much latency and how to fix it

8472 said
-----
You repeatedly cite law proposals. So yes, these proposals might need some work (or maybe not, depending on how they are interpreted). But you mistake them for the concepts advocated by network neutrality proponents themselves.
-----

If the laws are can be interepreted in such a way that it outlaws QOS, you should assume that that interpretation will prevail at some point. Even if you can't believe that, you can be certain that there will be plenty of costly lawsuits making that claim.



8472 said
-----
err... are we talking about a DSL provider or an IPTV provider? Because if you're saying that the DSL provider and IPTV provider are not only the same entity but also are inseparable you obviously get a conflict of interest and an anti-competitive market as all IPTV competitors would have a disadvantage compared to the DSL/IPTV conglomerate.
-----

Using your logic, you should be fighting for legislation that forces Echostar to let directTV use their satellites to transmit video. If the
cable company wants to set aside X MB/S for their video service, I see nothing wrong with that. Would you also argue that I should be able to sell my gasoline using my local Chevron's Pumps?

Infrastructure isn't free. I object to them giving preferential treatment to some external video suplier over another. I'd also object if
they made using another IP more expensive than it currently is. But there's a difference between Skype or Vonage, which don't require much
bandwidth, but do require QoS, and a Cable Network.

If you're actually arguing that Verizon can't reserve part of their network exclusively for their video service, then you've lost me, and I've
been on the NN train for several years. If you don't like that plan, you can go the route of Lafayette Louisiana, and have your city build it's own Fiber network...oh wait, they're
providing their own video service as well.

If your goal is to get people to call their congressmen up and tell them to oppose net neutrality, you're doing a good job.

With that said, your link to the engineer that helped create arpanet was interesting. I'm fairly certain that it's an extremely inefficient use of resources and will likely mean everyone
ends up with less up and down bandwidth, but it would force real numbers when selling service.

Personally, I think a better solution is that that figure is the minimum level of service, but that someone can use more of it, (albeit at a lower priority than anything up to the min) if its' available.

George, what's with the order of messages?

There's no order to them. 5/31 at the top, which goes to 6/1......6/1 at the bottom which goes to 6/2 then 3 then 1 then 4


It makes it very hard to follow threads.

Don't know why the order is badly messed up

I called tech support and they said it's a bug in this blogging module. Maybe I should ask them to reboot my instance or something.

I was the one who introduced Nate Anderson to Anagran's Flow Management technology

I was the one who introduced Nate Anderson to Anagran's Flow Management technology and I've stated in these threads that Anagran's protocol agnostic solution is a much better solution. However, I don't believe that Government should outlaw inferior and less expensive methods. The market place should shame ISPs in to using the better solutions and this is what happened with Comcast.

Re: Why BitTorrent causes so much latency and how to fix it

@notgonnatellya

> Infrastructure isn't free.

right. That's why you pay subscriber fees for your broadband access. Just like telco infrastructure isn't free and you pay your telco fees. That doesn't mean the telco gets an exclusive/reserved part of that line to shove other services towards you. At least over here telcos are forced to unbundle and resale services and it works quite well as it allows smaller competitors to achieve greater coverage by doing wholesale in some areas and run their own services in other areas.

My point is that reserving bandwidth is ok, as long as the user has the choice what to reserve the bandwidth to. If he wants to use VoD service X then he reserves bandwidth to that one, if he wants to use his ISP's service then he reserves bandwidth to that one. This offers _choice_, leads to more competition and ultimately benefits the consumer. Only providing reserved bandwidth for bundled services is a lock-in scheme and essentially leads to a monopoly.

Why is that so hard to understand?


-------
8472 said
-----
You repeatedly cite law proposals. So yes, these proposals might need some work (or maybe not, depending on how they are interpreted). But you mistake them for the concepts advocated by network neutrality proponents themselves.
-----

If the laws are can be interepreted in such a way that it outlaws QOS, you should assume that that interpretation will prevail at some point. Even if you can't believe that, you can be certain that there will be plenty of costly lawsuits making that claim.
-------

*sigh* Again. I did _not_ say that i'm supporting the current law proposals. I said i support network neutrality as a _concept_.

For me net neutrality is equal treatment of all services of the same class on the side when it is enforced by the ISP, regardless of vendor and protocol. Non-enforced (i.e. user-chosen) prioritizing of traffic would be ok as long as his choice is free (except for technical restrictions).

Examples:

Always prioritizing ISP's VoiP and Skype the same way is ok.
Always prioritizing HTTP over SMTP is also ok.
Optionally prioritizing ISP's VoD service over youtube at the users discretion would also be ok as long as the reverse would also be an option.
Always prioritizing ISP's VoD over Youtube OR prioritizing Youtube over the ISP's VoD would not be ok.
Always prioritizing VoiP (of any kind) over bittorrent but not over kazaa would not be ok.

I can't see any way how this would harm the consumer. The ISP would still be free to reserve bandwidth for his VoD services as long as the user has the option to disable this reservation or simply shift it to another service which he prefers.
QoS in general would also be ok.

While user choice would probably be technically challanging it is not mandatory, the ISP is free to not offer user choice and simply treat all services of the same class equally.

You do have a choice, you don't have to order the IPTV service

8472 says: "right. That's why you pay subscriber fees for your broadband access. Just like telco infrastructure isn't free and you pay your telco fees. That doesn't mean the telco gets an exclusive/reserved part of that line to shove other services towards you. At least over here telcos are forced to unbundle and resale services and it works quite well as it allows smaller competitors to achieve greater coverage by doing wholesale in some areas and run their own services in other areas."

In Germany, my friend gets IPTV service from his VDSL FTTN provider BUNDLED. When you choose that service, it MUST have QoS in the form of bandwidth guarantees AND latency guarantees.

8472 says: "My point is that reserving bandwidth is ok, as long as the user has the choice what to reserve the bandwidth to."

You do have a choice, you don't have to order the IPTV service if you don't want that bandwidth reserved. NOBODY buys IPTV service with the expectation that BitTorrent will mess up your TV service and you're being silly to suggest otherwise. It’s not shoved down your throat at all, you have a choice.

The consumer also should have a choice to tell their ISP what services they want to prioritize. You could have a situation where a household uses two Video on Demand services but they want one to get more priority than the other. So why shouldn’t the consumer tell their ISP to prioritize downstream iTunes more than XBOX Market Place or vice versa? The point is that for downstream prioritization, the user has to rely on the ISP’s gear because it is the transmitting device can exert the most control. We’re not talking about some sinister plot where Apple pays an ISP to lock out a competitor like Microsoft or vice versa and certainly I would be open to making that form of exclusive access illegal. But we’re talking about consumers choosing to prioritize a certain web site above other web sites and the consumer should be able to block or prioritize any service they want.

Re: Why BitTorrent causes so much latency and how to fix it

Recent interesting post on how some colleges are dealing with bit torrent: http://netequalizer.wordpress.com/2008/05/17/curbing-riaa-requests-on-your-student-network/

Re: Why BitTorrent causes so much latency and how to fix it

There are some great solutions here, but I have an ultimate solution that I use. Route your bittorent traffic over a VPN service like Relakks (there's many others out there). These low cost services allow you to route all your traffic through one connection. Solves the latency hands down, you'll still need to throttle in your app as not to choke off your pipe, but the effect of a VPN is that all those connections behave as one. It also adds privacy so you have less to worry about from the RIAA. Relakks will result in a slower top download speed, but I've found others which will run at full speed.

Why BitTorrent causes so much latency and how to fix it

Written by: George Ou 5/31/2008 4:54 PM Anyone VoIP or online gamer who has a roommate or a family member who uses BitTorrent (or any P2P application) knows what a nightmare it is when BitTorrent is in use. The ping (round trip latency) goes through th...
# nathanr|ca

Re: Why BitTorrent causes so much latency and how to fix it

I had the same problem with BT lagging everyone at home as well, I found that if we limit the upload rate on our BT applications, it doesn't cause as much of a problem for others...

Re: Why BitTorrent causes so much latency and how to fix it

The 8472 said:

"@notgonnatellya

> Infrastructure isn't free.

right. That's why you pay subscriber fees for your broadband access. Just like telco infrastructure isn't free and you pay your telco fees. That doesn't mean the telco gets an exclusive/reserved part of that line to shove other services towards you. At least over here telcos are forced to unbundle and resale services and it works quite well as it allows smaller competitors to achieve greater coverage by doing wholesale in some areas and run their own services in other areas."

If you say so, but it sure doesn't sound like a free market to me. Should the cable companies be forced to unbundle? What about satellite providers? Gas stations?

Why do you single out telcos? If AT&T or Verizon spend billions to create upgraded fiber-based networks, why should they be forced to allow other providers to use those networks when said providers didn't invest a dime?

When there was only one network (the telephone network), an argument for heavy regulation, etc. made some sense. But now there is competition: if I don't want TV from Comcast, I can get it from AT&T or DirecTV. If I don't want Internet from AT&T, I can get it from Comcast or Hughesnet or a fixed wireless provider or even a cellular company. And so on.

If you force such unbundling, what you are really doing is discouraging such infrastructure build-outs. Why should AT&T spend all that money to build out a new fiber network? They'll never make it back on $25/mo or $35/mo DSL services.

Re: Why BitTorrent causes so much latency and how to fix it

The problem isn't with the last mile link, but in the ISP itself. The latency seen is due to bandwidth constraints somewhere along the path of the VOIP traffic. Low latency is an artifact of a IP network with spare capacity. The spare capacity means that no packet is lost and all packets have a low consistent latency. VOIP doesn't really need low latencies, but stable latencies. A call to the exact opposite point in the world will need 68ms just for the trip of 12.5K miles alone at the speed of light. And that is best case. Most people won't notice even a 100ms delay from what you say to be heard on the other end. Even a satellite based phone call has a minimum 300ms delay with one hop (you to POTS to sat to POTS to other). What gets VOIP screwed up is latency changes. Some at 200ms and some at 5ms will cause many low grade VOIP systems to mess up. But a VOIP system that buffers 250ms at least would never have any problems.

But that variation is not just due to your end, but any blockages down the road. If an ISP is shaping bittorrent traffic at his forwarding connection, the packets will pile up in your router regardless of what BT does. Even 10KB/s will cause them to pile up. Yes slower and less often, but they will pile up even with one per connection. The BT doesn't see that TCP has to retransmit that 1.5KB packet any number of times to get it through UDP. It just sees 1.5KB of upload. And it will send about 6.6 such packets per second using some moving average plus some overhead ones as well. But your router just sees the average retransmissions required times 1.5KB IP packets times 6.6 or 10's of KB/s usage in the upload side. Those latency spikes you see are the IP retransmit packets piling up. And that retransmission can be required through any given part of any BT peer connection going through a site that is capacity constrained whether its deliberate or not.

The VOIP can similarly be messed by somewhere a network provider dropping packets at random to shoehorn traffic through a bottleneck. Strict network neutral carriers of that traffic actually would help streaming traffic by randomly dropping the packets. Since the packets are dropped randomly, the latency would stay within the typical gaussian distribution and slowly change allowing the applications to adjust. Since shaping requires intelligence, dumb networks have much higher capacity, 2-10 times as much as smart shaping ones. That extra capacity usually works to drastically reduce dropped packet rates and thus make latency both low and stable. Which makes them look like the ideal non blocking network. That is why most non QOS networks look like ideal ones. Also a faster last mile takes care of more problems. That is why a 100/100mbps FTTH connection doesn't really have these issues. The bottlenecks are all somewhere else. Even a 1.5KB packet is only 0.12ms on such a connection. 8000 packets per second can make any problem with 100-200 peer connections seem effortless.

Think about a HDTV signal. Its 19.2Mbps. Even composed of 1.5KB packets, thats 1600 packets per second. 250ms delay means 400 packets are buffered. One or two packets either way won't cause any problems. And I had U-Verse. The HD channel switch times were more like 2-5 seconds, not 1/2 second. So latency hiccups shouldn't affect VOIP traffic in the last mile.

Need more information!

We do not know from the article alone:
1.What BitTorrent client is used in testing.
...although in reading the 100+ replies, about the 100th reply in:
"I set BitTorrent (official client) to use 10, 28, and 47 KB/sec upstream out of a peak sustainable upstream of 52 KB/sec."
2.What settings are used in the BitTorrent client, BESIDES upload and download speed limits!
3.What torrent/s are the tests run on, specifically how many peers and seeds are there...and if the torrents are new, old, or even POISONED!
4.Firewalled conditions in BitTorrent?
5."Clean Line" tests...we don't know if the tests were done on an internet connection that is not also being hammered by spyware/viruses both incoming and outgoing...or old BitTorrent traffic (DHT and peers/seeds retrying torrents you long-ago stopped).
6.If the tests' computer/s is also running networking software and hardware known to interfere with BitTorrent traffic or the internet connection in general. (Zone Alarm, I choose you!)
7.What ISP service is being used? (some are real lemons who cripple the WHOLE line when BitTorrent traffic is detected!)
8.Even how testing was done between upload speed changes can have a huge effect on the outcome. A short "cooldown" time between each upload speed reduction might make the latency temporarily worse, as both the line and the BitTorrent client hasn't adapted to the new conditions.

In conclusion, the test is not scientific and the conclusion should not be treated as fact.

This old article is noteworthy in explaining much of the problem:
http://www.stuartcheshire.org/rants/Latency.html

Lastly, a BitTorrent/uTorrent troubleshooting guide:
http://forum.utorrent.com/viewtopic.php?id=15992

Re: Why BitTorrent causes so much jitter (high ping) and how to fix it

for me...using bittorrent at all causes online game play to be erratic

downloading at 3kb..downloading at 300 kb
uploading nothing or uploading at the max speed

it all causes games to feel screwy like there is some kind of packet loss or something

it doesnt have anything to do with limiting the upload...