Why BitTorrent causes so much jitter (high ping) and how to fix it
Anyone VoIP or online gamer who has a roommate or a family member who uses BitTorrent (or any P2P application) knows what a nightmare it is when BitTorrent is in use. The ping (round trip latency/jitter) goes through the roof and it stays there making VoIP packets drop out and game play impossible. I’ve personally experienced this and many of my friends have experienced it. When I had a roommate in 1999 and I had my first broadband connection, my roommate started using the hot P2P (peer-to-peer) application of its day called Napster. As soon as he started sharing, my online game would become slow as molasses and I’d experience a huge 500ms second spike in round-trip latency and any gamer knows that pings over 100 are problematic.
When it’s just myself using BitTorrent, I can pause it while I make VoIP phone calls or when I want to game. But when it’s time to ask your family member or roommate to stop using their application, it becomes awkward because why should they stop using their application just so you can use your application? My roommate paid for half the broadband fees so I couldn’t always tell him to stop and he couldn’t always ruin my latency making it a contentious situation for both of us. What makes it even more frustrating is that there’s plenty of bandwidth capacity to theoretically satisfy both our needs, but something about these P2P applications make them them very bad housemates.
I decided to do some research on this and ran a series of tests which resulted in some very interesting data. I fired up various applications at varying rates of bandwidth utilization in the upstream and downstream direction and I measured ping to my first router hop (ISP router) beyond my home router. I ran continuous pings to the ISP router to check the round-trip latency above the normal baseline latency of 12 ms and plotted out the results. This test methodology measures network jitter (the variation of delay) on the “last mile” which is what I’m primarily concerned about because that’s where most of the damage is done by local usage of P2P applications.
BitTorrent downloads cause huge amounts of latency
The first set of tests I did were download tests. First I downloaded a file using HTTP which ran at 260 KB/sec (2.08 Mbps) which is roughly 87% of my peak download performance. I then set BitTorrent to also download at 260 KB/sec to compare the effect on my ping times. To my surprise, there was a significant difference in the amount of ping increase between HTTP downloading and BitTorrent downloading despite the fact that both applications were downloading at the same average rate.
When you look at the two graphs immediately above, you see that BitTorrent causes an average increase in ping of more than 117 ms (milliseconds). Note that those 6 ping spikes in the graph were actually bad enough that they caused the ping to time out which means the ping was higher than one second. HTTP on the other hand didn’t increase the ping nearly as much but it still caused an average of 46.7 ms increase in ping and it only peaked at 76 ms. So how is this possible when both applications are using the same amount of bandwidth? I did some thinking and this is the hypothesis I came up with.
Burst traffic fills up transmit queues
The difference in ping time is caused by the fact that HTTP is a single source of data whereas BitTorrent is about 20 sources of data. Incoming data from from multiple sources via the fast core of the Internet can sometimes clump closely together when multiple sources happen to transmit data around the same time. So in the period of 20 ms (the interval between typical VoIP packets), up to 200 max-size 1472 BYTE packets can occasionally build up in the DSLAM transmit queue (blue square in illustration above labeled as “download queue”) causing my ping requests to time out above the 1 second mark. But on average, we get around 23 of these packets causing sitting in the DSLAM transmit queue causing an average increase in ping of 117 ms. When there is a single transmitter, it might burst and clump packets close together but it won’t be at the level of 20 transmitters.
With HTTP causing 76 ms of downstream delay, that means 15 of these 1472-BYTE packets are sitting in the DSLAM transmit queue causing a less extreme increase in ping. This is still problematic for VoIP communications and it can certainly ruin online gaming. So despite the fact that there is plenty of remaining bandwidth for my VoIP or online gaming traffic, it’s the non-uniformity of the incoming Internet traffic that causes my VoIP phone calls and games to perform badly. Unfortunately since this is the downstream we’re talking about, the consumer can’t do much about it on their own end of the pipe because the delay is at the DSLAM which belongs to the ISP.
The only way to fix this problem is for the ISP to implement intelligent packet scheduling and application prioritization at the DSLAM to re-order those VoIP or gaming packets to the front of the transmit queue. With packet prioritization (generally referred to as QoS), your family member, your roommate, or your own video downloads won’t need to be stopped and they won’t interfere with your VoIP or gaming applications which makes everyone happy. Unfortunately, these types of QoS services may never see the light of day if poorly conceived Net Neutrality legislation gets passed that ban the sale of packet prioritization.
BitTorrent uploads cause excessive jitter
BitTorrent or P2P uploads also cause a lot of upstream jitter. I compared various types of upload traffic patterns to see what kind of increase in the upstream ping times would result. First I tried running BitTorrent with a 47 KB/sec (376 Kbps) bandwidth cap which was about 90% of my upload capacity, then I ran BitTorrent with a 28 KB/sec (224 Kbps) bandwidth cap at 54% of my upload capacity, and then I ran BitTorrent with a 10 KB/sec cap at 19% of my upload capacity.
In either the 28 KB/sec test or 10 KB/sec test, I’m not being greedy by hogging all the available upstream bandwidth and I’m leaving more than the 11 KB/sec of bandwidth needed for VoIP and online gaming. Yet I found that the additional ping caused by BitTorrent uploads (seeding) was unacceptable for gaming and problematic for VoIP applications. Even when I severely limited upload throughput to 10 KB/sec, it didn’t reduce the ping time spikes although it reduced the frequency of those spikes. However, even fewer spikes in ping time can pose the same problems for VoIP applications because they have to adjust their buffers to account for the worst-case network conditions. This would seem to indicate that BitTorrent is bursting packets rather than releasing the packets in a uniform and evenly spaced stream.
Next I tried using FTP at full throttle and I managed to get an FTP session going at 47 KB/sec (90% of my peak load) yet the jitter caused by FTP at this extreme rate of throughput was less than the jitter caused by operating BitTorrent at an average of 10 KB/sec. This would seem to indicate that FTP is outputting data in a more consistent manner that BitTorrent.
Lastly, I ran some ping tests during some VoIP phone calls using the Lingo service (competitor of Vonage). I had set Lingo to use the uncompressed G.711 which uses 11 KB/sec for both upload and download which makes it very comparable to BitTorrent uploading at an average of 10 KB/sec. But as soon as I ran the ping tests, I was shocked to see virtually no increase in ping times. I realized that this is because the VoIP ATA (Analog Telephony Adapter) device pulses small packets at exact intervals of 20 milliseconds at 50 times a second.
Smoothing out the packet spacing to reduce jitter
After running these tests, I am beginning to conclude that it isn’t so much the amount of data that causes excessive jitter; it’s the uniformity of data transmission. If the transmissions are spaced evenly, other packets from other applications can slip in between the packets rather than getting stuck behind multiple packets in the transmit queue. So would it be possible to engineer BitTorrent to transmit data uniformly and what would be the effect? I came up with the following chart to illustrate what this could mean for peaceful coexistence between VoIP/Gaming and BitTorrent/P2P.
I calculated that the max-size 1472 BYTE packet takes 28.3 milliseconds to transmit over my 52,000 BYTE/sec broadband uplink. If BitTorrent bursts 3 packets upstream over a 100 Mbps FastEthernet LAN connection, they will all sit in the upload queue of my DSL modem for 85 ms. Meanwhile, my VoIP or gaming packets get stuck behind those three packets for an additional 57 ms in the queue. This is shown in the first example in the illustration below with large 1472-BYTE red packets representing BitTorrent and small 223-BYTE green packets representing VoIP.
In the second example, I show what happens if BitTorrent simply spaced their transmissions evenly. It’s still less than ideal but at least it significantly reduces the jitter for VoIP or gaming packets.
In the third example, I show the ideal scenario where BitTorrent would reduce its packet size to 815 BYTES or less and pulse them out in 20 ms intervals at 50 packets per second. BitTorrent could essentially create a “VoIP friendly mode” that allows VoIP packets to fit cleanly between the BitTorrent packets and the increase in jitter would be no greater than 15.7 ms and would typically average around 8 ms increase. BitTorrent could also have a “Game friendly mode” that uses 679-BYTE packets at 60 packets per second.
Now it is possible to solve this problem on the network level by prioritizing VoIP and gaming packets in the home DSL modem upload queue. Unfortunately, I don’t have administrative access to the modem and implementing VoIP or gaming prioritization on my home router seemed to have no effect. Packets in the home router get forwarded as soon as they arrive with 100 Mbps Ethernet on both ends so there is nothing to reorder in the queue. More advanced business-class routers like those from Cisco will allow you to configure the speed of the FastEthernet connection to match your DSL throughput so that the queue will migrate from the DSL modem to the router but this isn’t very practical for most people. So it would make sense for application writers to try and make their application work as well as possible on the majority of home networks and broadband networks without QoS.
While modifying BitTorrent or the P2P application may not significantly fix the downstream problem, it would definitely fix the upstream jitter problem which means that people will be more willing to seed and contribute to the health of BitTorrent. The download jitter may improve if the dozens of peers sending information to me did it in a more uniform manner, but it is still possible that those peers will transmit at the same time which still causes packet clumping and bursting.
So why would BitTorrent or any P2P application vendor want to do this? Why couldn’t they just say “it’s not my problem”? Because it would fix their reputation as a bad housemate and more people would be willing to seed if it didn’t interfere with their other applications like VoIP or gaming. Corporate IT departments may also ease their ban on the technology if it doesn’t trash their Internet connection for the entire office. I am certainly a huge proponent of network-based solutions for dealing with jitter problems, but improvements in the application can act to further reduce network jitter even in the presence of existing network-based solutions. Furthermore, the vast majority of consumers do not have access to network-based solutions and it only makes sense for a P2P application vendor to cater to this market.
I will try to reach out to BitTorrent corporation to see if there is any interest in this and perhaps someone would be interested in modifying open source Azureus. It would be a very good feature to have or it would be a very interesting academic experiment at the very least.