The story that the new BitTorrent client uTorrent 2.0 is “network friendly” is making the top headlines on the Web and mailing lists. The only problem with this story it that it has no actual data to back up its assertions. I took the time yesterday to run some tests on the new uTorrent 2.0 beta build 16850 which supports the new “friendly” BitTorrent UTP. Based on my initial testing, the claim that the new BitTorrent client is network friendly appears to be false.
The folks at breakitdownblog.com have accused Netflix of deliberately throttling their users to 480 Kbps (60 KB/sec). Their proof? Downloading a video stream using 10 TCP flows is 10 times faster than downloading videos using the normal single flow. The blogger at breakitdownblog.com concludes that this must mean Netflix is intentionally throttling their users because they assume that there must be extra available bandwidth that Netflix is keeping from their customers. Unfortunately, this is often what passes as credible news these days and the blog has been slashdotted so they’re probably getting on the order of 40,000 hits or so.
Well I hate to break the news to breakitdownblog.com, but this is normal congestion behavior. In fact, they’ve accidentally discovered the multi-flow cheat that works around Jacobson’s TCP congestion control algorithm which rations bandwidth on a per-flow basis and not per-user basis. Even if the server’s capacity is completely filled, asking for 10 times more TCP connections will allow a client to pull nearly 10 times more bandwidth at the expense of other normal clients who are only asking for one TCP connection. Peer-to-peer (P2P) applications employ the same technique to accelerate its own performance at the expense of other users.
So what this sounds like is that the particular Netflix server they’re connecting to is running out of streaming capacity and it can’t handle this many users. This may still be bad on Netflix’s part if the performance problems are consistent, but let’s not attribute occasional inadequacy to malice. I can’t imagine this being frequent since Netflix’s support lines would be ringing off the hook. I also just ran a quick test with my Netflix account and verified that everything is working fine. I’m not suggesting that the alleged problems don’t exist, but we need to have a rational discussion about them.
Update 8/12/2008 – Robb Topolski has graciously apologized for the personal attacks.
I have to admit that I was surprised at the speed at which I’ve been attacked on my latest FCC filing “Guide to protocol agnostic network management schemes“. Within hours of my filing, Robb Topolski who now works for Public Knowledge wrote an attack blog “George Ou: Protocol Agnostic doesn’t mean Protocol Agnostic” which, along with a series of ad hominem attacks, basically accuses me of being hypocritical. I won’t respond to the senseless personal attacks, but I am going to rebut the core accusation that favoring protocol protection in the context “Protocol Agnostic” network management schemes is somehow hypocritical.
Before we debate the meaning and context of the word “agnostic”, we must first ask ourselves what is the underlying problem we are trying to solve. The answer is that we want a broadband network management system that is more equitable at distributing bandwidth fairly between users in the same service/price tier and we want a broadband network that ensures the best possible experience for every protocol and their associated applications.
Network Architect Richard Bennett put it best when he described network management as a two-phase process. In his blog he recently wrote:
An ideal residential Internet access system needs to be managed in two different but equally important phases:1) Allocate bandwidth fairly among competing accounts; and then
2) Prioritize streams within each account according to application requirements.
Phase 1 keeps you from being swamped by your neighbor, and keeps you from swamping him, and Phase 2 prevents your VoIP session from being swamped by your BitTorrent session.
So to address the first problem, we need a way to measure bandwidth consumption between these “competing accounts”. The common solution on the market place today computes bandwidth consumption implicitly and indirectly from the protocols passing through the network. This method of looking at the protocol to determine bandwidth consumption was initially chosen because it is computationally simpler and cheaper than trying to track usage statistics of tens of thousands of simultaneous application flows in real-time. While this solution is mostly accurate, it isn’t completely accurate.
As the identification of protocols become increasingly difficult because of protocol obfuscation techniques and as the cost of computational power declines, we’re starting to see a shift towards network management systems that don’t use protocol analysis to determine which users consume the most bandwidth. These new systems compute bandwidth distribution by tracking the bandwidth consumption of every flow on the network without looking at what protocols are passing through the system and these are the systems we refer to as “Protocol Agnostic”. But being “agnostic” strictly applies to this first phase of network management.
The second phase of network management is to ensure that every protocol works well. This requires knowledge of what protocols are passing through the system so that they can be given the necessary protections from other protocols sharing the same network. Since protocols and their applications have unique network requirements, we must identify the protocols being used so we can give them the necessary protections. It would be foolish to apply the concept of agnosticism towards this phase of management because doing so would make this phase worthless.
The reality is that a network can’t simply be a dumb pipe if we want every protocol/application to work well and no one has ever claimed that “protocol agnostic” should mean a dumb pipe. P2P applications have the ability to not only grab nearly every bit of bandwidth available regardless of how much capacity there is; they can cause significant problems for real-time applications like VoIP or online gaming applications even when the P2P uses a small fraction of the network. I have published experimental data showing that even low-throughput mild usage of P2P applications can cause significant spikes in network latency and jitter (latency and jitter explained here) which can severely degrade the quality of real-time applications. Even without any knowledge of how networks operate, anyone who has ever tried to play online FPS (First Person Shooter) gaming or use a VoIP phone while a P2P application runs on the same broadband connection understands how miserable the experience is. In fact, this is a common source of contention between roommates and family members.
This is precisely why Comcast needs to protect VoIP traffic from service providers like Vonage. Even though Comcast has nothing to gain because their own telephony service runs outside of the Broadband network on a different frequency, these additional efforts to ensure proper VoIP operation are actually helping a competing telephony service. In theory, Comcast could just leave this issue alone and let Vonage and other competing VoIP services suffer while its own telephony service remains immune. In practice, many Comcast’s customers will blame Comcast if they had any problems once the new system is in place and Comcast needs to do everything it can to make their customers happy.
Robb Topolski claims on behalf of me that these problems are somehow caused by Comcast when he says that “George has shown that Comcast’s proposed Protocol Agnostic scheme has unacceptable side effects”. Aside from the fact that I’ve said or shown nothing of a sort, it’s ridiculous to paint these noble and reasonable efforts in to something sinister. The Free Press has used similar fear mongering tactics to discredit these new network management efforts by Comcast by questioning why Vonage needs these special deals with Comcast in order for their traffic to go through insinuating that this is some sort of protection racket. The fact of the matter is that Comcast not only doesn’t block Vonage, they’re trying to prevent Vonage from being blocked by self-inflicted damage whenever the user tries to use aggressive protocols like P2P simultaneously with VoIP.
But these advocates of strict Net Neutrality and the “dumb” Internet continue to shout that this is tantamount to “discrimination” against P2P protocols if we provide protocol protection for VoIP or any other real-time applications. They’ll often ask “why can’t you give P2P the same protection” but this question is silly when you look at the reality of the situation. P2P applications don’t need this kind of protection and they’re thriving at 30 times faster than the VoIP application on a 3 Mbps broadband connection.
If we take an objective look at the situation, the P2P application is already consuming over 96% of the network capacity so would it wise to mandate a dumb network that allowed the P2P application to consume 98% of the network and completely break the VoIP application? If the network could only fairly allocate 2 Mbps of bandwidth to its broadband customers during peak usage ours, would it be so wrong to expect the P2P application to drop down to 95% of the broadband link while the VoIP application remains at the same miniscule 87 Kbps bitrate? I guess according to Robb Topolski, this is somehow unfair because the P2P application is forced to slow down while the VoIP application doesn’t slow down.
If this is the kind of petty game we’re playing, we could simply have a network management scheme that allocates “equal” amounts of bandwidth to P2P and VoIP whenever both protocols are active which means both protocols will operate at 87 Kbps and we could call that “equality”. Obviously no one would want that kind of system and there’s no need to force this level of equality so the P2P application is allowed to consume whatever remaining bandwidth is left over from the VoIP application. But taking what’s left over is beneficial to the P2P application because the leftovers are at least an order of magnitude more than what the VoIP application gets.
Robb Topolski then pulls out the DPI (Deep Packet Inspection) Bogeyman and claims that any sort of protocol aware system would violate user privacy by snooping beyond the packet headers and looking in to the user data. That’s nonsense on multiple levels because a system wouldn’t necessarily need to look at user data to determine the protocol being used and even if it did, there’s nothing wrong with an automated system that parses through user data. In fact millions of emails a day are parsed by anti-spam systems to prevent your email inbox from being inundated with spam but we don’t consider that an invasion of privacy because of the context and scope of the system.
But the smart network management system could simply look at the protocol headers without parsing through any of the user data to determine what protocol is being used because it doesn’t have to fear protocol masquerading techniques. This is because the protocol agnostic part of the system already tracks usage statistics which allows the network operator to implement a priority budgeting system where everyone gets a certain amount of priority delivery bandwidth. So for example, if everyone had a daily budget of 12 hours of priority VoIP service or 12 hours of priority online gaming service or 3 hours of priority video conferencing, they’re free to squander that budget in 30 minutes with a burst of high-bandwidth P2P usage masquerading as VoIP traffic. After the daily allowance is used up, all traffic would go through but be treated as normal priority data.
Chief BT researcher Bob Briscoe who is leading an effort to reform TCP congestion control at the IETF actually explained this concept of saving up for priority quite nicely and network architect Richard Bennett has also spoken positively about this type of a system. Priority budget schemes would actually help solve the fundamental fairness problem where some users consume hundreds of times more volume than the majority of users even though everyone pays the same price. The network would essentially give the low-usage real-time application customers a responsive delivery service while the high-usage P2P customer would get all the volume they want and everyone would get what they want.
When we focus on the ultimate endgame of fairness, it isn’t hypocritical to advocate equitable distribution of bandwidth through protocol agnostic means while simultaneously advocating protocol-specific prioritization schemes that ensure the best possible experience for all people and all protocols. These two goals are completely separate and completely honorable.
Thursday morning I sat on a panel at the Innovation 08 Net Neutrality event at Santa Clara University. This came right at the heals of my Brussels trip where I gave a presentation on Net Neutrality to the some members of European Parliament and various industry folks. The jet lag wasn’t so bad but the bigger problem for me was missing my 6 year old daughter’s first big singing solo at her school which had to be at the same time as my panel. I spent a lot of time training her so it was certainly a big disappointment for me. The jet lag certainly did have a lot to do with why this blog wasn’t posted earlier yesterday.
Richard Whitt, George Ou, Ron Yokubaitis, Richard Bennett, Jay Monahan
Photo credit: Cade Metz
The story didn’t get too much coverage (yet) but here we have some coverage from Cade Metz. I guess it’s a slight improvement because Metz at least didn’t try to falsely insinuate that I was against transparency for Comcast this time. It was gushing with love for Google but at least he quoted me accurately and got my point across.
Ou is adamant that – whether it (Net Neutrality rules) forbids ISPs from prioritizing apps and services or it forbids them from selling prioritization – neutrality regulation would actually prevent things like video and voice from flourishing on our worldwide IP network. “If you forbid prioritization, you forbid converged networks,” he said. “And if you forbid converged networks, you get a bunch of tiny networks that are designed to do very specific things. Why not merge them into one fat pipe and let the consumer pick and choose what they want to run?
This is such an important point because latency/jitter is a killer for real-time applications like VoIP, gaming, and IPTV. As I showed in my research, even mild usage of BitTorrent on a single computer in a home can ruin the experience for everyone in that home. If prioritization technology is banned in Broadband, then we’ll simply end up with less functional broadband and we’ll have a statically separated IPTV service. With a converged IP broadband network that delivers IPTV and Internet access, the consumer gets a massive converged pipe and they have the power of control at their fingers when they turn off the IPTV to free up all that bandwidth for their Internet service. If the Government prohibits intelligent networks that guarantee quality of service, ISPs will be forced to separate their TV and Internet pipe with a fixed boundary and the consumer gets left with a permanent slow lane rather than getting a slow lane plus a fast lane that they can dynamically allocate to their TV or their Internet.
Metz also couldn’t resist from taking a personal jab at me and Richard Bennett:
“The panel also included George Ou and Richard Bennett, two networking-obsessed pals who have vehemently defended Comcast’s right to throttle peer-to-peer traffic, and Whitt received more than a few harsh words from Ou.”
The disparaging tone was uncalled for and when you put it side by side with the treatment he gave to Google, the bias is blatantly obvious and journalistically unprofessional.
Metz swooned for Google’s
“The question was raised by the top level management at Google: What do we think about network neutrality Ð about this notion that broadband companies have the power to pick winners and losers on the internet?” Whitt explained. “One position was that in the environment [proposed by Whitacre], Google would do quite well.”This side of the argument said: We were pretty well known on the internet. We were pretty popular. We had some funds available. We could essentially buy prioritization that would ensure we would be the search engine used by everybody. We would come out fine Ð a non-neutral world would be a good world for us.”
But then that Google idealism kicked in.
Idealism huh? Too bad Metz left out the part where Google’s Whitt admitted that they were against network intelligence and enhanced QoS even though he refused to answer a simple yes/no question on whether he and Google support the actual Net Neutrality legislation. Make no mistake, Google’s position is based on crippling their video competitors in the IPTV market which is critical to adding competition in the Cable and Satellite TV market which is far more expensive and relevant to every-day Americans. It has nothing to do with Google idealism.
Ron Yokubaitis went off with the typical spiel about how DPI (Deep Packet Inspection) was all about violating user privacy, reading consumer’s email to inject ads, a tool of the big bad RIAA/MPAA for figuring out what song or movie you’re downloading, and how this was similar to communist China. Yet DPI has nothing to do with reading email since that is a function of spam filters and it has nothing to do with violating people’s privacy. DPI is merely a mechanism that analyzes which protocol someone is using and it really isn’t a method used by the MPAA and RIAA.
I also pointed out that it’s ironic that it is companies like Google who wants to inject ads and data mine your Gmail account and it was ironic that we bash the telecoms when it’s companies like Google that censors information from the Chinese people. I’m also reminded that people were imprisoned in China for simply speaking out because of search engine providers like Yahoo turning them in to the Government. Perhaps this wasn’t a great tangent for me to go off on but I get irritated by the wrongly focused attacks on ISPs when it’s often more appropriate for search engine companies.
Richard Bennett also posted something about this event and wrote
What really happened is this: Google has invested hundreds of millions of dollars in server farms to put its content, chiefly YouTube, in an Internet fast lane, and it fought for the first incarnation in order to protect its high-priority access to your ISP. Now that we’re in a second phase that’s all about empowering P2P, Google has been much less vocal, because it can only lose in this fight. Good P2P takes Google out of the video game, as there’s no way for them to insert adds in P2P streams. So this is why they want P2P to suck. The new tools will simply try to convince consumers to stick with Google and leave that raunchy old P2P to the pirates.
I’m not so sure if Google really feels threatened by P2P since P2P cannot deliver a good on-demand streaming experience beyond 300 Kbps or whatever the common broadband upstream speed is. That’s the problem with out-of-order delivery from a bunch of peers that may or may not be there unless it had several times more seeders than downloaders. Since the normal ratio is several times more downloaders than seeders, you simply can’t do high-bandwidth in-order delivery of video. This is why you’re not seeing YouTube take a dive in popularity and every instant-play site uses the client-server CDN delivery model.
The main reason P2P is so popular is because there is so much “free” (read pirated) content available. The actual usability and quality sucks compared to commercial video on demand services. Not only do you get lower quality and lower bitrates in the 1 to 1.5 Mbps range, you have to wait hours for the video to finish before you can start watching it and it even hogs your upstream bandwidth in the process. Legal for-pay services such as Netflix all use the client-server CDN (Content Distribution Network, caching technology) delivery model because it offers an instant play experience and the video quality is much higher quality at 4 Mbps. Other services like Microsoft’s Xbox Live Market Places use client-server CDN to deliver roughly 6.9 Mbps.
While it may be possible to get 6.9 Mbps from a P2P client, it’s rare that a single Torrent will be healthy enough to hit that speed and it certainly won’t arrive in order making it impossible to view while you download.
In my last article on “Why BitTorrent causes so much latency and how to fix it“, I talked a little about packet sizes and some people asked me what the optimum packet size for maximum throughput is on a DSL broadband connection. In this article I’m going to show you how to optimize your DSL broadband connection in Windows Vista using the netsh command. I will also show you to find the optimum compromise between a Jumbo Frame LAN and a DSL PPPoE broadband connection.
It turns out that the default 1500 BYTE packet isn’t optimum for DSL PPPoE based broadband connections because of the additional 8-BYTE PPPoE overhead. If you send out maximum size packets using 1500 BYTES, then each packet will fragment in to two packets because it’s too big for the PPPoE DSL connection so you end up with double the packets and double the overhead.
Note that Cable Broadband users don’t need to change this because they don’t need to deal with the PPPoE overhead.
The first thing you need to do is open up a command prompt elevated to administrator. For Windows Vista, you do this by hitting the start button and typing the command “cmd” but don’t hit enter yet. Move your mouse up to the top of the menu and right click on cmd.exe and then left click “Run as administrator”.
The next step is to type or cut-paste the command:
netsh interface ipv4 set subinterface “Local Area Connection” mtu=1492 store=persistent
Note that this assumes you are using your typical default LAN connection which is usually labeled “Local Area Connection” but it could be labeled something else if you have more than one Ethernet port or you relabeled the connection name to something more specific. If it is called something else, then you need to adjust the command above accordingly.
This command takes effect immediately and there is no need to reboot. The “store=persistent” command tells Windows Vista that this is a permanent setting. You can confirm after any reboot with the following command which can be done in a normal console.
netsh interface ipv4 show subinterfaces
Windows XP users can use a tool called DrTCP.
Optimizing for Jumbo Frame and DSL PPPoE Broadband
The 1492-BYTE MTU is optimum for DSL broadband but not optimum for high-speed gigabit LAN. In fact, there are times when you want to use Jumbo Frames to avoid the problematic Vista MMCSS packet per second throttling behavior. But a lot of Jumbo Frame implementations are limited to 4082-BYTE Ethernet Frames. So the trick is to use an MTU size that is a multiple of the broadband-optimized packet. The IP packet is always 28 bytes smaller than the largest MTU so the broadband-optimized IP packet is 1464 Bytes. We multiply that by 2 and we get 2928 so the proper MTU size would be 2956 which is still smaller than 4082. So for users who want to have a good compromise between Jumbo Frames and PPPoE DSL, they can use an MTU of 2956. That means the LAN won’t fragment the packet but the broadband connection will do a clean two-for-one split that results in optimum size packets.
So the command to implement the 2956 MTU would be:
netsh interface ipv4 set subinterface “Local Area Connection” mtu=2956 store=persistent
You can actually test for larger numbers to see if your setup supports larger jumbo frame sizes like 6K or 7K Jumbo Frames. If it does, then you might be able to use an MTU of 5884 which is derived from 4 times 1464 plus 28. Here’s an illustration of how this clean fragmentation works
Verifying Jumbo Frame support
You should verify that you have a switch that supports Jumbo Frames and an Ethernet Adapter in the computer with proper drivers. You can enable Jumbo frames by going to the Control Panel, Network and sharing center, View status, Properties button, Configure button, Advanced tab, Jumbo Frame option.
To verify the whole thing by running the following test between two computers on your LAN that are configured to support Jumbo Frames which we’ll call Computer-A and Computer-B.
On Computer-A, run the following command to verify jumbo frame support.
ping Computer-B-IP-Address -l 4054 -f
Note that “-l” in the command above is a dash lower-case L.
If you get a proper reply without an error message, that means your computers and Ethernet Switch supports 4082-BYTE MTU.
Anyone VoIP or online gamer who has a roommate or a family member who uses BitTorrent (or any P2P application) knows what a nightmare it is when BitTorrent is in use. The ping (round trip latency/jitter) goes through the roof and it stays there making VoIP packets drop out and game play impossible. I’ve personally experienced this and many of my friends have experienced it. When I had a roommate in 1999 and I had my first broadband connection, my roommate started using the hot P2P (peer-to-peer) application of its day called Napster. As soon as he started sharing, my online game would become slow as molasses and I’d experience a huge 500ms second spike in round-trip latency and any gamer knows that pings over 100 are problematic.
When it’s just myself using BitTorrent, I can pause it while I make VoIP phone calls or when I want to game. But when it’s time to ask your family member or roommate to stop using their application, it becomes awkward because why should they stop using their application just so you can use your application? My roommate paid for half the broadband fees so I couldn’t always tell him to stop and he couldn’t always ruin my latency making it a contentious situation for both of us. What makes it even more frustrating is that there’s plenty of bandwidth capacity to theoretically satisfy both our needs, but something about these P2P applications make them them very bad housemates.
I decided to do some research on this and ran a series of tests which resulted in some very interesting data. I fired up various applications at varying rates of bandwidth utilization in the upstream and downstream direction and I measured ping to my first router hop (ISP router) beyond my home router. I ran continuous pings to the ISP router to check the round-trip latency above the normal baseline latency of 12 ms and plotted out the results. This test methodology measures network jitter (the variation of delay) on the “last mile” which is what I’m primarily concerned about because that’s where most of the damage is done by local usage of P2P applications.
BitTorrent downloads cause huge amounts of latency
The first set of tests I did were download tests. First I downloaded a file using HTTP which ran at 260 KB/sec (2.08 Mbps) which is roughly 87% of my peak download performance. I then set BitTorrent to also download at 260 KB/sec to compare the effect on my ping times. To my surprise, there was a significant difference in the amount of ping increase between HTTP downloading and BitTorrent downloading despite the fact that both applications were downloading at the same average rate.
When you look at the two graphs immediately above, you see that BitTorrent causes an average increase in ping of more than 117 ms (milliseconds). Note that those 6 ping spikes in the graph were actually bad enough that they caused the ping to time out which means the ping was higher than one second. HTTP on the other hand didn’t increase the ping nearly as much but it still caused an average of 46.7 ms increase in ping and it only peaked at 76 ms. So how is this possible when both applications are using the same amount of bandwidth? I did some thinking and this is the hypothesis I came up with.
Burst traffic fills up transmit queues
The difference in ping time is caused by the fact that HTTP is a single source of data whereas BitTorrent is about 20 sources of data. Incoming data from from multiple sources via the fast core of the Internet can sometimes clump closely together when multiple sources happen to transmit data around the same time. So in the period of 20 ms (the interval between typical VoIP packets), up to 200 max-size 1472 BYTE packets can occasionally build up in the DSLAM transmit queue (blue square in illustration above labeled as “download queue”) causing my ping requests to time out above the 1 second mark. But on average, we get around 23 of these packets causing sitting in the DSLAM transmit queue causing an average increase in ping of 117 ms. When there is a single transmitter, it might burst and clump packets close together but it won’t be at the level of 20 transmitters.
With HTTP causing 76 ms of downstream delay, that means 15 of these 1472-BYTE packets are sitting in the DSLAM transmit queue causing a less extreme increase in ping. This is still problematic for VoIP communications and it can certainly ruin online gaming. So despite the fact that there is plenty of remaining bandwidth for my VoIP or online gaming traffic, it’s the non-uniformity of the incoming Internet traffic that causes my VoIP phone calls and games to perform badly. Unfortunately since this is the downstream we’re talking about, the consumer can’t do much about it on their own end of the pipe because the delay is at the DSLAM which belongs to the ISP.
The only way to fix this problem is for the ISP to implement intelligent packet scheduling and application prioritization at the DSLAM to re-order those VoIP or gaming packets to the front of the transmit queue. With packet prioritization (generally referred to as QoS), your family member, your roommate, or your own video downloads won’t need to be stopped and they won’t interfere with your VoIP or gaming applications which makes everyone happy. Unfortunately, these types of QoS services may never see the light of day if poorly conceived Net Neutrality legislation gets passed that ban the sale of packet prioritization.
BitTorrent uploads cause excessive jitter
BitTorrent or P2P uploads also cause a lot of upstream jitter. I compared various types of upload traffic patterns to see what kind of increase in the upstream ping times would result. First I tried running BitTorrent with a 47 KB/sec (376 Kbps) bandwidth cap which was about 90% of my upload capacity, then I ran BitTorrent with a 28 KB/sec (224 Kbps) bandwidth cap at 54% of my upload capacity, and then I ran BitTorrent with a 10 KB/sec cap at 19% of my upload capacity.
In either the 28 KB/sec test or 10 KB/sec test, I’m not being greedy by hogging all the available upstream bandwidth and I’m leaving more than the 11 KB/sec of bandwidth needed for VoIP and online gaming. Yet I found that the additional ping caused by BitTorrent uploads (seeding) was unacceptable for gaming and problematic for VoIP applications. Even when I severely limited upload throughput to 10 KB/sec, it didn’t reduce the ping time spikes although it reduced the frequency of those spikes. However, even fewer spikes in ping time can pose the same problems for VoIP applications because they have to adjust their buffers to account for the worst-case network conditions. This would seem to indicate that BitTorrent is bursting packets rather than releasing the packets in a uniform and evenly spaced stream.
Next I tried using FTP at full throttle and I managed to get an FTP session going at 47 KB/sec (90% of my peak load) yet the jitter caused by FTP at this extreme rate of throughput was less than the jitter caused by operating BitTorrent at an average of 10 KB/sec. This would seem to indicate that FTP is outputting data in a more consistent manner that BitTorrent.
Lastly, I ran some ping tests during some VoIP phone calls using the Lingo service (competitor of Vonage). I had set Lingo to use the uncompressed G.711 which uses 11 KB/sec for both upload and download which makes it very comparable to BitTorrent uploading at an average of 10 KB/sec. But as soon as I ran the ping tests, I was shocked to see virtually no increase in ping times. I realized that this is because the VoIP ATA (Analog Telephony Adapter) device pulses small packets at exact intervals of 20 milliseconds at 50 times a second.
Smoothing out the packet spacing to reduce jitter
After running these tests, I am beginning to conclude that it isn’t so much the amount of data that causes excessive jitter; it’s the uniformity of data transmission. If the transmissions are spaced evenly, other packets from other applications can slip in between the packets rather than getting stuck behind multiple packets in the transmit queue. So would it be possible to engineer BitTorrent to transmit data uniformly and what would be the effect? I came up with the following chart to illustrate what this could mean for peaceful coexistence between VoIP/Gaming and BitTorrent/P2P.
I calculated that the max-size 1472 BYTE packet takes 28.3 milliseconds to transmit over my 52,000 BYTE/sec broadband uplink. If BitTorrent bursts 3 packets upstream over a 100 Mbps FastEthernet LAN connection, they will all sit in the upload queue of my DSL modem for 85 ms. Meanwhile, my VoIP or gaming packets get stuck behind those three packets for an additional 57 ms in the queue. This is shown in the first example in the illustration below with large 1472-BYTE red packets representing BitTorrent and small 223-BYTE green packets representing VoIP.
In the second example, I show what happens if BitTorrent simply spaced their transmissions evenly. It’s still less than ideal but at least it significantly reduces the jitter for VoIP or gaming packets.
In the third example, I show the ideal scenario where BitTorrent would reduce its packet size to 815 BYTES or less and pulse them out in 20 ms intervals at 50 packets per second. BitTorrent could essentially create a “VoIP friendly mode” that allows VoIP packets to fit cleanly between the BitTorrent packets and the increase in jitter would be no greater than 15.7 ms and would typically average around 8 ms increase. BitTorrent could also have a “Game friendly mode” that uses 679-BYTE packets at 60 packets per second.
Now it is possible to solve this problem on the network level by prioritizing VoIP and gaming packets in the home DSL modem upload queue. Unfortunately, I don’t have administrative access to the modem and implementing VoIP or gaming prioritization on my home router seemed to have no effect. Packets in the home router get forwarded as soon as they arrive with 100 Mbps Ethernet on both ends so there is nothing to reorder in the queue. More advanced business-class routers like those from Cisco will allow you to configure the speed of the FastEthernet connection to match your DSL throughput so that the queue will migrate from the DSL modem to the router but this isn’t very practical for most people. So it would make sense for application writers to try and make their application work as well as possible on the majority of home networks and broadband networks without QoS.
While modifying BitTorrent or the P2P application may not significantly fix the downstream problem, it would definitely fix the upstream jitter problem which means that people will be more willing to seed and contribute to the health of BitTorrent. The download jitter may improve if the dozens of peers sending information to me did it in a more uniform manner, but it is still possible that those peers will transmit at the same time which still causes packet clumping and bursting.
So why would BitTorrent or any P2P application vendor want to do this? Why couldn’t they just say “it’s not my problem”? Because it would fix their reputation as a bad housemate and more people would be willing to seed if it didn’t interfere with their other applications like VoIP or gaming. Corporate IT departments may also ease their ban on the technology if it doesn’t trash their Internet connection for the entire office. I am certainly a huge proponent of network-based solutions for dealing with jitter problems, but improvements in the application can act to further reduce network jitter even in the presence of existing network-based solutions. Furthermore, the vast majority of consumers do not have access to network-based solutions and it only makes sense for a P2P application vendor to cater to this market.
I will try to reach out to BitTorrent corporation to see if there is any interest in this and perhaps someone would be interested in modifying open source Azureus. It would be a very good feature to have or it would be a very interesting academic experiment at the very least.
The whole Comcast issue is being kicked around in the press in recent days because the Max Planck Institute released a study showing the rates of TCP resets happening throughout the world. But this whole issue is being mischaracterized as the “blocking” of BitTorrent and it’s being portrayed as a free speech issue when it is nothing of a sort.
Richard Bennett explained why this shouldn’t be considered blocking and Andrew Orlowski wrote a pretty good editorial raising the concern that this is trivializing real free speech violations. One thing in Orlowski editorial that really caught my eye was the continual misrepresentation of the facts by people like Ben Scott of the Free Press.
Ben Scott said: “I would disagree with your characterization of RST packets. This is in fact blocking by definition. It think your analogy is inapt. It would be the equivalent of traffic stops sending me back home to start driving to work all over again.”
Ben Scott explained to Orlowski that his portrayal of the facts are backed up by “experts” so I have to wonder who these experts are? Jon Peha? Peha demonstrated at Stanford and in his FCC filing that he doesn’t even understand the multi-stream principle or the auto-restart feature in P2P applications like BitTorrent. Even when every TCP stream (often up to 40 streams) of a P2P download are being continuously reset and a full blockage is in place, you never start all over again. Jon Peha and Ben Scott would have you believe that BitTorrent forces the user to manually resume like a phone call that’s been disrupted and that BitTorrent has to restart a file transfer from scratch. While I and others like Richard Bennett have pointed this out to the FCC and directly to Jon Peha, these deceptions continue to be propagated.
Then there are other “experts” like Software Tester Rob Topolski and some unnamed professors that Free Press lawyer Melvin Ammori likes to cite. They claim that BitTorrent can “only” use 4 upstream TCP flows per Torrent which on the face of it is laughable because 4 is a soft limit that can easily be exceeded. The bigger problem with this claim is that uploads aren’t the issue since uploads continue on indefinitely and uploads aren’t the endgame. What should be looked at are the number of completed file transfers which is the real endgame and the number of upload streams an individual P2P seeder can offer is irrelevant. The uploaders are merely contributing to a pool of available P2P servers so what matters to the completion of file transfers is the time it takes to download a file.
If Jon Peha, Rob Topolski, and Ben Scott are actually correct in their description of Comcast’s Sandvine system, then a good case can be made that what Comcast is doing is not reasonable network management. But the arguments they make are easily proven wrong and BitTorrent corporation or any P2P software company would take offense at the claim that their product is incapable of automatic resume and incapable of resuming where they stopped.
A traffic light does “block” cars at stop signs for a short duration and even longer at stop lights. But when that stop light is broken and it defaults to a stop sign, overall traffic begins to flow much slower. If we take the stop sign out and let people go whenever they feel like it and there’s a collision, then cars gets ensnared in a severe traffic jam. This is precisely what happens when you have no reasonable network management and you get less effective throughput so applications (including P2P) are actually sped up by reasonable network management.
It’s true that Comcast’s management scheme has some problems because it harms rare torrents that are unhealthy to begin, but there are workarounds for these problems and the problems are overblown. I even showed how you can force Comcast to seed you 100 times faster without consuming your own upstream or your neighbor’s upstream. So while it’s reasonable to argue that Comcast’s P2P management system is far from ideal and needs improvement (which Comcast promises to do by the end of this year), it’s flagrant deception to portrait this as application blocking and it’s ridiculous to turn this in to a free speech debate.
As many of you reading this blog probably already know, Comcast has been disconnecting a certain percentage of TCP streams emanating from BitTorrent and other P2P (peer-to-peer) seeders. This effectively delays and degrades the ability of Comcast customers to seed files using P2P applications. For normal healthy Torrents that are distributed across multiple users and multiple ISPs, losing a few seeders intermittently isn’t too noticeable. But for rare Torrents or Torrents that have to originate from a Comcast broadband customer, this can pose some challenges. The rare Torrent becomes even less reliable than they already are while popular Torrents originating from Comcast’s broadband network take much longer to become healthy.
While Comcast has stated they will try to move off of their Sandvine system that uses TCP resets by the end of this year, there’s no guarantee that they will complete on schedule and there’s no relief in the mean time for customers who are having a tough time seeding their files for distribution. Even without the challenges posed by TCP resets, seeding a torrent file is still problematic and burdensome. Not only does the seeder have to turn his/her computer in to a server, they must also allocate significant portions of their upstream bandwidth – as well as their neighbor’s bandwidth which they share – to seeding while providing relatively minimal capacity to the Torrent.
The opposite challenges of client-server and peer-to-peer
One way around this is to use the gigabyte of web space that Comcast provides you, but there are downsides to a pure HTTP client-server model. While an HTTP client-server model works great because it’s persistent and the web server offers much higher capacity than any individual seeder, there’s a fixed amount of web server capacity which quickly divides as the number of clients increases. That means client-server works great when there are fewer clients but it degrades in proportion to the number of clients who want to use the server.
The P2P model has just the opposite problem in that it works better and better as more clients/peers are added because each client becomes an additional server, but it’s horrible when there are few clients and quickly accelerates to death as it loses popularity. The less popular a Torrent is the less likely it is to attract peers because people tend to avoid less popular Torrents fearing slower speeds or being stuck with a Torrent that can never complete.
Combining the two file distribution models
Given the fact that client-server is weak where peer-to-peer is strong and the fact that client server is strong where peer-to-peer is weak; I thought what if we combined the two models in to a single client or a single protocol that gave you the best of both worlds! All we need to do is have a BitTorrent pull from an HTTP source that remains persistent so that there will always be a fast seed to fall back upon even when there are no other peers in the Torrent.
This not only guarantees a survivable Torrent which attracts more peers, it also provides a rocket boost in performance because web servers tend to be 50 to 100 times faster than any Broadband consumer seed. The original seeder would be able to seed and forget and avoid saturating his own upstream connection as well his neighbor’s upstream connection. Given the fact that a typical Cable ISP operating DOCSIS 1.1 protocol allocates 1.25 megabytes/sec of upstream capacity for 200-400 users in a neighborhood, a hybrid web seed would not only benefit consumers but the ISP as well because it alleviates the biggest bottle neck on Cable ISP’s network.
But before I could celebrate my “new” idea and invention, Ludvig Strigeus (creator of the best and most popular BitTorrent client uTorrent) with tongue placed firmly in cheek explained that I was a few years late in my discovery and that the concept of web seeding was already proposed by John Hoffman and “DeHackEd”. Furthermore, the feature is already implemented in uTorrent as well as the official BitTorrent client called “BitTorrent” and Azureus all support web seeding. Those three clients command the lion’s share of BitTorrent clients and there is a full list of compatible clients here.
Note: It’s not your imagination that BitTorrent looks just like uTorrent with a few extra features; BitTorrent Corp acquired uTorrent from Ludvig Strigeus at the end of 2006 because it was technically superior.
Web seeding in practice
One of the problematic Torrents that has been hotly contended in the Comcast TCP reset fiasco is the King James Bible. The is a classic example of something that isn’t popular in BitTorrent because it’s relatively small and readily available on the world wide web over HTTP but it still serves as a good example of what happens to rare Torrents using the pure peer-to-peer model. I created a web seeded Torrent for the King James Bible using Richard Bennett’s gigabyte of web space provided at no extra charge by his ISP Comcast to serve as a persistent and lightning fast seeder. I also took this traditional P2P Torrent of Leonardo da Vinci’s complete Notebooks and converted it to this web seeded torrent.
The difference in speed for this rare torrent was an astounding 100 fold increase in performance. The reliable of this Torrent also shot up because it doesn’t depend on an individual seeder in a residential Cable broadband network who may shut off his computer or P2P application or get TCP reset. The web seed sits outside of the Cable broadband network in some ultrafast server farm that is spared from TCP resets. The ultimate irony here is that Comcast who is known to slow down everyone’s BitTorrent seeds is now giving you a free 100 fold boost in performance! Not only that, but they’re alleviating their last-mile congestion as well as giving you more reliability.
How to create your own web seed
By now I’m sure you want to know how to create your very own web seed Torrent. Simply fire up a copy of uTorrent or BitTorrent and press Control-N or go to the File menu and hit “Create new Torrent”.
You will see the screen on the right and all you need to do is choose a source file from your hard drive, enter in a list of trackers you plan on using, then hit “Create and save as”.
Once you save the .Torrent file, the next step is to copy the source file (the book, video, or song) you want to distribute up to a publicly accessible web server such as the free web space Comcast provides their customers. Many other ISPs also provide free web space if you look around or ask.
Update 5/1/2008 – Robb Topolski posted a comment for this article saying that the example I’m using is actually the getright web seeding specification. Thanks for the clarification Robb!
At this point in time there is no automated support for creating web seeds so you’re going to have to use a program called BEncode Editor to modify the .Torrent file you just created. Once you open up the .Torrent file with BEncode Editor, you will see the following window.
All you need to do is hit the + button and add a “url-list” to the root. You will see the following window where you will add a list of URLs that have a copy of the file that you want to distribute. Note that if you want to distribute multiple files, you point to the folder containing all those files and not just the file. Also note that Apache web servers don’t properly work for filenames with spaces for web seeding while the IIS 7.0 server I tested did work for web seeding even if the filename had spaces in it.
The final step is to publish the .Torrent file on your preferred tracker. For the purpose of this discussion, I used ThePirateBay.org as the tracker and uploaded the files there after creating an account on their site. After I finished uploading the .Torrent file, I got this page which gives you a public link for the .Torrent that you can distribute on a website or via email. You can also email people the .Torrent file directly or you can direct link to the .Torrent file from the tracker. At this point you don’t even need to have your computer participate as a seeder and the seed is extremely fast and reliable. You can even publish the link to the HTTP site if people don’t want to or can’t (office policy) use the Torrent version and those people can download from their web browser directly.
The final outcome is that Comcast doesn’t have to worry about you hogging all the upstream bandwidth in your neighborhood, you free up your computer and bandwidth, the seed runs up to 100 times faster, and now everyone’s happy because of a technical solution.
Update: How Comcast customers can seed 100 times faster and bypass TCP resetsThe FCC hearing at Stanford University on April 17th 2008 was filled with inaccurate testimony from various witnesses. Since that testimony seems to be carrying significant weight both on Capitol Hill and in the media, I feel compelled to set the record straight. I have filed a copy of this letter on FCC docket 07-52.
Problems with Jon Peha’s testimony
Jon Peha testified that BitTorrent was like a telephone and implied that if a TCP reset is used by Comcast to stop a TCP stream, then that constituted a blockage of BitTorrent. Furthermore, Professor Peha implied through his telephone analogy that if BitTorrent is blocked, then the user must manually redial to reestablish the connection. These assertions are highly inaccurate and here’s why.
The first problem is that Jon Peha did not understand the multi-stream aspect of BitTorrent or P2P. Peha seemed very surprised immediately before our panel at Stanford when I told him that a P2P download typically used 10 to 30 TCP streams at the same time. His surprised reply to me was “all active?” and I replied yes. The reality is that if a certain percentage of BitTorrent TCP streams are reset and temporarily blocked by an ISP, say 15% for example1, then the “Torrent” (the file that’s being exchanged amongst multiple peers over the BitTorrent protocol) is essentially slowed down by an average of 15%. In other words, the “Torrent” would suffer a 15% partial blockage which is accurately described as a “delay” since the file transfer didn’t actually stop. This would be like filling up a bath tub with 20 faucets and you closed 3 of those faucets. The rate of water flowing in to the tub would slow but not stop.
The second problem with Jon Peha’s testimony is his implication that the user must take some sort of action to resume the BitTorrent connection or else the connection wouldn’t resume. Peha’s assertion can easily be proven false by a simple experiment with BitTorrent. One can easily confirm that BitTorrent will always resume a lost connection within a matter of seconds without any user intervention just by physically disconnecting a network cable on the test machine and reconnecting it. Not only does BitTorrent automatically resume, it picks up where it left off and does not need to start all over again. So even if all TCP streams in a Torrent were simultaneously blocked for a short period of time, it will quickly resume by itself and eventually finish the file transfer. Therefore this is by definition a “delay” and not a “blockage”.
This is not to say that Comcast’s existing form of network management is without problem because it is clear that the system has flaws and unintended consequences like the accidental blockage of IBM Lotus Notes. The use of TCP resets also have a more drastic effect on rare Torrents, which are BitTorrent files that are not popular and have few seeders or other peers who have parts of the file to download from. These rare Torrents aren’t healthy to begin with and a TCP reset can in some cases trigger a complete temporary blockage. The rare Torrent will still get there eventually but it will suffer significantly more than a normal Torrent that is resilient to partial blockage.
It should be noted that BitTorrent in general is not an appropriate file transfer protocol for rare Torrents. BitTorrent tracker sites tend to rank and sort a Torrent based on the Torrent’s “health” which is based on the number of available seeders and pre-seed peers. Users generally tend to avoid the “unhealthy” Torrents on the bottom of the list. Since Comcast offers a vastly superior alternative where they provide 1 gigabyte of web storage space, Comcast customers can use that service to distribute files 10 to 20 times faster than any single Comcast BitTorrent seeder could ever provide. To further illustrate this point, Richard Bennett
Problems with Robert Topolski’s testimony
Robert Topolski also had problems in his testimony. Topolski, a software tester who does not work in the networking field, insists that the TCP reset mechanism isn’t common in network devices and declared that I was wrong in my testimony. In my experience as a Network Engineer who designed and built networks for Fortune 100 companies, it is my experience that the TCP reset mechanism is common in routers and firewalls. For many years, Internet service providers, including LARIAT (owned and operated by Brett Glass, who has filed comments in this docket) have used TCP RST packets to protect the privacy of dialup Internet users. When a dialup user’s call is over, RST packets are sent to terminate any remaining connections. This prevents a subsequent caller from receiving information — some of which might be confidential — that was intended for the caller who disconnected. Thus, the transmission of RST packets by a device other than the one(s) which established a connection — for the purpose of informing the endpoints that the connection has been terminated — is not only commonplace but salutary. Network architect Richard Bennett who works for a Router maker explained to me that TCP resets are the standard mechanism used by consumer routers to deal with NAT table overflow, which itself is typically caused by excessive P2P connections. Are we to believe a single software tester or three networking experts?
The second key problem with Topolski’s testimony is that to my knowledge, he has never provided any forensic data from Comcast in the form of packet captures that can be independently analyzed. Even if Topolski did produce packet captures and we assumed that those packet captures are authentic, one man’s packet captures wouldn’t be a large enough sample to draw any conclusions by any legal or scientific standards. The Vuze data may constitute a large enough sample but the data isn’t very granular because it doesn’t tell us what percentage of reset TCP sessions are due to an ISP versus other possible sources
Furthermore, even if a TCP reset was used by Comcast at 1:45AM, we cannot assume that there was no spike in congestion at 1:45AM. As I have indicated in the past, just 26 fulltime BitTorrent seeders in a neighborhood of 200 to 400 users can consume all of the available upstream capacity in a DOCSIS 1.1 Cable Broadband network. That means less than 10% of the population seeding all day and night can cause congestion at any time of the day. Based on what little evidence presented by Robb Topolski, no conclusions can be drawn regarding the question of whether Comcast uses TCP resets for purposes other than congestion management.
1. The reason I use 15% as the example is because the Vuze data gathered via thousands of user’s computers indicated that a Comcast broadband network typically suffered 14% to 23% for all TCP streams during 10-minute sampling periods. That 23% figure is not restricted to just BitTorrent or P2P traffic and even those TCP resets pertaining to BitTorrent aren’t necessarily from Comcast. It’s quite possible that the reset actually came from the client on the other end or it may have come from a customer-premise router on either end-point trying to conserve on NAT (Network Address Translation) resources. It is undeniable that a certain percentage the TCP reset have nothing to do with Comcast’s network management practices. We really can’t know what percentage of those TCP resets were due to Comcast and figuring out the exact percentage isn’t trivial because there are so many factors to consider.Note: Any assertions on behalf of Brett Glass and Richard Bennett that I have made in this document have been approved by Brett Glass and Richard Bennett.