Search
Tuesday, December 02, 2008 ..:: Home ::.. Register  Login
Blog roll

Topic search

UsersOnline
Membership Membership:
Latest New User Latest: bokonon42
New Today New Today: 0
New Yesterday New Yesterday: 0
User Count Overall: 118

People Online People Online:
Visitors Visitors: 0
Members Members: 0
Total Total: 0

Online Now Online Now:

Blogs
Jul 19

Written by: George Ou
7/19/2008 8:11 AM

Update 8/12/2008 - Robb Topolski has graciously apologized for the personal attacks.

I have to admit that I was surprised at the speed at which I've been attacked on my latest FCC filing "Guide to protocol agnostic network management schemes".  Within hours of my filing, Robb Topolski who now works for Public Knowledge wrote an attack blog "George Ou: Protocol Agnostic doesn't mean Protocol Agnostic" which, along with a series of ad hominem attacks, basically accuses me of being hypocritical.  I won't respond to the senseless personal attacks, but I am going to rebut the core accusation that favoring protocol protection in the context "Protocol Agnostic" network management schemes is somehow hypocritical.

Before we debate the meaning and context of the word "agnostic", we must first ask ourselves what is the underlying problem we are trying to solve.  The answer is that we want a broadband network management system that is more equitable at distributing bandwidth fairly between users in the same service/price tier and we want a broadband network that ensures the best possible experience for every protocol and their associated applications.

Network Architect Richard Bennett put it best when he described network management as a two-phase process.  In his blog he recently wrote:

An ideal residential Internet access system needs to be managed in two different but equally important phases:

1) Allocate bandwidth fairly among competing accounts; and then

2) Prioritize streams within each account according to application requirements.

Phase 1 keeps you from being swamped by your neighbor, and keeps you from swamping him, and Phase 2 prevents your VoIP session from being swamped by your BitTorrent session.

So to address the first problem, we need a way to measure bandwidth consumption between these "competing accounts".  The common solution on the market place today computes bandwidth consumption implicitly and indirectly from the protocols passing through the network.  This method of looking at the protocol to determine bandwidth consumption was initially chosen because it is computationally simpler and cheaper than trying to track usage statistics of tens of thousands of simultaneous application flows in real-time. While this solution is mostly accurate, it isn't completely accurate.

As the identification of protocols become increasingly difficult because of protocol obfuscation techniques and as the cost of computational power declines, we're starting to see a shift towards network management systems that don't use protocol analysis to determine which users consume the most bandwidth.  These new systems compute bandwidth distribution by tracking the bandwidth consumption of every flow on the network without looking at what protocols are passing through the system and these are the systems we refer to as "Protocol Agnostic".  But being "agnostic" strictly applies to this first phase of network management.

The second phase of network management is to ensure that every protocol works well.  This requires knowledge of what protocols are passing through the system so that they can be given the necessary protections from other protocols sharing the same network.  Since protocols and their applications have unique network requirements, we must identify the protocols being used so we can give them the necessary protections.  It would be foolish to apply the concept of agnosticism towards this phase of management because doing so would make this phase worthless.

The reality is that a network can't simply be a dumb pipe if we want every protocol/application to work well and no one has ever claimed that "protocol agnostic" should mean a dumb pipe.  P2P applications have the ability to not only grab nearly every bit of bandwidth available regardless of how much capacity there is; they can cause significant problems for real-time applications like VoIP or online gaming applications even when the P2P uses a small fraction of the network.  I have published experimental data showing that even low-throughput mild usage of P2P applications can cause significant spikes in network latency and jitter (latency and jitter explained here) which can severely degrade the quality of real-time applications.  Even without any knowledge of how networks operate, anyone who has ever tried to play online FPS (First Person Shooter) gaming or use a VoIP phone while a P2P application runs on the same broadband connection understands how miserable the experience is.  In fact, this is a common source of contention between roommates and family members.

This is precisely why Comcast needs to protect VoIP traffic from service providers like Vonage.  Even though Comcast has nothing to gain because their own telephony service runs outside of the Broadband network on a different frequency, these additional efforts to ensure proper VoIP operation are actually helping a competing telephony service.  In theory, Comcast could just leave this issue alone and let Vonage and other competing VoIP services suffer while its own telephony service remains immune.  In practice, many Comcast's customers will blame Comcast if they had any problems once the new system is in place and Comcast needs to do everything it can to make their customers happy.

Robb Topolski claims on behalf of me that these problems are somehow caused by Comcast when he says that "George has shown that Comcast’s proposed Protocol Agnostic scheme has unacceptable side effects".  Aside from the fact that I've said or shown nothing of a sort, it's ridiculous to paint these noble and reasonable efforts in to something sinister.  The Free Press has used similar fear mongering tactics to discredit these new network management efforts by Comcast by questioning why Vonage needs these special deals with Comcast in order for their traffic to go through insinuating that this is some sort of protection racket.  The fact of the matter is that Comcast not only doesn't block Vonage, they're trying to prevent Vonage from being blocked by self-inflicted damage whenever the user tries to use aggressive protocols like P2P simultaneously with VoIP.

But these advocates of strict Net Neutrality and the "dumb" Internet continue to shout that this is tantamount to "discrimination" against P2P protocols if we provide protocol protection for VoIP or any other real-time applications.  They'll often ask "why can't you give P2P the same protection" but this question is silly when you look at the reality of the situation.  P2P applications don't need this kind of protection and they're thriving at 30 times faster than the VoIP application on a 3 Mbps broadband connection.

If we take an objective look at the situation, the P2P application is already consuming over 96% of the network capacity so would it wise to mandate a dumb network that allowed the P2P application to consume 98% of the network and completely break the VoIP application?  If the network could only fairly allocate 2 Mbps of bandwidth to its broadband customers during peak usage ours, would it be so wrong to expect the P2P application to drop down to 95% of the broadband link while the VoIP application remains at the same miniscule 87 Kbps bitrate?  I guess according to Robb Topolski, this is somehow unfair because the P2P application is forced to slow down while the VoIP application doesn't slow down.

If this is the kind of petty game we're playing, we could simply have a network management scheme that allocates "equal" amounts of bandwidth to P2P and VoIP whenever both protocols are active which means both protocols will operate at 87 Kbps and we could call that "equality".  Obviously no one would want that kind of system and there's no need to force this level of equality so the P2P application is allowed to consume whatever remaining bandwidth is left over from the VoIP application.  But taking what's left over is beneficial to the P2P application because the leftovers are at least an order of magnitude more than what the VoIP application gets.

Robb Topolski then pulls out the DPI (Deep Packet Inspection) Bogeyman and claims that any sort of protocol aware system would violate user privacy by snooping beyond the packet headers and looking in to the user data.  That's nonsense on multiple levels because a system wouldn't necessarily need to look at user data to determine the protocol being used and even if it did, there's nothing wrong with an automated system that parses through user data.  In fact millions of emails a day are parsed by anti-spam systems to prevent your email inbox from being inundated with spam but we don't consider that an invasion of privacy because of the context and scope of the system.

But the smart network management system could simply look at the protocol headers without parsing through any of the user data to determine what protocol is being used because it doesn't have to fear protocol masquerading techniques.  This is because the protocol agnostic part of the system already tracks usage statistics which allows the network operator to implement a priority budgeting system where everyone gets a certain amount of priority delivery bandwidth.  So for example, if everyone had a daily budget of 12 hours of priority VoIP service or 12 hours of priority online gaming service or 3 hours of priority video conferencing, they're free to squander that budget in 30 minutes with a burst of high-bandwidth P2P usage masquerading as VoIP traffic.  After the daily allowance is used up, all traffic would go through but be treated as normal priority data.

Chief BT researcher Bob Briscoe who is leading an effort to reform TCP congestion control at the IETF actually explained this concept of saving up for priority quite nicely and network architect Richard Bennett has also spoken positively about this type of a system.  Priority budget schemes would actually help solve the fundamental fairness problem where some users consume hundreds of times more volume than the majority of users even though everyone pays the same price.  The network would essentially give the low-usage real-time application customers a responsive delivery service while the high-usage P2P customer would get all the volume they want and everyone would get what they want.

When we focus on the ultimate endgame of fairness, it isn't hypocritical to advocate equitable distribution of bandwidth through protocol agnostic means while simultaneously advocating protocol-specific prioritization schemes that ensure the best possible experience for all people and all protocols.  These two goals are completely separate and completely honorable.

Tags:

5 comments so far...

Re: Fairness is the ultimate end-game in network management

George,

Can go back a couple of steps and explain why you believe an ISP and not the consumer is responsible for item 2 - Prioritize streams within each account according to application requirements? How do you get to the conclusion "This is precisely why Comcast needs to protect VoIP traffic from service providers like Vonage."

It seems to me that an ISP is not and cannot be in a position to fully implement this kind of QoS. I'm not familiar with the ins and outs of overall broadband market, but I am familiar with enterprise networks that perform QoS management and with my own home network. In the enterprise network, the same IT department that manages the network also manages the host devices and the software they run. If someone ran software that was highly disruptive to the network, then it would be a policy violation for installing unauthorized software or the network would have to be improved.

For my home network, a router I own and manage performs NAT and multiplexes my and my wife's computers to one IP address that my ISP provides. Even if my ISP wanted to manage my modem, they do not have the policy information to correctly prioritize and throttle streams because the policy should consider which host the application is running on. My wife's comm's should take highest priority, except for my work VPN. I would also expect a VPN would also knock out any ISP-level management because it could not differentiate voip, streaming video for a training session, or a large file tranfer when they are encapsulated in IPsec or SSL.

I agree that item2 is a concern, but I think that ISPs should stick with item1. For item2, the ISP may support a prioritization and tagging scheme, but the prioritzation policy and identification should be the responsibility of the applications and routers on the customer side of the demarc. Alternately, the ISP could try to sell me a service where they own and manage more network and computing devices, but this would be a very, very hard sell.

By anon on   7/20/2008 1:48 PM

anon, prioritization works in 2 directions

Anon, there are multiple reasons why the ISP shouldn’t just say “it’s not my problem” and leave it to the consumers.

Prioritization works in 2 directions and there's a limit to how much good you can do from either end. I explain this in the paper http://www.itif.org/index.php?id=162. You can exert control on bandwidth consumption on either end in either direction but you can only clean up jitter on the transmit side. So you need cooperation on both ends of the Broadband link.

The second issue is that the vast majority of consumers do NOT have the expertise nor will to configure any kind of QoS. There are home routers on the market that claim to have QoS but their effectiveness varies greatly. So when the consumer has trouble with VoIP because their P2P is in use, some of them will blame the ISP and many of them will simply shut off P2P because of the annoying side effects. I’ve published details of those problems here http://www.formortals.com/Home/tabid/36/EntryID/57/Default.aspx.

So when the ISP protects VoIP or other real-time applications, not only does it help the vast majority of consumers, it actually helps P2P because people are less inclined to shut it down and they’ll continue to seed.

Lastly, we could let you, as a minority, opt out of phase 2. However, if you want low jitter on the downstream, you still need the ISP to do that on their end of the Broadband link.

George Ou

By host on   7/20/2008 1:51 PM

Re: Fairness is the ultimate end-game in network management

Thanks for the reply George.

I'm interested to learn how you might address control for jitter that would probably span multiple ISPs. My most common webcam chat goes through three isps - my isp, a backbone provider, and my friend's isp. I would expect jitter to be most affected by the large number of router hops. Are there any numbers describing the impact of the the kind of management you advocate in this multiple provider environment? It would be great to see a cost benefit analysis comparing adding such a management solution vs adding more capacity.

I wonder if an ISP could sell a service that used packet tagging and diffserv. This solution also has the problem that the policy must be described on the consumer side, but good defaults could go a long way. One benefit is that this solution could be much more protocol agnostic and modular.

I think that the biggest challenge with both a diffserv solution and the one you describe is transparency. How does the ISP achieve Powell's fourth freedom - consumers should receive meaningful
information regarding their service plans?

By anon on   7/21/2008 10:53 PM

Anon, jitter caused by queuing delay is primarily a problem with the last mile

Anon,

Jitter caused by queuing delay is primarily a problem with the last mile caused by a huge speed mismatch going both directions. If you look at my paper (http://www.itif.org/index.php?id=162), I have a section on basic queuing theory that explains this. So jitter actually isn't a big problem on the core, the distribution, and the server-end of the network and I also explain this in my paper. If the ISP fixes the download queues on the DSLAM or CMTS which is the last-mile, nearly all of the download problems go away by you're still responsible for upload queuing delays. This doesn't mean the capacity problem goes away; just that the jitter problems mostly go away.

P2P download is actually a much more difficult problem and you can see that with the charts I created in my blog posting on BitTorrent and latency (http://www.formortals.com/Home/tabid/36/EntryID/57/Default.aspx).

Transparency can and SHOULD be achieved externally by treating the ISP as a black box. This way you don't rely on the ISP's word; you're actually verifying it yourself and it would be a totally independent audit. Vuze created a primitive check with their Azureus plug-in that measured RST packets; but the data they gathered wasn't comprehensive enough and they missed the most important metric which is how well uploads and downloads are working and not how many reset packets you're measuring. The software on the end-points can measure every performance metric and aggregate the data on all ISPs and give us meaningful and comprehensive data.

I think there needs to be some consistent and rigorous disclosure standard for the entire industry. The whole industry needs to get together and set some strict disclosure standards. We can’t just have clear advertising on maximum speeds; the minimum speed (excluding maybe the 0.1% statistical outlier) must be disclosed as well. If that means we need government mediation or a little push from the Government, so be it because it will be good for the entire industry.

Right now too many consumers believe that the advertised maximum speed means the minimum speed and this is a level of consumer expectation that companies will never satisfy. This leads to costly regulatory and legal challenges for the entire industry and they just don’t need that. If the public perception is that ISPs rip them off, then this perception is the ISP’s painful reality. Right now no ISP will unilaterally disclose more than they need to because their competitors will gain an unfair advantage for not disclosing everything. If the standard was consistent across the board, then there’s nothing to worry about.

George Ou

By host on   7/22/2008 4:14 PM

Re: Fairness is the ultimate end-game in network management

Another good article on this topic George.

I used to work at an ISP here in Australia where the network admin put in a system based on packet filtering via protocol to selectively slow down users that were torrent sharing to a speed that the owner of the company considered reasonable based on the ADSL profile of the user. This filtering only applied to p2p traffic via torrents. All other protocols were left untouched so downloads via http were at the same speed, as was gaming and everything else. At that time the ISP was one of the few remaining major ISPs here in Australia that had unlimited downloads on their ADSL plans.

While this filtering did place some overhead on other applications (pings typically jumped by 50-100 ms) this had more to do with how the packets were being filtered and the equipment used, rather than what was being done - at least that is what the network admin told me. Just applying this filtering alone reduced the data consumption of the ISP by over 1/3rd while still allowing all other applications to run at full speed. Doing so caused the business to lose around 5% of the user base but as they were the extra-heavy users, that was actually a cost saving to the business.

I'm one of many that agrees with you that there needs to be some form of bandwidth control placed by ISPs at the customer's end to prevent one user taking more than their share of network bandwidth, and to apportion it in such a way that time critical applications such as VoIP and video conferencing takes priority over other apps.

Michael C

By mcoombes on   8/4/2008 6:26 PM

Your name:
Title:
Comment:
Add Comment    Cancel  

Links

Blog_Archive

New_Blog
You must be logged in and have permission to create or edit a blog.

Search_Blog
Print  

Copyright 2008 by George Ou or Justin James   Terms Of Use  Privacy Statement
DotNetNuke® is copyright 2002-2008 by DotNetNuke Corporation