[Discuss] Geoff Huston's talk at NANOG53

[tmetro+blu at gmail.com (Tom Metro)]

Chuck Anderson wrote:
>Tom Metro wrote:
>> Doesn't tunneling provide a bypass to reach
>> competitors who can provide that transition point?
> 
> Tunneling isn't really a viable solution, since it will always perform
> worse than native connectivity.

Sure, but...

> If those rendez-vous points and tunnel endpoints aren't co-located
> along the native traffic route, you are introducing latency and
> possible bottlenecks in the path.

...if your tunneling provider is well connected, you're adding maybe 5ms
for a few more backbone hops? That's going to matter to some, but not to
most.

The bigger problem with tunneling is that it isn't consumer friendly.
Unless some tunneling provider does a really good job of selling the
benefits of tunneling to retail customers, and provides them with
pre-configured routers, it isn't going to turn into a mainstream option.

Unfortunately this barrier may be more than enough to nullify the
competitive pressure tunneling can put on the ISPs.


> If the last-mile providers implement Carrier Grade NATs and other such
> technologies instead of IPv6, then there will be serious impediments
> to the open Internet.

Agreed.


> Those last-mile-providers will be able to extract fees from content
> providers and end-users for "better" connectivity...

This doesn't require speculation, as they've already tried to do this
for bandwidth and "priority" routing. (I'm not sure if any ISPs actually
implemented this, as news of it led to net neutrality legislation
threats, and they seemed to back off.)

I'm not familiar with the details, but supposedly one of the counter
forces to this is the peering arrangements between the major ISPs and
the backbone providers. And Google being a backbone provider of sorts,
wields some power. (Perhaps someone who understand the details better
can elaborate on this.)

Netflix and Akamai have been able to successfully make a case to have
ISPs co-locate their servers as a way to have ISPs reduce their backbone
expenses. These content providers are effectively getting priority
service, but instead of paying the ISPs, the ISPs are benefiting from
reduced costs. Maybe at some point that won't be enough and they'll
start demanding payment as well.


>> I'm not sure I buy that, if you look at the stats for an ISP like
>> Comcast, who supposedly makes more profit on net connections than they
>> do selling video. Maybe in the big picture content (aggregated across
>> everything available on the Internet) is were the majority of the money
>> is, but that doesn't mean that the infrastructure part of the business
>> can't itself be quite profitable, and easily afford v6 equipment.
> 
> Comcast is vertically-integrated.  They own 100% of NBC Universal as
> of March 2013 (and 51% before that).

They're not integrated in the same way Ma Bell was in the 1970's.
Comcast owns a tiny fraction of the content they are delivering, and
they make an increasingly smaller profit margin on the remainder of the
content they resell.


> As a result, they can afford to cross-subsidize the infrastructure.
> And look what has happened--Comcast is a leader in residential IPv6
> deployment.

I'm not sure there is a casual link between the two. If this was true,
net connections (no content) wouldn't be more profitable.

But numbers are easy to misinterpret. I don't recall if that stat
referred to the profitability of net services in aggregate or just the
per-subscriber profitability. Probably the latter. If the majority of
their customers are still video subscribers, most of their profits in
aggregate may still be coming from video, which may in fact have
resulted in them feeling rich enough to invest in IPv6.

I've heard in other talks[1] that Comcast's own internal needs were
strongly pushing them towards IPv6, because they had simply grown so big
that their scale was straining IPv4. Remember, every device Comcast
rents to you has an IP address. (Though I'm not sure why those devices
would need to be globally reachable within their network...maybe that
lets them reduce costs by centralizing management equipment.)

The point I was making is that the numbers from Comcast suggests that
over time its income from content will go down as its income from
connectivity goes up, and they seem to have found a model to deliver
that quite profitably, without requiring that it be subsidized by content.

1. I believe it was at the February BLU talk where I heard that
http://www.blu.org/meetings/2013/02/201005-hagerty.pdf


> ...as soon as the customer picks up that phone and calls the
> helpdesk, the last mile provider has lost their profit on that
> customer for 2 years.

I found that stat hard to believe. I'd like to see the source material
where Huston found that, as it doesn't seem to mesh with the numbers
I've seen reported[2] on Comcast's profit margins for connectivity
services.

Unless support calls are costing them hundreds of dollars, they have
room to swallow one or more calls per quarter and still break even.
(There is a big gap between what Comcast charges users for net service
(~$150 ($50 x 3) retail) and their cost to provide it (supposedly $7 per
quarter in 2010 and presumably lower now). Even if you low ball the
retail price at a discounted $25/mo=$75/quarter, that's $68/quarter in
profit. That should cover the cost[3] of a few support calls.)


2. see The Open Technology Institute report titled "Capping the Nation's
Broadband Future?" that I quoted in this BLU post:
http://www.mail-archive.com/discuss at blu.org/msg05791.html

3. "According to supportindustry.com and Help Desk 2000, the average
cost per call for 75% of the support organizations they surveyed is $25
or less. The analysts expect this cost to fall to $18 per call within
three years as a result of e-support technologies."
http://answers.google.com/answers/threadview/id/25.html
That was early 2000-era data, with costs trending down.


> I'm saying it could be dangerous to force the only way for the
> last-mile providers to make money be by them becoming "toll-booths" of
> the Internet, and that they would be forced into that role in order to
> survive if they are not allowed to also provide services (and become
> vertically integrated).

So you're saying 1000% markup isn't enough to make a viable business?
:-) ($7 x 1000% = $70)

Even if those source numbers are off by an order of magnitude (elsewhere
I see "cable broadband providers enjoy gross margins as high as 95
percent"; see [2]), it would seem like there is a huge margin for anyone
willing to make the necessary capital investment.

I think we need to dig a bit deeper into the industry's claims to make
sure we aren't buying in to their propaganda indented to stave off
regulation.

Maybe other ISPs don't have nearly the rosy numbers that Comcast does?
We know Google seems to think Fiber can actually be a profitable
business, even with their vastly higher-end, while lower-cost service
offering. And we know Verizon, at least its wireless division, is
substantially profitable.

So its hard to see the case that ISPs just don't have the money to pay
for IPv6. It seems more likely that you could increase their profits by
another 50% and they still wouldn't change their behavior, because the
same logic still applies - why spend money on new equipment if you can
make the same profits from the old equipment. It would be against their
fiduciary responsibility to their shareholders to do otherwise.

The only way to get existing ISPs to invest in IPv6 and higher bandwidth
is to either 1. take away their ability to make money from IPv4, or 2.
come up with a compelling case for how they can make easy money from IPv6.

I'm not sure many of them have the vision to take option #2, regardless
of how compelling the pitch might be. Instead they'll wait and see how
Comcast's deployment goes, and perhaps be shamed into it as Google Fiber
expands across the country.

Huston never makes the claim we should go back to the vertical
integration model. He only points out that moving away from it has led
to the infrastructure problems we have. I don't think it is possible to
go back to the old model, unless you are willing to turn ISPs into
modern day AOLs - where the *only* content you can access is that
provided by your ISP.

Short of that, any infrastructure improvements the ISPs make will
inherently end up being "shared" by their content competitors, and this
reduces their motivation to invest in infrastructure.

The Internet has reached and exceeded critical mass as a content source,
and it is going to be impossible for any one ISP to capture the majority
of their customer's content consumption when competing against that. So
whether you bar ISPs from providing content or now, soon enough they'll
still be in the same position of wanting to toll access to competing
content.


>> require less regulation, as government won't care how it implements the
>> infrastructure, as long as it sticks to that as its only business.
>> Though this still leaves plenty of opportunity for an ILEC to provide
>> minimum required services at the least cost and milk the monopoly. It
> 
> "minimum required services".  This is why we don't have gigabit to the
> home in the US.

Not really..."required" used above was in the context of regulation. The
current market is unregulated with respect to bandwidth (there is no
requirement). We don't have it, because current ISPs don't see the
profit in it.


> Or alternatively, open up all those rights of way to allow other
> providers to build their own last-mile access infrastructure alongside
> the ILECs and cable providers'.

My understanding is that the right-of-way is not the primary limiting
factor. Sure, we've all heard plenty of stories of Verizon FIOS not
being allowed to enter some town, or some other town having a franchise
agreement that bars competing providers from operating there, but I
think this is often not the limiting factor for why we don't have more
local providers.

I have 3 ISPs with wires on my poles, which is a lot more than most
places, but it is 3 and not 15 because of the capital costs of putting
them there, not because my town made it too hard for them to lease the
right-of-way.

No doubt there are tens of thousands of towns across the country that
would *pay* an ISP (if they could afford it) to put wires in their town.


> Google may be the game-changer here since they have a huge purse to
> cross-subsidize the infrastructure development (Google Fiber).

I think it is a mistake to look at Google Fiber as only being viable due
to cross-subsidization. Without question it is Google main business of
search advertising that motivated them to go through the pain (time and
tied up capital) of getting into this business, but while numbers
haven't been released, they've said on several occasions they expect
Fiber to be a profitable business on its own.

 -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: http://tmetro.venturelogic.com/