Transit vs peering: what is worth using when?

Traditionally, content providers and ISPs each buy transit service from larger networks. While “eyeball” networks can directly exchange the traffic free of charge, most of them need to use middlemen. The right combination of Internet transit and peering is a critical point for Internet business.

Peering in Internet business

Unlike the telephony world, where nearly all interconnections involve payment, the Internet peering is usually free of charge, also known as sender keeps all (the money paid by their customer). Internet Service Providers charge their customers for both incoming and outgoing traffic, so the business case still works without interconnection fees.

So how does peering work in practice?

There are two options: direct or private peering, or peering over a public internet exchange (IX). Technically, private peering can work over a long distance, but it largely happens within the same building, or at least the same city. Private peering really only makes sense when there's a significant traffic involved, because each peer requires a separate router or switch port and a circuit: even within the same building a cable to a peering partner can be costly.

Peering over a public internet exchange implies a connection to the IX, which is basically a large Ethernet switch or set of Ethernet switches. Every IX member/customer connects a router to the IX peering LAN. All these routers get an IP address within the same subnet, so they can exchange data directly among each other. The only thing required to let peering traffic flow between two IX members is a BGP session between the respective BGP-speaking routers.

However, no matter how polite your peering requests are, some prospective peers will say "no". For instance, global ISPs typically aren't willing to peer with small regional networks, otherwise they would end up transporting the traffic most of the way without getting paid for it.

Obviously, networks that mainly host content and thus mostly have outbound traffic primarily benefit from peering with networks that mostly connect consumers and thus mostly have incoming traffic, and vice versa. If traffic is mostly in one direction, there's little benefit in peering with a network with similar traffic path. However, some larger networks require that incoming and outgoing traffic is roughly balanced in order to set up peering.

So before connecting to an IX or setting up private peering, it's essential to well estimate how much traffic you'll be able to handle over a public or private peering connection, so you can figure out whether the money you save on transit costs will pay for the peering costs. For private peering within the same building, this can be as little as an unused router or switch port and the costs of a cross connect. But peering can get rather expensive if it involves leasing a circuit to a remote city and pay a possibly steep IX connection fee.

Also don't underestimate the efforts required: connecting to a transit ISP or an IX takes roughly the same amount of work. However, once a transit connection is up, it requires very little attention. With an IX connection, the real effort is setting up peering with other networks connected to the IX. In order to figure out the amount of work involved, consider twenty minutes to set up a typical peering session and then one minute per month for maintenance is a reasonable estimate. Multiply this by 100 peers and that's four days full time to set up the peering connections and then two hours a month for maintenance.

Determine whether peering is worth compared to  buying transit

First of all, figure out the total cost of peering connection setup: hardware, circuits, port and member fees, and work. In the US, the biggest internet exchanges are the   Equinix and Any2 in New York, Ashburn, Los Angeles, the Bay Area and Miami, and the  NYIIX in New York. The three largest IXes in Europe and the world are AMS-IX (Amsterdam), DE-CIX (Frankfurt) and LINX (London). Their average price is $650 p/m for a 1 Gbps connection and $1900 p/m for a 10 Gbps connection. The bad news is that you pay this amount even if you don't send a single packet over your connection, but the good news is that you don't pay any extra even if you exceed the volume agreed for this connection. For simplicity, let's assume that the total cost of connecting on an IX are $1000 per month at 1 Gbps and $2500 per month at 10 Gbps.

Whether it makes sense financially to set up a peering connection to an IX depends on the amount of money this could allow you to save on transit costs. Transit costs vary widely by region, on- or off-net location and volume commitment. If the variable part of cost of a megabit of transit is $1, then it doesn't worth installing a 1 Gbps IX connection for $1000, as the best you could hope to do is reduce transit costs by the same $1000, and in practice you can't use a connection at 100%. You'd have to use a 10 Gbps IX connection at 2500 Mbps to break even. However, if your transit costs are $10 per megabit, a 1 Gbps IX connection would make sense starting at 100 Mbps.

It's hard to determine in advance how much traffic to expect exchanged through peering. By peering with the route server of an internet exchange, you automatically peer with everyone else who peers with this route server. As first step, you can ask a network similar to you what percentage of total traffic they exchange with route server peers to have an idea of how much you can expect to exchange if you join.

The second step is to determine if your prospective peers have an open peering policy. You can find this information in PeeringDB. Unfortunately sometimes this information is out of date, and peering requests go unanswered. Don't be afraid to ask the largest prospective peers if they want to peer with you before committing financially, even if their peering policy is "selective" or "restrictive", unless they have a published peering policy with requirements that you clearly don't meet.

Once you have a list of networks that you expect to peer with, you need to estimate how much traffic you'll be sending to and receiving from those networks. There is only one way to do this: using Netflow system that allows routers sample their traffic. The samples can then be analyzed in various ways, including determining the amount of traffic to/from a given autonomous system.

The calculations for private peering are largely the same as for peering over an IX, but with the advantage that you know exactly who you'll be peering with in advance, so there's less uncertainty. Despite the fact that there are no IX port costs involved, private peering usually only makes sense between relatively large networks. For smaller networks, there's just not enough traffic between the two networks to bother with.

If you're not quite ready to start peering yourself, but you have a big amount of traffic that could be handled regionally through peering, it makes sense to negotiate a discount transit rate for that regional traffic: the ISP could provide transit only to those IPv4 prefixes that they have access to through their own peering. Such partial transit can be much cheaper than regular transit service.

Whatever your final peering mix consists of, you must make sure that the proper policies are in place to efficiently route the traffic across your providers and peers. Running BGP buys  a lot of peace of mind. However, BGP routing as well as the internal routing OSPF (and the  interactions between BGP and OSPF) could add a lot of complexity. Sometimes, these protocols don't fix connectivity issues the way they should. Or worse, they create outages of their own that wouldn't have occurred in a simpler network.

For the most part, BGP will route around   network outages. However, sometimes packets disappear into a black hole: for some reason, the packets are lost, but the routing protocols don't notice that there's a problem, so traffic continues to flow towards the black hole rather than be rerouted. One way to detect this problem is to monitor reachability of main remote services. Another is analyzing the total traffic: in the presence of a black hole the total traffic will be much lower than usual. Once detected, recovering from routing black hole affecting an ISP or peer is quite simple: shut down the BGP session towards that provider until they've fixed the problem. Alternatively you can automate the process of selecting the best performing transit provider or peer by deploying a route optimizer - Noction Intelligent Routing Platform – which evaluates all your IXes, ISPs and partial peers for packet loss and latency and automatically reroutes traffic through the most reliable path.