Terabits in a Rack and Petabits in the Data Center

By Scott Kipp • April 24, 2012

As the American scholar Albert Bartlett said, “The greatest shortcoming of the human race is our inability to understand the exponential function.” Bartlett was mainly referring to population growth, but the IT industry is seeing unbelievable, exponential bandwidth growth today. While human populations are growing at about 1%/year, bandwidths are growing at tens of percent per year.

For example, server output bandwidth is growing on average at 41%/year¹. That means servers are pushing twice as much data into the network every two years. This exponential growth is what is driving servers to move from Gigabit Ethernet (GbE) to 10GbE. These 10GbE server connections are having an avalanche effect on the rest of the network in the data center. Within the data center, the growth in core network bandwidth is expected to grow at 58%/year¹ – doubling every 18 months. Exponential growths at over 10% per year lead to amazing growths.

We’ve all heard about how the Internet is growing exponentially, but it is only growing at 32%/year². OK, that was a joke because 32%/year is crazy fast growth, but it is still quite smaller than 58% growth. The bandwidth within the data center is expected to grow faster than the bandwidth out to the Internet because there is much more server-to-server traffic within the data center than server-to-Internet traffic. Switched bandwidth within the data center can be orders of magnitude less expensive than routed bandwidth to the Internet. The bandwidth within the data center is expected to grow considerably faster than bandwidth on the Internet.

Exponential growths are hard to imagine as Barnet alludes to. To get a better understanding of the growth, I pick a few examples and see how bandwidth grows over a decade in Table 1. The first growth rate in the table shows how a server that pushes 1 Gbps in 2010 and grows at 41%/year would drive 32 Gbps of bandwidth in 2020. This shows how by the end of the decade, 10GbE won’t be enough bandwidth for servers. Ethernet networks have reached giga-levels so they are growing in Gbps per year now.

Table 1: Examples of Exponential Bandwidth Growth

Traffic Type	Bandwidth in 2010	Bandwidth Growth/Year	Months to Double Bandwidth	Bandwidth in 2020
Server to Switch	1 Gbps	41%	24	32 Gbps
Switch to Switch	5 Gbps	58%	18	506 Gbps
Data Center to Internet	40 Gbps	32%	30	642 Gbps
Total IP Traffic	61 Tbps	32%	30	979 Tbps

The network is growing even faster as these server bandwidths are aggregated. If a switch-to-switch link was pushing 5Gbps in 2010 and growing at 58%/year, then that link will be pushing 506 Gbps in 2020. This type of growth makes 100GbE look slow. This growth is hard to imagine, but that is what the numbers tell us. We can listen or not.

I show a couple of other examples in the table for traffic growth to the Internet. If an Internet data center is delivering 40Gbps in 2010 and growing at 32% per year, then the data center will need 642 Gbps by 2020. Likewise, the aggregate bandwidth of the Internet was 61 Tbps in 2010. If the IP traffic continues to grow at 32%/year, then the Internet will push 979Tbps of traffic in 2020. These astronomical numbers are hard to imagine, but data center operators are getting use to it.

Many data center planners have gotten used to these giga-scale increases. If they are deploying 100 servers with 10Gbps connections, then they have Tbps of aggregate bandwidth. Assuming a scenario where each server is delivering 25Gbps of output within the decade, then a rack with 40 servers can produce a terabit/second (Tbps) of data. If you have a thousand of these racks in a mega data center, then you’ll get a petabit per second (Pbps or 1,000 Terabits) of bandwidth. While few mega data centers are being deployed at this scale, lots of data centers can deploy hundreds of 10GbE servers and produce a Tbps of bandwidth within a few racks. These astronomical numbers are the result of exponential growth.

I’ll be giving a keynote at the “Terabit Optical and Data Networking Conference” on Thursday April 19^th at 9:45am in Cannes, France. The speech is titled “Terabits in a Rack and Petabits in a Data Center” and I’ll discuss how 10GbE servers can drive terabits of data from a rack and petabits (1,000 terabits) of data through the data center every second. Imagine moving a petabit/s (Pbps) of bandwidth within the data center to scale applications to unprecedented levels.

I’ve also written a story for Network World titled “Exponential Growth in Bandwidth and Cost Declines”. This article shows how the costs of networks will need to decline exponentially to maintain the growth rates that are being forecast. Read the article online here.

Enjoy,

Scott Kipp

President of the Ethernet Alliance

PS. Tom Palkert will also be representing the Ethernet Alliance and giving a presentation on 100GbE standards on Wednesday, April 18^th at 2:05pm.