From the April, 2015 Issue of Cabling Installation & Maintenance Magazine
The IEEE 802.3 Ethernet Working Group is keeping busy with a number of fiber- and copper-based specifications for data center networking.
By Patrick McLaughlin
A commonly deployed local area network (LAN) architecture within a data center includes multiple layers of switching, through which signals pass when traveling from a server of origin to a destination server. In the widely deployed three-tier architecture servers connect to access switches (tier 1), which connect back to aggregation switches (tier 2), which connect back to core switches (tier 3). Generally a three-tier network contains fewer core switches than aggregation switches, and fewer aggregation switches than access switches.
The landscape is changing in some data centers, where the number of hops required for signals to go through three tiers of switching can negate some of the efficiencies gained by technologies such as server virtualization. As a result, over the past couple years several other switch architectures, or switch fabrics as they often are called, have emerged. These new architectures decrease the number of hops, and thereby decrease the amount of latency, between servers.
In any architecture, the larger the amount of data traveling through a connection—be it server-to-switch or switch-to-switch—the higher the speed required to successfully facilitate that data traffic. In a three-tier architecture, connections to core switches represent the need for the highest speeds. From a physical layer standpoint, the cabling used to make these highest-speed connections is considered backbone cabling. In many cases the connections between core and aggregation switches is also considered backbone cabling, supporting higher speeds than the cabling between aggregation and access switches, and between access switches and servers.
Tracking trends
The extent to which high-speed connections are being deployed in these data center LANs is tracked by a number of research and analyst firms. Crehan Research (www.crehanresearch.com) is one such firm, and this year Crehan has published studies sizing up server and switch consumption.
In March Crehan published a regular installment of its data center switch report, which summarized the market's calendar year 2014. Key points from that study include the following.
- 40-Gbit/sec Ethernet data center switch shipments more than tripled year-over-year.
- 10GBase-T data center switching rapidly expanded in 2014, exiting the year with an annual run rate of more than 2 million ports.
- Despite steep price declines in some individual data center Ethernet switch market segments, overall market pricing remained relatively stable in 2014 because of greater adoption of higher-speed switches.
That report also examined branded and white-box switches, finding "customer deployments of branded data center Ethernet switches grew 10 percent in 2014, a slight increase over the 2013 growth rate, despite an increase in white box switch offerings … Branded Ethernet switch shipments in the high-speed top-of-rack, or fixed, segment—where white box switch offerings are most prevalent—increased by almost 40 percent in 2014."
Seamus Crehan, the firm's president, noted, "Most of the white box and ODM-direct switch market volume is still driven by a few of the very large, hyperscale cloud service providers, and its growth has remained largely a function of these few service providers building out their data center networks. Over the past few years, the branded data center switch vendors have really narrowed the gap on the perceived advantages of white box switches. Although it varies by vendor, this has included lowering prices significantly, incorporation of more merchant silicon, offering more programmability and SDN [software-defined networking] features, disaggregation of hardware and software, and bringing open compute and networking designs to market."
In January Crehan published a server-class adapter and LAN-on-motherboard (LOM) long-range forecast report. The forecast indicates that the availability of new data center Ethernet speeds will lead to a much stronger server networking upgrade cycle than seen over the past decade. Crehan specifically points to the arrival of 25- 50- and 100-Gbit/sec Ethernet products, in combination with existing 10- and 40-Gbit/sec Ethernet products. Over the next three years, more than two-thirds of total networking ports will migrate to one of these high speeds of Ethernet, the report says.
Seamus Crehan commented, "Over the past decade or so, the data center has seen many significant changes, and it is no longer a one-size-fits-all market. Consequently, we are seeing solutions that are highly optimized for the needs of specific customer segments, whether it be a fully vertically integrated system for a converged enterprise network or a bare-bones disaggregated system for a massively scalable cloud service provider. As a result, the market is currently looking for more-targeted Ethernet networking solutions."
Crehan made public the following general findings from its report.
- 25-Gbit/sec Ethernet is expected to see a strong initial ramp from deployment by cloud providers like Google and Microsoft. That's an area of the market in which 10-Gbit/sec Ethernet server networking is currently prevalent.
- 10-Gbit/sec Ethernet is likely to see its third, and largest, adoption phase as the mainstream enterprise market upgrades its mostly 1GBase-T server and server access infrastructure to 10GBase-T.
- 40-Gbit/sec Ethernet is beginning to ramp significantly, benefitting from a current window of opportunity as the most attractively priced data center Ethernet speed, from a bandwidth perspective.
Busy bodies
The formation of specifications for this high-speed Ethernet communication takes place within the Institute of Electrical and Electronics Engineers (IEEE) 802.3 Ethernet Working Group. Currently several task forces and study groups within 802.3 are working to define specifications for the transmission speeds Crehan discusses, as well as others.
Siemon (www.siemon.com) maintains a section of its corporate blog called the Standards Informant. Primarily updated by Siemon's director of standards and technology Valerie Maguire, the Standards Informant provides updated information on the works-in-progress from standards organizations the Telecommunications Industry Association (TIA), the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC), and the IEEE. The following paragraphs describing IEEE 802.3's "bm" and "bs" projects on high-speed network standards are excerpted from Siemon's Standards Informant.
The 802.3bm 40 Gb/s and 100 Gb/s Fiber Optic Task Force—The rapid growth of server, network, and internet traffic is driving the need for higher data rate, higher density, and lower cost optical fiber Ethernet solutions, especially in data centers. The 100 Gb/s optical fiber Ethernet applications specified in IEEE 802.3ba include a 10-lane electrical interface (20 fibers total) for operation over multimode optical fiber cabling (100GBase-SR10). Advances in technology support the specification of a new multimode 100 Gb/s physical layer specification with reduced lane count, which will simplify upgrades from 40GBase-SR4 using the existing cabling plant. In addition, this project adds a 40 Gb/s PHY specification for operation over up to 40 km of singlemode optical fiber cabling and two new attachment interfaces supporting 100 Gb/sec transmission in electrical backplanes.
100GBase-SR4 is 100 Gb/s transmission using a 4-lane electrical interface (8 fibers total) for operation over multimode optical fiber cabling and reach up to at least 100 meters. 40GBase-ER4 is 40 Gb/s transmission using four wavelength-division multiplexing (WDM) lanes (2 fibers total) for operation over singlemode optical fiber cabling with reach up to at least 40 kilometers.
Draft specifications have Om3 cabling supporting 40GBase-SR4 to 70 meters and Om4 supporting 40GBase-SR4 to 100 meters.
The 802.3bs 400 Gb/s Ethernet Task Force—Industry experts predict that the necessary processing capacity in 2015 will be 10x that required in 2010 and the necessary capacity in 2020 will be 100x that required in 2010 (Source: IEEE 802.3 Industry Connections Ethernet Bandwidth Assessment report). 400 Gb/s Ethernet solutions will be needed to support this increased capacity. This project will define 400 Gb/s physical layer specifications for operation over up to 100 meters of multimode optical fiber cabling, up to 10 kilometers of singlemode optical fiber cabling, up to 2 kilometers of singlemode optical fiber cabling, and up to 500 meters of singlemode optical fiber cabling with optional Energy Efficient Ethernet (EEE), and at least one new attachment interface for supporting 400 Gb/s transmission in electrical backplanes.
Emergence of 25G
As we have covered in previous issues, the IEEE's 802.3 bq task force is developing 40GBase-T specifications, and along with them, 25GBase-T (see "IEEE to study new Base-T data rates for data centers and enterprise applications," February 2015; and "Category 8's 30-meter reach is not a stretch," March 2015). That group is targeting the early part of 2016 as a completion date. A separate task force, 802.3by, is working to specify 25-Gbit/sec Ethernet over a single lane. When the IEEE formed a study group on single-lane 25 Gbits/sec in July 2014 (a task group forms only after a study group's work has been completed and its request accepted), the study group's chair, Mark Nowell, commented, "The application of single-lane 25-Gbit/sec signaling technology provides Ethernet with a solution set that can be reused by those companies building the data center of tomorrow … The heavy lifting in developing and standardizing 25-Gbit/sec signaling technologies has been done as part of the development of 100-Gbit/sec Ethernet. These technologies can be reused to enable a single-lane 25-Gbit/sec Ethernet solution set for server interconnects for these future data centers."
Now for both copper and fiber, 25-Gbit/sec Ethernet standards are in development. The physical setup of connections in data centers is changing, as characteristics such as the number of switching tiers and the number of servers in a virtualized network are reduced. In many cases these types of changes carry with them the need for fewer, but faster, connections. At the same time, the once-automatic tenfold increase in speeds with each successive technology generation is no longer absolute. The inclusion of 40 Gbit/sec Ethernet along with 100-Gbit/sec in the IEEE 802.3bm altered the course. Now the in-development projects to specify standalone 25-Gbit/sec transmission are being written to provide data center administrators with more options to accommodate those faster connection requirements.
Patrick McLaughlin is our chief editor.
Archived CIM Issues